Posts in Go (20 found)
iDiallo Yesterday

That's it, I'm cancelling my ChatGPT

Just like everyone, I read Sam Altman's tweet about joining the so-called Department of War, to use ChatGPT on DoW classified networks. As others have pointed out, this is the entry point for mass surveillance and using the technology for weapons deployment. I wrote before that we had the infrastructure for mass surveillance in place already, we just needed an enabler. This is the enabler. This comes right after Anthropic's CEO wrote a public letter stating their refusal to work with the DoW under their current terms. Now Anthropic has been declared a public risk by the President and banned from every government system. Large language models have become ubiquitous. You can't say you don't use them because they power every tech imaginable. If you search the web, they write a summary for you. If you watch YouTube, one appears right below the video. There's a Gemini button on Chrome, there's Copilot on Edge and every Microsoft product. There it is in your IDE, in Notepad, in MS Paint. You can't escape it. Switching from one LLM to the next makes minimal to no difference for everyday use. If you have a question you want answered or a document to summarize, your local Llama will do the job just fine. If you want to compose an email or proofread your writing, there's no need to reach for the state of the art, any model will do. For reviewing code, DeepSeek will do as fine a job as any other model. A good use of ChatGPT's image generator. All this to say, ChatGPT doesn't have a moat. If it's your go-to tool, switching away from it wouldn't make much of a difference. At this point, I think the difference is psychological. For example, my wife once told me she only ever uses Google and can't stand any other search engine. What she didn't know was that she had been using Bing on her device for years. She had never noticed, because it was the default. When I read the news about OpenAI, I was ready to close my account. The only problem is, well, I never use ChatGPT. I haven't used it in years. My personal account lay dormant. My work account has a single test query despite my employer trying its hardest to get us to use it. But I think none of that matters when OpenAI caters to a government agency with a near-infinite budget. For every public account that gets closed, OpenAI will make up for it with deeper integration into classified networks. Not even 24 hours later, the US is at war with Iran. So while we're at it, here is a nice little link to help you close your OpenAI account .

0 views
devansh Yesterday

Bypassing egress filtering in BullFrog GitHub Action

GitHub Actions runners are essentially ephemeral Linux VMs that execute your CI/CD pipelines. The fact that they can reach the internet by default has always been a quiet concern for security-conscious teams — one malicious or compromised step can silently exfiltrate secrets, environment variables, or runner metadata out to an attacker-controlled server. A handful of tools have been built to address exactly this problem. One of them is BullFrog — a lightweight egress-filtering agent for GitHub Actions that promises to block outbound network traffic to domains outside your allowlist. The idea is elegant: drop everything except what you explicitly trust. So naturally, I poked at it. BullFrog ( ) is an open-source GitHub Actions security tool that intercepts and filters outbound network traffic from your CI runners. You drop it into your workflow as a step, hand it an list and an , and it uses a userspace agent to enforce that policy on every outbound packet. A typical setup looks like this: After this step, any connection to a domain not on the allowlist should be blocked. The idea is solid. Supply chain attacks, secret exfiltration, dependency confusion — all of these require outbound connectivity. Cutting that off at the network layer is a genuinely good defensive primitive. The BullFrog agent ( ) intercepts outbound packets using netfilter queue (NFQUEUE). When a DNS query packet is intercepted, the agent inspects the queried domain against the allowlist. If the domain matches — the packet goes through. If it doesn't — dropped. For DNS over UDP, this is fairly straightforward: one UDP datagram, one DNS message. But DNS also runs over TCP, and TCP is where things get interesting. DNS-over-TCP is used when a DNS response exceeds 512 bytes (common with DNSSEC, large records, etc.), or when a client explicitly prefers TCP for reliability. RFC 1035 specifies that DNS messages over TCP are prefixed with a 2-byte length field to delimit individual messages. Crucially, the same TCP connection can carry multiple DNS messages back-to-back — this is called DNS pipelining (RFC 7766). This is the exact footgun BullFrog stepped on. BullFrog's function parses the incoming TCP payload, extracts the first DNS message using the 2-byte length prefix, checks it against the allowlist, and returns. It never looks at the rest of the TCP payload. If there are additional DNS messages pipelined in the same TCP segment, they are completely ignored. The consequence: if the first message queries an allowed domain, the entire packet is accepted — including any subsequent messages querying blocked domains. Those blocked queries sail right through to the upstream DNS server. The smoking gun is at agent/agent.go#L403 : The function slices , decodes that single DNS message, runs the policy check on it, and returns its verdict. Any bytes after — which may contain one or more additional DNS messages — are never touched. It's a classic "check the first item, trust the rest" mistake. The guard is real, but it only covers the front door. The first query acts as camouflage. The second is the actual payload — it can encode arbitrary data in the subdomain (hostname, runner name, env vars, secrets) and have it resolved by a DNS server the attacker controls. They observe the DNS lookup on their end and retrieve the exfiltrated data — no HTTP, no direct socket to a C2, no obvious telltale traffic pattern. The workflow setup to reproduce this: The script below builds two raw DNS queries, wraps each with a TCP 2-byte length prefix per RFC 1035, concatenates them into a single payload, and sends it over one TCP connection to . Runner metadata (OS, kernel release, hostname, runner name) is embedded in the exfiltration domain. Running this against a real workflow with BullFrog configured to allow only , the runner's OS, kernel version, hostname, and env variable were successfully observed in Burp Collaborator's DNS logs — proving that the second DNS query bypassed the policy entirely. I reported this to the BullFrog team on November 28th, 2025 via their GitHub repository. After roughly three months with no response, acknowledgment, or patch, I'm disclosing this publicly. The vulnerability is straightforward to exploit and affects any workflow using BullFrog with that routes DNS over TCP — which Google's supports natively. Affected Versions : v0.8.4 and likely all prior versions Fixed Versions : None as of disclosure date (did not bother to check) What is BullFrog? How It Works DNS Over TCP Vulnerability Vulnerable Code Proof of Concept Attack Scenario The PoC Script Disclosure Timeline Discovery & Report : 28th November 2025 Vendor Contact : 28th November 2025 Vendor Response : None Public Disclosure : 28th February 2026

0 views
Martin Fowler 6 days ago

Fragments: February 23

Do you want to run OpenClaw? It may be fascinating, but it also raises significant security dangers. Jim Gumbley, one of my go-to sources on security, has some advice on how to mitigate the risks. While there is no proven safe way to run high-permissioned agents today, there are practical patterns that reduce the blast radius. If you want to experiment, you have options, such as cloud VMs or local micro-VM tools like Gondolin. He outlines a series of steps to consider ❄                ❄                ❄                ❄                ❄ Caer Sanders shares impressions from the Pragmatic Summit . From what I’ve seen working with AI organizations of all shapes and sizes, the biggest indicator of dysfunction is a lack of observability. Teams that don’t measure and validate the inputs and outputs of their systems are at the greatest risk of having more incidents when AI enters the picture. I’ve long felt that people underestimated the value of QA in production . Now we’re in a world of non-deterministic construction, a modern perspective of observability will be even more important Caer finishes by drawing a parallel with their experience in robotics If I calculate the load requirements for a robot’s chassis, 3D model it, and then have it 3D-printed, did I build a robot? Or did the 3D printer build the robot? Most people I ask seem to think I still built the robot, and not the 3D printer. … Now, if I craft the intent and design for a system, but AI generates the code to glue it all together, have I created a system? Or did the AI create it? ❄                ❄                ❄                ❄                ❄ Andrej Karpathy is “very interested in what the coming era of highly bespoke software might look like.” He spent half-an-hour vibe coding a individualized dashboard for cardio experiments from a specific treadmill the “app store” of a set of discrete apps that you choose from is an increasingly outdated concept all by itself. The future are services of AI-native sensors & actuators orchestrated via LLM glue into highly custom, ephemeral apps. It’s just not here yet. ❄                ❄                ❄                ❄                ❄ I’ve been asked a few times about the role LLMs should play in writing. I’m mulling on a more considered article about how they help and hinder. For now I’ll say two central points are those that apply to writing with or without them. First, acknowledge anyone who has significantly helped with your piece. If an LLM has given material help, mention how in the acknowledgments. Not just is this being transparent, it also provides information to readers on the potential value of LLMs. Secondly, know your audience. If you know your readers will likely be annoyed by the uncanny valley of LLM prose, then don’t let it generate your text. But if you’re writing a mandated report that you suspect nobody will ever read, then have at it. (I hardly use LLMs for writing, but doubtless I have an inflated opinion of my ability.) ❄                ❄                ❄                ❄                ❄ In a discussion of using specifications as a replacement to code while working with LLMs, a colleague posted the following quotation “What a useful thing a pocket-map is!” I remarked. “That’s another thing we’ve learned from your Nation,” said Mein Herr, “map-making. But we’ve carried it much further than you. What do you consider the largest map that would be really useful?” “About six inches to the mile.” “Only six inches!” exclaimed Mein Herr. “We very soon got to six yards to the mile. Then we tried a hundred yards to the mile. And then came the grandest idea of all! We actually made a map of the country, on the scale of a mile to the mile!” “Have you used it much?” I enquired. “It has never been spread out, yet,” said Mein Herr: “the farmers objected: they said it would cover the whole country, and shut out the sunlight! So we now use the country itself, as its own map, and I assure you it does nearly as well.” from Lewis Carroll, Sylvie and Bruno Concluded, Chapter XI, London, 1893, acquired from a Wikipedia article about a Jorge Luis Borge short story. ❄                ❄                ❄                ❄                ❄ Grady Booch: Human language needs a new pronoun, something whereby an AI may identify itself to its users. When, in conversation, a chatbot says to me “I did this thing”, I - the human - am always bothered by the presumption of its self-anthropomorphizatuon. ❄                ❄                ❄                ❄                ❄ My dear friends in Britain and Europe will not come and visit us in Massachusetts. Some folks may think they are being paranoid, but this story makes their caution understandable. The dream holiday ended abruptly on Friday 26 September, as Karen and Bill were trying to leave the US. When they crossed the border, Canadian officials told them they didn’t have the correct paperwork to bring the car with them. They were turned back to Montana on the American side – and to US border control officials. Bill’s US visa had expired; Karen’s had not. “I worried then,” she says. “I was worried for him. I thought, well, at least I am here to support him.” She didn’t know it at the time, but it was the beginning of an ordeal that would see Karen handcuffed, shackled and sleeping on the floor of a locked cell, before being driven for 12 hours through the night to an Immigration and Customs Enforcement (ICE) detention centre. Karen was incarcerated for a total of six weeks – even though she had been travelling with a valid visa. Prioritize isolation first. Clamp down on network egress. Don’t expose the control plane. Treat secrets as toxic waste. Assume the skills ecosystem is hostile. Run endpoint protection.

0 views
Carlos Becker 1 weeks ago

Announcing GoReleaser v2.14

Happy 2026! The first release of the year is here, and it is packed with goodies!

0 views
Filippo Valsorda 1 weeks ago

Turn Dependabot Off

Dependabot is a noise machine. It makes you feel like you’re doing work, but you’re actually discouraging more useful work. This is especially true for security alerts in the Go ecosystem. I recommend turning it off and replacing it with a pair of scheduled GitHub Actions, one running govulncheck, and the other running your test suite against the latest version of your dependencies. On Tuesday, I published a security fix for filippo.io/edwards25519 . The method would produce invalid results if the receiver was not the identity point. A lot of the Go ecosystem depends on filippo.io/edwards25519, mostly through github.com/go-sql-driver/mysql (228k dependents only on GitHub). Essentially no one uses . Yesterday, Dependabot opened thousands of PRs against unaffected repositories to update filippo.io/edwards25519. These PRs were accompanied by a security alert with a nonsensical, made up CVSS v4 score and by a worrying 73% compatibility score , allegedly based on the breakage the update is causing in the ecosystem. Note that the diff between v1.1.0 and v1.1.1 is one line in the method no one uses . We even got one of these alerts for the Wycheproof repository, which does not import the affected filippo.io/edwards25519 package at all . Instead, it only imports the unaffected filippo.io/edwards25519/field package. We have turned Dependabot off. But isn’t this toil unavoidable, to prevent attackers from exploiting old vulnerabilities in your dependencies? Absolutely not! Computers are perfectly capable of doing the work of filtering out these irrelevant alerts for you. The Go Vulnerability Database has rich version, package, and symbol metadata for all Go vulnerabilities. Here’s the entry for the filippo.io/edwards25519 vulnerability , also available in standard OSV format . Any decent vulnerability scanner will at the very least filter based on the package, which requires a simple . This already silences a lot of noise, because it’s common and good practice for modules to separate functionality relevant to different dependents into different sub-packages. 1 For example, it would have avoided the false alert against the Wycheproof repository. If you use a third-party vulnerability scanner, you should demand at least package-level filtering. Good vulnerability scanners will go further, though, and filter based on the reachability of the vulnerable symbol using static analysis. That’s what govulncheck does! govulncheck noticed that my project indirectly depends on filippo.io/edwards25519 through github.com/go-sql-driver/mysql, which does not make the vulnerable symbol reachable, so it chose not to notify me. If you want, you can tell it to show the package- and module-level matches. It’s easy to integrate govulncheck into your processes or scanners, either using the CLI or the golang.org/x/vuln/scan Go API. You can replace Dependabot security alerts with this GitHub Action. It will run every day and only notify you if there is an actual vulnerability you should pay attention to. False positive alerts are not only a waste of time, they also reduce security by causing alert fatigue and making proper triage impractical. A security vulnerability should be assessed for its impact: production might need to be updated, secrets rotated, users notified! A business-as-usual dependency bump is a woefully insufficient remediation for an actual vulnerability, but it’s the only practical response to the constant stream of low-value Dependabot alerts. This is why as Go Security Team lead back in 2020–2021 I insisted the team invest in staffing the Go Vulnerability Database and implement a vulnerability scanner with static analysis filtering. The govulncheck Action will not automatically open a PR for you, and that’s a good thing! Now that security alerts are not mostly noise, you can afford to actually look at them and take them seriously, including any required remediation. Noisy vulnerability scanners also impact the open source ecosystem. I often get issues and PRs demanding I update the dependencies of my projects due to vulnerabilities that don’t affect them, because someone’s scanner is failing to filter them. That’s extra toil dropped at the feet of open source maintainers, which is unsustainable. The maintainer’s responsibility is making sure projects are not affected by security vulnerabilities. The responsibility of scanning tools is making sure they don’t disturb their users with false positives. The other purpose of Dependabot is to keep dependencies up to date, regardless of security vulnerabilities. Your practices and requirements will vary, but I find this misguided, too. Dependencies should be updated according to your development cycle, not the cycle of each of your dependencies. For example you might want to update dependencies all at once when you begin a release development cycle, as opposed to when each dependency completes theirs. There are two benefits to quick updates, though: first, you can notice and report (or fix) breakage more rapidly, instead of being stalled by an incompatibility that could have been addressed a year prior; second, you reduce your patch delta in case you need to update due to a security vulnerability, reducing the risk of having to rush through a refactor or unrelated fixes. You can capture both of those benefits without actually updating the dependencies by simply running CI against the latest versions of your dependencies every day. You just need to run before your test suite. In the npm ecosystem, you just run instead of . This way, you will still be alerted quickly of any potential issues, without having to pay attention to unproblematic updates, which you can defer to whenever fits your project best. This is a lot safer, too, because malicious code recently added to a dependency will not rapidly reach users or production, but only CI. Supply chain attacks have a short half-life! You can further mitigate the risk by using a CI sandboxing mechanism like geomys/sandboxed-step , which uses gVisor to remove the ambient authority that GitHub Actions grants every workflow, including supposedly read-only ones . For more spicy open source opinions, follow me on Bluesky at @filippo.abyssdomain.expert or on Mastodon at @[email protected] . The Tevere has overflowed its lower banks, so a lot of previously familiar landscapes have changed slightly, almost eerily. This is the first picture I took after being able to somewhat safely descend onto (part of) the river’s banks. My work is made possible by Geomys , an organization of professional Go maintainers, which is funded by Ava Labs , Teleport , Tailscale , and Sentry . Through our retainer contracts they ensure the sustainability and reliability of our open source maintenance work and get a direct line to my expertise and that of the other Geomys maintainers. (Learn more in the Geomys announcement .) Here are a few words from some of them! Teleport — For the past five years, attacks and compromises have been shifting from traditional malware and security breaches to identifying and compromising valid user accounts and credentials with social engineering, credential theft, or phishing. Teleport Identity is designed to eliminate weak access patterns through access monitoring, minimize attack surface with access requests, and purge unused permissions via mandatory access reviews. Ava Labs — We at Ava Labs , maintainer of AvalancheGo (the most widely used client for interacting with the Avalanche Network ), believe the sustainable maintenance and development of open source cryptographic protocols is critical to the broad adoption of blockchain technology. We are proud to support this necessary and impactful work through our ongoing sponsorship of Filippo and his team. This also makes it possible to prune the tree of dependencies only imported by packages that are not relevant to a specific dependent, which has a large security benefit.  ↩ This also makes it possible to prune the tree of dependencies only imported by packages that are not relevant to a specific dependent, which has a large security benefit.  ↩

0 views
(think) 1 weeks ago

Supercharging Claude Code with the Right (CLI) Tools

I’ve been using Claude Code quite a bit lately, and I got curious – what if I asked it directly which tools would make it more productive? Not the usual suspects like , or , but tools it wishes it had access to, tools that would genuinely extend its capabilities. So I did exactly that. I asked Claude Code: “What are the most valuable CLI tools I could install for you, outside of the ones you already have?” The answer was surprisingly thoughtful and insightful, so I figured I’d share it here along with my own commentary. Here are 10 tools, ranked by how useful they’d be for an AI coding assistant. Note: I write all my blog posts old-school, but this time around I took the liberty to just extend with my comments the output generated by Claude Code. Note also that the post includes some installation instructions that are macOS-specific. That’s what I got from Claude on my local machine (a Mac mini), and I felt it didn’t make much sense to tweak them given how many combinations of operating systems and package managers exist. This was Claude’s number one pick, and I can see why. ast-grep does structural code search and refactoring using AST patterns. Instead of fumbling with regex to find “all calls to function X with 3 arguments”, you write patterns that look like actual code: This is the kind of thing where regex is fragile and error-prone, but AST matching just works. Supports 20+ languages via tree-sitter . A structural diff tool that understands syntax. difftastic compares files by AST nodes rather than lines, so it won’t flag whitespace changes or reformatting as meaningful diffs. This makes reviewing AI-generated changes much clearer – and let’s be honest, reviewing changes is half the job when working with an AI assistant. AI assistants generate a lot of shell commands, and shell scripting is notoriously full of pitfalls (unquoted variables, vs. , POSIX compatibility…). ShellCheck catches these before they blow up. Given that shell bugs can be destructive (e.g., expanding to ), having a safety net here is valuable. A modern replacement with sane regex syntax – no more escaping nightmares. Uses standard PCRE-style regex and has a string-literal mode ( ) for replacing code strings full of metacharacters. Simple, but it eliminates a whole class of errors when generating substitution commands. Sloc Cloc and Code – a fast code counter that gives you an instant overview of a codebase: languages, lines of code, complexity estimates. Understanding the shape of a project before diving in is genuinely useful context for an AI assistant, and this is hard to replicate by manually scanning files. Note: I was under the impression that cloc is a better tool, but perhaps I was mistaken. 1 for YAML (and JSON, TOML, XML). Modern projects are drowning in YAML – GitHub Actions workflows, Kubernetes manifests, Docker Compose files. yq can programmatically query and update YAML while preserving comments and formatting, which is much more reliable than text-based editing that can break indentation. Structural search and replace that works across languages without needing a full parser. Complements ast-grep for simpler pattern matching – it understands delimiters (braces, parens, quotes) but doesn’t need tree-sitter grammar support. Great for quick refactoring across less common languages or config files. Note: I was happy to see that was written in OCaml, but when I installed it I got a warning that the project was deprecated and doesn’t support OCaml 5, so I’m not sure about its future. A command-line benchmarking tool that runs commands multiple times and gives you proper statistical analysis. When you ask an AI to optimize something, it’s nice to have real numbers. The flag produces results ready for a PR description. A file watcher that executes commands when files change. Useful for setting up persistent feedback loops – rerun tests on save, rebuild docs when markdown changes, restart a dev server after config edits. One command instead of cobbling together something with and shell scripts. A syntax-highlighting pager for and friends. Provides word-level diff highlighting, so when only a variable name changes in a long line, you see exactly that. Mostly benefits the human reviewing the AI’s work, but that’s arguably where it matters most. If you only install one tool from this list, make it . It’s the biggest capability gap – an AI assistant limited to regex-based search and replace is like a carpenter limited to a hand saw. Everything else is nice to have, but structural code understanding is a genuine superpower. You can install everything at once if you’re feeling adventurous: I’m not ashamed to admit that I had never heard of some of the tools (e.g. , and ), and I had only one of them installed ( ). 2 It’s never too late to learn something new! By the way, keep in mind that depending on the programming languages that you’re using there are other language specific tools that you can benefit from, so make sure to ask your favorite AI coding tool about those. That’s all I have for you today. Keep hacking! I asked Claude about this as well and it told me that it prefers because it’s written in Go (as opposed to Perl) and therefore it’s much faster than .  ↩ Of course, I didn’t really have it installed - I only thought I did, otherwise Claude wouldn’t have suggested it. (I switch between computers and my setup on all of them is not exactly the same)  ↩ I asked Claude about this as well and it told me that it prefers because it’s written in Go (as opposed to Perl) and therefore it’s much faster than .  ↩ Of course, I didn’t really have it installed - I only thought I did, otherwise Claude wouldn’t have suggested it. (I switch between computers and my setup on all of them is not exactly the same)  ↩

1 views
Carlos Becker 1 weeks ago

How GoReleaser strengthened security through GitHub's Secure Open Source Fund

GoReleaser builds and ships release artifacts for thousands of projects, making it a high-value supply-chain target. That’s why we were thrilled to be selected for the third session of the GitHub Secure Open Source Fund .

0 views
Danny McClelland 1 weeks ago

Running My Own XMPP Server

About a year ago I moved my personal messaging to Signal as part of a broader push to take ownership of my digital life. That went well. Most of my contacts made the switch, and I’m now at roughly 95% Signal for day-to-day conversations. But Signal is still one company running one service. If they shut down tomorrow or change direction, I’m back to square one. XMPP fixes that. It’s federated, meaning your server talks to other XMPP servers automatically and you’re never locked into a single provider. Your messages live on your hardware. The protocol has been around since 1999 and it’s not going anywhere. I’d tried XMPP years ago and bounced off it, but the clients have come a long way since then. Monal and Conversations are genuinely nice to use now. This post covers everything I did to get a fully working XMPP server running with Prosody in Docker, from DNS records through to voice calls. XMPP uses SRV records to let clients and other servers find yours. You’ll need these in your DNS: Port 5222 is for client connections, 5269 is for server-to-server federation. You’ll also want an A record pointing to your server’s IP. If you want HTTP file uploads (I’d recommend it), add a CNAME or A record for pointing to the same server. Same for if you want group chats with a clean subdomain, though Prosody handles this internally either way. Prosody won’t start without certificates. I use Let’s Encrypt with the Cloudflare DNS challenge so I don’t need to expose port 80: The file contains your API token: After certbot runs, fix the permissions so Prosody can read the certs: Set up a cron to renew monthly: Two ports exposed: 5222 for clients, 5269 for federation. The data volume holds user accounts and message archives. Config and certs are mounted read-only. This is the core of it. I’ll walk through the key sections rather than dumping the whole file. Prosody is modular. My module list: The ones I found matter most for a good mobile experience: syncs messages across all your devices instead of delivering to whichever one happened to be online. (Stream Management) handles flaky connections gracefully, so messages aren’t lost when your phone briefly drops signal. enables push notifications so mobile clients don’t need a persistent connection, which is essential for battery life. And (Message Archive Management) stores history server-side for search and cross-device sync. All connections are encrypted and registration is disabled since I create accounts manually with . I’ve enabled , which means Prosody will reject connections from servers with self-signed or misconfigured certificates. You’ll lose federation with some poorly configured servers, but if you’re self-hosting for privacy reasons it doesn’t make much sense to relax authentication for other people’s mistakes. TLS encrypts connections in transit, but the server itself can still read your messages. If you’re self-hosting, that means you’re trusting yourself, which is fine. But if other people use your server, or if you just want the belt-and-braces approach, OMEMO adds end-to-end encryption so that not even the server operator can read message content. OMEMO is built on the same encryption that Signal uses, so I’m comfortable trusting it. There’s nothing to configure on the server side either. OMEMO is handled entirely by the clients. Monal, Conversations, and Gajim all support it, and in most cases it’s enabled by default for new conversations. I’d recommend turning it on for everything and leaving it on. Messages are kept for a year and archiving is on by default. Clients can opt out per-conversation if they want. Prosody serves HTTP on port 5280 internally. I leave HTTPS to my reverse proxy (Caddy), which handles TLS termination. The tells Prosody what URL to hand clients when they upload files. The MUC (Multi-User Chat) component gives you group chats with message history via . I restrict room creation to local users so random federated accounts can’t spin up rooms on my server. The file share component handles image and file uploads. A 10 MB limit and 30-day expiry keeps disk usage under control. Prosody’s HTTP port needs to be reachable from the internet for file uploads to work. I use Caddy: When a client sends an image, Prosody hands it a URL like and the receiving client fetches it over HTTPS. With registration disabled, accounts are created from the command line: It prompts for a password. Done. Log in from any XMPP client. Open the XMPP ports: Port 80/443 for the reverse proxy if you haven’t already. If your server is behind a router, forward 5222 and 5269. Text and file sharing work at this point. Voice and video calls need one more piece: a TURN/STUN server. Without it, clients behind NAT can’t establish direct media connections. I run coturn alongside Prosody. The two share a secret, and Prosody generates temporary credentials for clients automatically. Generate a shared secret: The coturn : It runs with because TURN needs real network interfaces to handle NAT traversal. Docker’s port mapping breaks this. If your server is behind NAT, add: Then tell Prosody about it. Add to your modules, and inside the block: Open the firewall ports: Verify with . On iOS I went with Monal , which is open source and supports all the modern XEPs. Push notifications work well. On Android, Conversations seems to be the go-to. On desktop, Gajim covers Linux and Windows, and Monal has a macOS build. All of them support OMEMO encryption, file sharing, group chats, and voice/video calls. Prosody has solid built-in diagnostics: This checks DNS records, TLS certificates, connectivity, and module configuration. Fix anything it flags. The error messages are genuinely helpful. The XMPP Compliance Tester is worth running too. Mine scored above 90% after getting the config right. The whole setup runs in two small Docker containers and a reverse proxy entry. Prosody, file uploads, message archive, push notifications, group chats, voice calls. I still use Signal for most day-to-day conversations and I’m not planning to stop. But having my own XMPP server means I’m not entirely dependent on any single service. I can message anyone on any XMPP server, not just people who signed up to the same one. It’s a nice fallback to have. If you’re already running Docker on a server somewhere, it’s a good weekend project. A server with Docker and Docker Compose A domain you control TLS certificates (Let’s Encrypt works well)

0 views
Stone Tools 2 weeks ago

dBASE on the Kaypro II

The world that might have been has been discussed at length. In one possible world, Gary Kildall's CP/M operating system was chosen over MS-DOS to drive IBM's then-new "Personal Computer." As such, Bill Gates's hegemony over the trajectory of computing history never happened. Kildall wasn't constantly debunking the myth of an airplane joyride which denied him Microsoft-levels of industry dominance. Summarily, he'd likely be alive and innovating the industry to this day. Kildall's story is pitched as a "butterfly flaps its wings" inflection point that changed computing history. The truth is, of course, there were many points along our timeline which led to Kildall's fade and untimely death. Rather, I'd like to champion what Kildall did . Kildall did co-host Computer Chronicles with Stewart Chiefet for seven years. Kildall did create the first CD-ROM encyclopedia. Kildall did develop (and coin the term for) what we know today as the BIOS. Kildall did create CP/M, the first wide-spread, mass-market, portable operating system for microcomputers, possible because of said BIOS. CP/M did dominate the business landscape until the DOS era, with 20,000+ software titles in its library. Kildall did sell his company, Digital Research Inc., to Novell for US $120M. Kildall did good . Systems built to run Kildall's CP/M were prevalent, all built around the same 8-bit limits: an 8080 or Z80 processor and up to 64KB RAM. The Osborne 1, a 25lb (11kg) "portable" which sold for $1795 ($6300 in 2026), was the talk of the West Coast Computer Faire in 1981. The price was sweet, considering it came bundled with MSRP $1500 in software, including Wordstar and Supercalc . Andy Kay's company, Non-Linear Systems, debuted the Kaypro II (the "I" only existed in prototype form) the following year at $1595, $200 less (and four pounds heavier) than the Osborne. Though slower than an Osborne, it arguably made it easier to do actual work, with a significantly larger screen and beefier floppy disk capacity. Within the major operating system of its day, on popular hardware of its day, ran the utterly dominant relational database software of its day. PC Magazine , February 1984, said, "Independent industry watchers estimate that dBASE II enjoys 70 percent of the market for microcomputer database managers." Similar to past subjects HyperCard and Scala Multimedia , Wayne Ratcliff's dBASE II was an industry unto itself, not just for data-management, but for programmability, a legacy which lives on today as xBase. Strangely enough, dBASE also decided to attach "II" to its first release; a marketing maneuver to make the product appear more advanced and stable at launch. I'm sure the popularity of the Apple II had nothing to do with anyone's coincidentally similar roman numeral naming scheme whatsoever. Written in assembly, dBASE II squeezed maximum performance out of minimal hardware specs. This is my first time using both CP/M and dBASE. Let's see what made this such a power couple. I'm putting on my tan suit and wide brown tie for this one. As the owner of COMPUTRON/X, a software retail shop, I'm in Serious Businessman Mode™. I need to get inventory under control, snake the employee toilet, do profit projections, and polish a mind-boggling amount of glass and chrome. For now, I'll start with inventory and pop in this laserdisc to begin my dBASE journey. While the video is technically for 16-bit dBASE III , our host, Gentry Lee of Jet Propulsion Laboratory, assures us that 8-bit dBASE II users can do everything we see demonstrated, with a few interface differences. This is Gail Fisher, a smarty pants who thinks she's better than me. Tony Lima, in his book dBASE II for Beginners , concurs with the assessment of dBASE II and III 's differences being mostly superficial. Lima's book is pretty good, but I'm also going through Mastering dBASE II The Easy Way , by Paul W. Heiser, the official Kaypro dBASE II Manual, and dBase II for the First Time User by Alan Freedman. That last one is nicely organized by common tasks a dBASE user would want to do, like "Changing Your Data" and "Modifying Your Record Structure." I find I return to Freedman's book often. As I understand my time with CP/M, making custom bootable diskettes was the common practice. dBASE II is no different, and outright encourages this, lest we risk losing US$2000 (in 2026 dollars) in software. Being of its time and place in computing history, dBASE uses the expected UI. You know it, you love it, it's "a blinking cursor," here called "the dot prompt." While in-program is available, going through the video, books, and manual is a must. dBASE pitches the dot prompt as a simple, English language interface to the program. for example sets the default save drive to the B: drive. You could never intuit that by what it says, nor guess that it even needs to be done, but when you know how it works, it's simple to remember. It's English only in the sense that English-like words are strung together in English-like order. That said, I kind of like it? creates a new database, prompting first for a database name, then dropping me into a text entry prompt to start defining fields. This is a nice opportunity for me to feign anger at The Fishers, the family from the training video. Fancy-pants dBASE III has a more user-friendly entry mode, which requires no memorization of field input parameters. Prompts and on-screen help walk Gail through the process. In dBASE II , a field is defined by a raw, comma-delimited string. Field definitions must be entered in the order indicated on-screen. is the data type for the field, as string, number, or boolean. This is set by a one-letter code which will never be revealed at any time, even when it complains that I've used an invalid code. Remind me to dog-ear that page of the manual. For my store, I'm scouring for games released for CP/M. Poking through Moby Games digs up roughly 30 or so commercial releases, including two within the past five years . Thanks, PunyInform ! My fields are defined thusly, called up for review by the simple command. The most frustrating part about examining database software is that it doesn't do anything useful until I've entered a bunch of data. At this stage in my learning, this is strictly a manual process. Speaking frankly, this part blows, but it also blows for Gail Fisher, so my schadenfreude itch is scratched. dBASE does its best to minimize the amount of keyboard shenanigans during this process, and in truth data entry isn't stressful. I can pop through records fairly quickly, if the raw data is before me. The prompt starts at the first field and (not !) moves to the next. If entry to a field uses the entire field length (as defined by me when setting up the fields earlier), the cursor automatically jumps to the next field with a PC-speaker beep. I guess dBASE is trying to "help," but when touch typing I'm looking at my data source, not the screen. I don't know when I'm about to hit the end of a field, so I'm never prepared when it switches input fields and makes that ugly beep. More jarring is that if the final field of a record is completely filled, the cursor "helpfully" jumps to the beginning of a new record instantly, with no opportunity to read or correct the data I just input. It's never not annoying. Gail doesn't have these issues with dBASE III and her daughter just made dinner for her. Well, I can microwave a burrito as well as anyone so I'm not jealous . I'm not. In defining the fields, I have already made two mistakes. First, I wanted to enter the critic score as a decimal value so I could get the average. Number fields, like all fields, have a "width" (the maximum number of characters/bytes to allocate to the field), but also a "decimal places" value and as I type these very words I see now my mistake. Rubber ducking works . I tricked myself into thinking "width" was for the integer part, and "decimal places" was appended to that. I see now that, like character fields, I need to think of the entire maximum possible number as being the "width." Suppose in a value we expect to record . There are 2 decimal places, and a decimal point, and a leading 0, and potentially a sign, as or . So that means the "width" should be 5, with 2 "decimal places" (of those 5). Though I'm cosplaying as a store owner, I'm apparently cosplaying as a store owner that sucks! I didn't once considered pricing! Gah, Gail is so much better at business than I am! Time to get "sorta good." Toward that end, I have my to-do list after a first pass through data entry. Modifying dBASE "structures" (the field/type definitions) can be risky business. If there is no data yet, feel free to change whatever you want. If there is pre-existing data, watch out. will at least do the common decency of warning you about the pile you're about to step into. Modifying a database structure is essentially verboten, rather we must juggle files to effect a structure change. dBASE let's us have two active files, called "work areas," open simultaneously: a and a . Modifications to these are read from or written to disk in the moment; 64K can't handle much live data. It's not quite "virtual memory" but it makes the best of a tight situation. When wanting to change data in existing records, the command sounds like a good choice, but actually winds up being more useful. will focus in on specified fields for immediate editing across all records. It's simple to through fields making changes. I could to edit everything at once, but I'm finding it safer while learning to make small incremental changes or risk losing a large body of work. Make a targeted change, save, make another change, save, etc. 0:00 / 0:03 1× I laughed every time Gentry Lee showed up, like he's living with The Fishers as an invisible house gremlin. They never acknowledge his presence, but later he eats their salad! Being a novice at dBASE is a little dangerous, and MAME has its own pitfalls. I have been conditioned over time to when I want to "back out" of a process. This shuts down MAME instantly. When it happens, I swear The Fishers are mocking me, just on the edge of my peripheral vision, while Gentry Lee helps himself to my tuna casserole. dBASE is a relational database. Well, let's be less generous and call it "relational-ish." The relational model of data was defined by Edgar F. Codd in 1969 where "relation is used here in its accepted mathematical sense." It's all set theory stuff; way over my head. Skimming past the nerd junk, in that paper he defines our go-to relationship of interest: the join. As a relational database, dBASE keeps its data arranged VisiCalc style, in rows and columns. So long as two databases have a field in common, which is defined, named, and used identically in both , the two can be "joined" into a third, new database. I've created a mini database of developer phone numbers so I can call and yell at them for bugs and subsequent lost sales. I haven't yet built up the grin-and-bear-it temperament Gail possesses toward Amanda Covington. Heads will roll! You hear me, Lebling? Blank?! 64K (less CP/M and dBASE resources) isn't enough to do an in-memory join. Rather, joining creates and writes a completely new database to disk which is the union of two databases. The implication being you must have space on disk to hold both original databases as well as the newly joined database, and also the new database cannot exceed dBASE 's 65,535 record limit after joining. In the above , means and means , so we can precisely specify fields and their work area of origin. This is more useful for doing calculations at time, like to join only records where deletes specific records, if we know the record number, like . Commands in dBASE stack, so a query can define the target for a command, as one would hope and expect in 2026. Comparisons and sub-strings can be used as well. So, rather than deleting "Infocom, Inc." we could: The command looks for the left-hand string as a case-sensitive sub-string in the right-hand string. We can be a little flexible in how data may have been input, getting around case sensitivity through booleans. Yes, we have booleans! Wait, why am I deleting any Infocom games? I love those! What was I thinking?! Once everything is marked for deletion, that's all it is: marked for deletion. It's still in the database, and on disk, until we do real-deal, non-reversible, don't-forget-undo-doesn't-exist-in-1982, destruction with . Until now, I've been using the command as a kind of ad-hoc search mechanism. It goes through every record, in sequence, finding record matches. Records have positions in the database file, and dBASE is silently keeping track of a "record pointer" at all times. This represents "the current record" and commands without a query will be applied to the currently pointed record. Typing in a number at the dot prompt moves the pointer to that record. That moves me to record #3 and display its contents. When I don't know which record has what I want, will move the pointer to the first match it finds. At this point I could that record, or to see a list of records from the located record onward. Depending on the order of the records, that may or may not be useful. Right now, the order is just "the order I typed them into the system." We need to teach dBASE different orders of interest to a stripmall retail store. While the modern reaction would be to use the command, dBASE's Sort can only create entirely new database files on disk, sorted by the desired criteria. Sort a couple of times on a large data set and soon you'll find yourself hoarding the last of new-old 5 1/4" floppy disk stock from OfficeMax, or being very careful about deleting intermediary sort results. SQL brainiacs have a solution to our problem, which dBASE can also do. An "index" is appropriate for fast lookups on our columnar data. We can index on one or more fields, remapping records to the sort order of our heart's desire. Only one index can be used at a time, but a single index can be defined against multiple fields. It's easier to show you. When I set the index to "devs" and , that sets the record pointer to the first record which matches my find. I happen to know I have seven Infocom games, so I can for fields of interest. Both indexes group Infocom games together as a logical block, but within that block Publisher order is different. Don't get confused, the actual order of files in the database is betrayed by the record number. Notice they are neither contiguous nor necessarily sequential. would rearrange them into strict numerical record order. An Index only relates to the current state of our data, so if any edits occur we need to rebuild those indexes. Please, contain your excitement. Munging data is great, but I want to understand my data. Let's suppose I need the average rating of the games I sell. I'll first need a count of all games whose rating is not zero (i.e. games that actually have a rating), then I'll need a summation of those ratings. Divide those and I'll have the average. does what it says. only works on numeric fields, and also does what it says. With those, I basically have what I need. Like deletion, we can use queries as parameters for these commands. dBASE has basic math functions, and calculated values can be stored in its 64 "memory variables." Like a programming language, named variables can be referenced by name in further calculations. Many functions let us append a clause which shoves a query result into a memory variable, though array results cannot be memorized this way. shoves arbitrary values into memory, like or . As you can see in the screenshot above, the rating of CP/M games is (of 100). Higher than I expected, to be perfectly honest. As proprietor of a hot (power of positive thinking!) software retail store, I'd like to know how much profit I'll make if I sold everything I have in stock. I need to calculate, per-record, the following but this requires stepping through records and keeping a running tally. I sure hope the next section explains how to do that! Flipping through the 1,000 pages of Kaypro Software Directory 1984 , we can see the system, and CP/M by extension, was not lacking for software. Interestingly, quite a lot was written in and for dBASE II, bespoke database solutions which sold for substantially more than dBASE itself. Shakespeare wrote, "The first thing we do, let's kill all the lawyers." Judging from these prices, the first thing we should do is shake them down for their lunch money. In the HyperCard article I noted how an entire sub-industry sprung up in its wake, empowering users who would never consider themselves programmers to pick up the development reigns. dBASE paved the way for HyperCard in that regard. As Jean-Pierre Martel noted , "Because its programming language was so easy to learn, millions of people were dBASE programmers without knowing it... dBASE brought programming power to the masses." dBASE programs are written as procedural routines called Commands, or .CMD files. dBASE helpfully includes a built-in (stripped down) text editor for writing these, though any text editor will work. Once written, a .CMD file like can be invoked by . As Martel said, I seem to have become a dBASE programmer without really trying. Everything I've learned so far hasn't just been dot prompt commands, it has all been valid dBASE code. A command at the dot prompt is really just a one-line program. Cool beans! Some extra syntax for the purpose of development include: With these tools, designing menus which add a veneer of approachability to a dBASE database are trivial to create. Commands are interpreted, not compiled (that would come later), so how were these solutions sold to lawyers without bundling a full copy of dBASE with every Command file? For a while dBASE II was simply a requirement to use after-market dBASE solutions. The 1983 release of dBASE Runtime changed that, letting a user run a file, but not edit it. A Command file bundled with Runtime was essentially transformed into a standalone application. Knowing this, we're now ready to charge 2026 US$10,000 per seat for case management and tracking systems for attorneys. Hey, look at that, this section did help me with my profit calculation troubles. I can write a Command file and bask in the glow of COMPUTRON/X's shining, profitable future. During the 8 -> 16-bit era bridge, new hardware often went underutilized as developers came to grips with what the new tools could do. Famously, Visicalc 's first foray onto 16-bit systems didn't leverage any of the expanded RAM on the IBM-PC and intentionally kept all known bugs from the 8-bit Apple II version. The word "stop gap" comes to mind. Corporate America couldn't just wait around for good software to arrive. CP/M compatibility add-ons were a relatively inexpensive way to gain instant access to thousands of battle-tested business software titles. Even a lowly Coleco ADAM could, theoretically, run WordStar and Infocom games, the thought of which kept me warm at night as I suffered through an inferior Dragon's Lair adaptation. They promised a laserdisc attachment! For US$600 in 1982 ($2,000 in 2026) your new-fangled 16-bit IBM-PC could relive the good old days of 8-bit CP/M-80. Plug in XEDEX's "Baby Blue" ISA card with its Z80B CPU and 64K of RAM and the world is your slowly decaying oyster. That RAM is also accessible in 16-bit DOS, serving dual-purpose as a memory expansion for only $40 more than IBM's own bare bones 64K board. PC Magazine' s February 1982 review seemed open to the idea of the card, but was skeptical it had long-term value. XEDEX suggested the card could someday be used as a secondary processor, offloading tasks from the primary CPU to the Z80, but never followed through on that threat, as far as I could find. Own anApple II with an 8-bit 6502 CPU but still have 8-bit Z80 envy? Microsoft offered a Z80 daughter-card with 64K RAM for US$399 in 1981 ($1,413 in 2026). It doesn't provide the 80-column display you need to really make use of CP/M software, but is compatible with such add-ons. It was Bill Gates's relationship with Gary Kildall as a major buyer of CP/M for this very card that started the whole ball rolling with IBM, Gates's purchase of QDOS, and the rise of Microsoft. A 16K expansion option could combine with the Apple II's built-in 48K memory, to get about 64K for CP/M usage. BYTE Magazine 's November 1981 review raved, "Because of the flexibility it offers Apple users, I consider the Softcard an excellent buy." Good to know! How does one add a Z80 processor to a system with no expansion slots? Shove a Z80 computer into a cartridge and call it a day, apparently. This interesting, but limited, footnote in CP/M history does what it says, even if it doesn't do it well. Compute!'s Gazette wrote, "The 64 does not make a great CP/M computer. To get around memory limitations, CP/M resorts to intensive disk access. At the speed of the 1541, this makes programs run quite slowly," Even worse for CP/M users is that the slow 1541 can't read CP/M disks. Even if it could, you're stuck in 40-column mode. How were users expected to get CP/M software loaded? We'll circle back to that a little later. At any rate, Commodore offered customers an alternative solution. Where it's older brother had to make do with a cartridge add-on, the C128 takes a different approach. To maintain backward compatible with the C64 it includes a 6510 compatible processor, the 8502. It also wants to be CP/M compatible, so it needs a Z80 processor. What to do, what to do? Maybe they could put both processors into the unit? Is that allowed? Could they do that? They could, so they did. CP/M came bundled with the system, which has a native 80-column display in CP/M mode. It is ready to go with the newer, re-programmable 1571 floppy drive. Unfortunately, its slow bus speed forces the Z80 to run at only 2MHz, slower even than a Kaypro II. Compute!'s Gazette said in their April 1985 issue, "CP/M may make the Commodore 128 a bargain buy for small businesses. The price of the Commodore 128 with the 1571 disk drive is competitive with the IBM PCjr." I predict rough times ahead for the PCjr if that's true! Atari peripherals have adorable industrial design, but were quite expensive thanks to a strange system design decision. The 8-bit system's nonstandard serial bus necessitated specialized data encoding/decoding hardware inside each peripheral, driving up unit costs. For example, the Atari 910 5 1/4" floppy drive cost $500 in 1983 (almost $2,000 in 2026) thanks to that special hardware, yet only stored a paltry 90K per disk. SWP straightened out the Atari peripheral scene with the ATR8000. Shenanigans with special controller hardware are eliminated, opening up a world of cheaper, standardized floppy drives of all sizes and capacities. It also accepts Centronics parallel and RS-232C serial devices, making tons of printers, modems, and more compatible with the Atari. The device also includes a 16K print buffer and the ability to attach up to four floppy drives without additional controller board purchases. A base ATR8000 can replace a whole stack of expensive Atari-branded add-ons, while being more flexible and performant. The saying goes, "Cheaper, better, faster. Pick any two." The ATR8000 is that rare device which delivered all three. Now, upgrade that box with its CP/M compatibility option, adding a Z80 and 64K, and you've basically bought a second computer. When plugged into the Atari, the Atari functions as a remote terminal into the unit, using whatever 40/80-column display adapter you have connected. It could also apparently function standalone, accessible through any terminal, no Atari needed. That isn't even its final form. The Co-Power-88 is a 128K or 256K PC-compatible add-on to the Z80 CP/M board. When booted into the Z80, that extra RAM can be used as a RAM disk to make CP/M fly. When booted into the 8088, it's a full-on PC running DOS or CP/M-86. Tricked out, this eight pound box would set you back US$1000 in 1984 ($3,000 in 2026), but it should be obvious why this is a coveted piece of kit for the Atari faithful to this day. For UK£399 in 1985 (£1288 in 2026; US$1750) Acorn offered a Z80 with dedicated 64K of RAM. According to the manual, the Z80 handles the CP/M software, while the 6502 in the base unit handles floppies and printers, freeing up CP/M RAM in the process. Plugged into the side of the BBC Micro, the manual suggests desk space clearance of 5 ft wide and 2 1/2 feet deep. My god. Acorn User June 1984 declared, "To sum up, Acorn has put together an excellent and versatile system that has something for everyone." I'd like to note that glowing review was almost exclusively thanks to the bundled CP/M productivity software suite. Their evaluation didn't seem to try loading off-the-shelf software, which caused me to narrow my eyes, and stroke my chin in cynical suspicion. Flip through the manual to find out about obtaining additional software, and it gets decidedly vague. "You’ll find a large and growing selection available for your Z80 personal computer, including a special series of products that will work in parallel with the software in your Z80 pack." Like the C128, the Coleco ADAM was a Z80 native machine so CP/M can work without much fuss, though the box does proclaim "Made especially for ADAM!" Since we don't have to add hardware (well, we need a floppy; the ADAM only shipped with a high-speed cassette drive), we can jump into the ecosystem for about US$65 in 1985 ($200 in 2026). Like other CP/M solutions, the ADAM really needed an 80-column adapter, something Coleco promised but never delivered. Like Dragon's Lair on laserdisc! As it stands, CP/M scrolls horizontally to display all 80 columns. This version adds ADAM-style UI for its quaint(?) roman numeral function keys. OK, CP/M is running! Now what? To be honest, I've been toying with you this whole time, dangling the catnip of CP/M compatibility. It's time to come clean and admit the dark side of these add-on solutions. There ain't no software! Even when the CPU and CP/M version were technically compatible, floppy disc format was the sticking point for getting software to run any given machine. For example, the catalog for Kaypro software in 1984 is 896 pages long. That is all CP/M software and all theoretically compatible with a BBC Micro running CP/M. However, within that catalog, everything shipped expressly on Kaypro compatible floppy discs. Do you think a Coleco ADAM floppy drive can read Kaypro discs? Would you be even the tiniest bit shocked to learn it cannot? Kaypro enthusiast magazine PRO illustrates the issue facing consumers back then. Let's check in on the Morrow Designs (founded by Computer Chronicles sometimes co-host George Morrow!) CP/M system owners. How do they fare? OK then, what about that Baby Blue from earlier? The Microsoft Softcard must surely have figured something out. The Apple II was, according to Practical Computing , "the most widespread CP/M system" of its day. Almost every product faced the same challenge . On any given CP/M-80 software disk, the byte code is compatible with your Z8o, if your floppy drive can read the diskette. You couldn't just buy a random CP/M disk, throw it into a random CP/M system, and expect it to work, which would have been a crushing blow to young me hoping to play Planetfall on the ADAM. So what could be done? There were a few options, none of them particularly simple or straightforward, especially to those who weren't technically-minded. Some places offered transfer services. XEDEX, the makers of Baby Blue, would do it for $100 per disk . I saw another listing for a similar service (different machine) at $10 per disk. Others sold the software pre-transferred, as noted on a Coleco ADAM service flyer. A few software solutions existed, including Baby Blue's own Convert program, which shipped with their card and "supports bidirectional file transfer between PC-DOS and popular CP/M disk formats." They also had the Baby Blue Conversion Software which used emulation to "turn CP/M-80 programs into PC-DOS programs for fast, efficient execution on Baby Blue II." Xeno-Copy, by Vertex Systems, could copy from over 40+ disk formats onto PC-DOS for US$99.50 ($313 in 2026); their Plus version promised cross-format read/write capabilities. Notably, Apple, Commodore, Apricot, and other big names are missing from their compatibility list. The Kermit protocol , once installed onto a CP/M system disk, could handle cross-platform serial transfers, assuming you had the additional hardware necessary. "CP/M machines use many different floppy disk formats, which means that one machine often cannot read disks from another CP/M machine, and Kermit is used as part of a process to transfer applications and data between CP/M machines and other machines with different operating systems." The Catch-22 of it all is that you have to get Kermit onto your CP/M disk in the first place. Hand-coding a bare-bones Kermit protocol (CP/M ships with an assembler) for the purposes of getting "real" Kermit onto your system so you could then transfer the actual software you wanted in the first place, was a trick published in the Kermit-80 documentation . Of course, this all assumes you know someone with the proper CP/M setup to help; basically, you're going to need to make friends. Talk to your computer dealer, or better yet, get involved in a local CP/M User's Group. It takes a village to move Wordstar onto a C64. I really enjoyed my time learning dBASE II and am heartened by the consistency of its commands and the clean interaction between them. When I realized that I had accidentally learned how to program dBASE , that was a great feeling. What I expected to be a steep learning curve wasn't "steep" per se, but rather just intimidating. That simple, blinking cursor, can feel quite daunting at the first step, but each new command I learned followed a consistent pattern. Soon enough, simple tools became force multipliers for later tools. The more I used it, the more I liked it. dBASE II is uninviting, but good. On top of that, getting data out into the real world is simple, as you'll see below in "Sharpening the Stone." I'm not locked in. So what keeps me from being super enthusiastic about the experience? It is CP/M-80 which gives me pause. The 64K memory restriction, disk format shenanigans, and floppy disk juggling honestly push me away from that world except strictly for historical investigations. Speaking frankly, I don't care for it. CP/M-86 running dBASE III+ could probably win me over, though I would probably try DR-DOS instead. Memory constraints would be essentially erased, DOSBox-X is drag-and-drop trivial to move files in and out of the system, and dBASE III+ is more powerful while also being more user-friendly. Combine that with Clipper , which can compile dBASE applications into standalone .exe files, and there's powerful utility to be had . By the way, did you know dBASE is still alive ? Maybe. Kinda? Hard to say. The latest version is dBASE 2019 (not a typo!), but the site is unmaintained and my appeal to their LinkedIn for a demo has gone unanswered. Its owner, dBase LTD, sells dBASE Classic which is dBASE V for DOS running in DOSBox, a confession they know they lost the plot, I'd humbly suggest. An ignominious end to a venerable classic. Ways to improve the experience, notable deficiencies, workarounds, and notes about incorporating the software into modern workflows (if possible). When working with CP/M disk images, get to know cpmtools . This is a set of command line utilities for creating, viewing, and modifying CP/M disk images. The tools mostly align with Unix commands, prefixed with Those are the commands I wound up using with regularity. If your system of choice is a "weirdo system" you may be restricted in your disk image/formatting choices; these instructions may be of limited or no help. knows about Kaypro II disk layouts via diskdefs. This Github fork makes it easy to browse supported types. Here's what I did. Now that you can pull data out of CP/M, here's how to make use of it. Kaypro II emulation running in MAME. Default setup includes Dual floppies Z80 CPU at 2.4MHz dBase II v2.4 See "Sharpening the Stone" at the end of this post for how to get this going. Personally, I found this to be a tricky process to learn. Change the of the rating field and add in that data. Add pricing fields and related data. Add more games. and allow decision branching does iterations and will grab a character or string from the user prints text to screen at a specific character position and give control over system memory will run an assembly routine at a known memory location For this article I specifically picked a period-authentic combo of Kaypro II + CP/M 2.2 + dBASE II 2.4. You don't have to suffer my pain! CP/M-86 and dBASE III+ running in a more feature-rich emulator would be a better choice for digging into non-trivial projects. I'm cold on MAME for computer emulation, except in the sense that in this case it was the fastest option for spinning up my chosen tools. It works, and that's all I can say that I enjoyed. That's not nothing! I find I prefer the robust settings offered in products like WinUAE, Virtual ADAM, VICE , and others. Emulators with in-built disk tools are a luxury I have become addicted to. MAME's interface is an inelegant way to manage hardware configurations and disk swapping. MAME has no printer emulation, which I like to use for a more holistic retro computing experience. Getting a working, trouble-free copy of dBASE II onto a Kaypro II compatible disk image was a non-trivial task. It's easier now that I know the situation, but it took some cajoling. I had to create new, blank disks, and copy CP/M and dBASE over from other disk images. Look below under "Getting Your Data into the Real World" to learn about and how it fits into the process. Be careful of modern keyboard conventions, especially wanting to hit to cancel commands. In MAME this will hard quit the emulator with no warning! Exported data exhibited strange artifacts: The big one: it didn't export any "logical" (boolean) field values from my database. It just left that field blank on all records. Field names are not exported. Garbage data found after the last record; records imported fine. On Linux and Windows (via WSL) install thusly : view the contents of a CP/M disk image. Use the flag to tell it the format of the disk, like for the Kaypro II. : format a disk image with a CP/M file system : copy files to/from other disk or to the host operating system : remove files from a CP/M disk image : for making new, blank disk image files (still needs to be formatted) : makes a blank disk image to single-sided, double-density specification : formats that blank image for the Kaypro II : copies "DBASE.COM" from the current directory of the host operating system into the Kaypro II disk image. : displays the contents of the disk : copies "FILE.TXT" from the disk image into the current directory of the host operating system (i.e. ) dBASE has built-in exporting functionality, so long as you use the extension when saving ( in dBASE lingo). That creates a bog-standard ASCII text file, each record on its own line, comma-delimited (and ONLY comma-delimited). It is not Y2K compatible, if you're hoping to record today's date in a field. I tackled this a bit in the Superbase post . It is probably possible to hack up a Command file to work around this issue, since dates are just strings in dBASE . dBASE II doesn't offer the relational robustness of SQL. Many missing, useful tools could be built in the xBase programming language. It would be significant work in some cases; maybe not worth it or consider if you can do without those. Your needs may exceed what CP/M-80 hardware can support; its 8-bit nature is a limiting factor in and of itself. If you have big plans , consider dBASE III+ on DOS to stretch your legs. (I read dBASE IV sucks) The user interface helps at times, and is opaque at other times. This can be part of the fun in using these older systems, mastering esoterica for esoterica's sake, but may be a bridge too far for serious work of real value. Of course, when discussing older machines we are almost always excluding non-English speakers thanks to the limitations of ASCII. The world just wasn't as well-connected at the time.

0 views
Playtank 2 weeks ago

The Systemic Pitch

The past few years prove that there is a market for premium systemic singleplayer games. Few have listened to us (developers) when we tried to pitch such games for the past decade. Or ever. This isn’t because there is some kind of conspiracy against systemic games. Not at all. Systemic designs are very hard to sell. Not just to publishers or to customers; systemic designs are hard to sell even to your own team. We get caught by our own excitement, or we try to sell technology or tools as if they are designs when they are really not. The fleeting nature of systemic design feels like losing control, or like a potentially bloated mess of optional sandbox content that we have to make for the simulation to be complete but that will add little for the average player. In this post, I try to address pitching systemic games, based on my own mistakes. There’s a wealth of literature on how to sell something. Including how to make a good sales pitch. For this post, we’ll stick to a very general high level idea of what pitching needs to achieve, and leave all of that expertise to the real experts. One great book you can take a look at is the book Pitch Anything, by Oren Klaff. For this post, let’s assume a pitch needs to do three things: The origins of AAA (or “triple-A”) is from credit ratings. It stands for a rating of low risk, high reward. Or, in the terms of investment banks, “exceptional creditworthiness with minimal risk of default.” It was used in early game development, including pitching, to signify projects that were safe bets with high financial potential. Today, AAA is used to describe anything from team size to budget size. Like many of the terms we use in game development, it’s become almost meaningless, or at least the interpretations have become too varied for consistent use. But these origins are relevant. When you pitch something, the value proposition still needs as small a risk as possible with as big of a potential reward as possible. Pitching is to ask for something. It may be funding to get your game across the finish line, developer buy-in to make the next feature, or something else. Be specific with what you are asking for and you will have a less frustrating conversation. When you pitch, you must offer something to the people you are pitching to. Money, ownership, future commitments. It’s not enough to offer them a potentially great game, you need to show how it’s more than the sum of its parts. Game development comes with many risks that you must address with your pitch. You don’t have to call them out and tell people what your solution will be, but you should consider them and hold yourself accountable for them. The systemic value proposition is extremely tricky. For many external stakeholders it’s not the same as the creative argument or the design paradigms . Many stakeholders want replayable content that’s cheap to make , and will read “systemic” as making content production easier or cheaper. Perhaps using procedural generation techniques to generate multiple levels from a small set of content, thereby not needing as many level designers or prop artists. This is not the same as systemic design at all. Systemic design is about letting go of authorial control and allowing players to have the fun. This almost invariably makes a systemic design sound more expensive to make, since it implies a high degree of freedom and a sandbox nature. If you detect excitement around systemic ideas, be really careful to note what is generating the excitement, or this could be the fundamental misunderstanding you’re experiencing. Anyone can have ideas but everyone can’t turn ideas into games. You must be able to prove why you should be the person making your thing. What to lead your credibility pitch with is tricky. For systemic games, it helps to demonstrate technical expertise immediately. To line up all the key roles that will address the risks you’ve already mentioned and explain how you’ve filled them. Studios may get a lot of attention after releasing games that sold many copies, attracted many concurrent players, or gained high scores in reviews and good media attention. Though this front row attention may be fleeting in the media, it will matter a lot for your credibility to be part of these journeys. People may even talk about the best place to be at a given time. If you’ve mostly worked as a salaried employee, your studio pedigree will be the most important thing you can offer to state your credibility. It also tends to be the first thing many will ask to know. If you worked at particularly big modern studios, you mention which roles you held and not just the name of the studio. This is because if your title was Junior Assistant to the Senior Assistant’s Assistant, your impact was probably quite small, and talking about this studio doesn’t actually provide much credibility. Simple maths. If you have fewer years of experience, you have probably learned less. But the keyword here is “relevant.” If you are pitching a big sprawling open world role-playing game after working on first-person shooters for 15 years, people may believe that you know how to make games, but may be weary that you haven’t made this type of game before. This will then become a risk that you must address. Something that’s almost a joke at this point is to sum up the experience of your team and use that in your pitch. E.g., “250 years of combined gamedev experience.” You can of course do this anyway, but it doesn’t really mean anything. The easiest and most quantifiable way to demonstrate that this isn’t your first rodeo (if it isn’t) is to list the box art for the games you shipped. If this is a very long list, you can stick to the ones that were important or are more likely to be known by the people you are pitching to. Similarly to studio experience, you may want to specify what you did on each game too, but only if it becomes too anonymous otherwise. You shouldn’t turn the credibility element of your pitch into a reason to talk about yourself at length. For an external stakeholder, technology that exists and has been proven through previous game releases is worth a lot more than experimental technology. For this reason, technology becomes part of your credibility. If you come to a pitch and you say you are working on your own engine that will probably be finished a couple of months ahead of release, this will send off warning flags for everyone in the room. But if you say that your team is working with an established third-party engine and you have a working prototype already in place, this will give a much better first impression. It helps to have a team already lined up and waiting for your go-signal. A team shortlist is a list of people who have agreed to let you put their name down for if you find the funds. It’s very rarely treated as a commitment, more a way to lend weight to a pitch. It’s better than saying you’ll start recruiting when you have your funding, but it’s not as good as having people already onboard. The packaging refers to how you communicate value and credibility. There’s no right or wrong way here, but it will matter a lot based on who you are talking to. According to the It Was a Sh!tshow podcast, Futurama spent two to three years in preparation before it was pitched to studios. During that time, they explored characters, key art, technology, and many other things. In game development, we rarely have this room for pitch or concept development. But you do need to prepare as much as you possibly can. You need to figure out the risks, foresee what potential stakeholders will be worried about, and proactively respond allay their concerns. Stories resonate with people. Introductions, reversals, and climaxes. Presenting your pitch as a story doesn’t mean you should lead with your game’s story, it means that your whole pitch should be a story with a proper beginning and end. Start with a bang, end with a call to action. There are some pitfalls you should avoid, however. Don’t ask open-ended rhetorical questions, e.g. “Have you ever thought about why dogs have two ears?!” Because chances are that they only confuse people and don’t actually make them think the way you want. Take charge of the story and walk through your pitch’s narrative beat by beat. Leave nothing to chance. If you want to frame your pitch as a story, use video and visual aids as much as possible and let the story come from you rather than the pitch deck. A strong metaphor can also carry a pitch. In an interview with Designer Notes, games venture capitalist Mitch Lasky talked about his EA pitch using a container to illustrate the benefits of standardisation. Metaphors can of course be traps as well, if they in fact illustrate something you don’t intend, but you’ll figure that out as you work through it. Use a strong metaphor if it fits your whole pitch and doesn’t leave strange questions. A sad fact is that no one wants novelty. Novelty almost always looks like a risk more than a gain. From Railroad Tycoon to The Sims , many of the most groundbreaking games through the years had few fans in management. Similarly, Markus Persson (“Notch”) said that no publisher would’ve cared about Minecraft if he had pitched it to them: it could only happen by selling it directly to players. You can absolutely lead with how your game is different and new, but be aware of this risk. Only focus on novelty if you can incorporate a strong why into your pitch. I have a whole post that laments the use of the word “content.” But it’s enough to say that it’s a word often equated with quantity and used by both developers and consumers. Developers will talk about how much content they offer, while consumers will usually ask for more of it no matter how much is on offer. Most systemic games are not built to funnel “content” to users. Churning out downloadable content (DLC) fits really poorly, and most of the time replayability is a matter of smoke and mirrors. Choosing A instead of B, or approaching through the secret door instead of the main entrance. Functionally, the very same content , but a different experience . Some systemic games manage to pull it off, like Prey and its excellent Mooncrash DLC, but at other times it ends up feeling artificial and a bit forced. Thief wouldn’t benefit from offering you a special gold-lined blackjack, for example. It would only risk diminishing the core experience. Therefore, if you want to offer a systemic game, don’t pitch your game on its quantity of content . People generally use harsher words than good or bad. What often gets lost is the reasons why we think something is good or bad. Particularly when good or bad is applied to specific parts of a game, such as its story or gameplay. If you didn’t like the gameplay, maybe this made you dislike the story. If you really loved the premise, then maybe you felt better about the gameplay than it actually deserved. This means that good or bad is mostly a loud declaration of opinion that muddles any real qualities or faults of the thing being touted. If you disagree with the zeitgeist effectively countless “masterpieces,” the gamer population is quick to call you an idiot. Publishers may not call you an idiot, but you should still avoid calling things good or bad. It may even be that the thing you’re trash-talking is something one of the people you’re pitching to happened to work on. Therefore, avoid value terms and comparisons to other games . There are many words for this. Janky, buggy, glitchy, broken, even unplayable. Some game companies have gained a reputation for their games feeling “janky,” while others may have segments that are more or less so. “Eurojank” is sometimes used as its own term. The issue with this language is that it’s also often applied to complex games. Having to use multiple mapped controls, or using menus in certain ways, will become conversationally equivalent to jank. No matter what you do, do not use these words to describe your own game unless it’s deeply intentional. If you are making the next Goat Simulator or Gang Beasts , then by all means call it janky. You can also acknowledge if your audience calls your concept janky, and run with it, but don’t introduce the term on your own. Don’t talk about your own game as janky, buggy, or messy . In the marketing buildup to the release of No Man’s Sky , if you followed games media to any extent, you would’ve seen Hello Games’ Sean Murray. He was the face and voice for the project and his infectious enthusiasm built a gigaton of hype. But he was also first and foremost a developer. Someone who was enthusiastic not just about what the game actually was but what it could be. A mentality that everyone in game development understands. If you talk about technology and its potential, be careful to not promise more than you can deliver . Maybe it’s because of the clickbait era and the tendency for a lot of people to only read the headlines and not the main points, using examples can actually be a problem. If you describe a general system you have and you then say that it’ll be used to generate something like a metal ball, it’ll then become the Metal Ball System and nothing else to some of the people listening to the pitch. It’s better to provide a framework for your systems and let the audience’s imagination put things together, or you can easily fall into the trap that you need to start defending your example or expand on what makes the example interesting. You don’t want to be put in that spot. Even worse, if you use examples from other games , they may infer different things for the audience than what you had in mind. Avoid using examples, or they may become people’s most concrete takeaways. Let’s get one thing perfectly clear: you can’t convert your own excitement into a signed contract . Excitement is important when you deliver your message, since we’re social beings, and it can definitely affect your believability and how compelling your message is. But in every other way, your excitement is strictly yours. No one cares about your deep lore, the motivations of your characters, the 1,000 years of world history you wrote for your fantasyland, or anything else of that sort, until after they have crossed what I call the excitement threshold . At that point, everyone won’t care, but a subset may suddenly care to the point of obsession. For every player that praises all the well-written logbooks and audio journals, there are nine others who completely ignore them. For every publisher rep you talk to who loves the cool technical solutions you’re talking about, there are nine others who have never seen Visual Studio and will simply not get what you are trying to make. For every complex systemic thing you added to your prototype, there will be someone at the other end of the table who asks why the shadows look wrong or why your prototype doesn’t look as visually polished as the one from some other pitch they just saw. All this and more is bound to happen, and you must learn to really read the room when it comes to which parts of the message to focus on. As an example of the excitement threshold, take a look at the original trailer for the movie Raiders of the Lost Ark . Note how this trailer focuses, not on the Indiana Jones we know today, but on the mystery of the Ark of the Covenant and its terrifying implications in the 1930s. Lost artifacts, Egyptian ruins, and nazis: as pulp as it gets. Compare this to a modern trailer for the same movie . A trailer that focuses clearly on all the character flaws and iconic shenanigans that many of us remember fondly from the original movie. This illustrates the difference excitement makes. In the original trailer, no one cares about the character of Indiana Jones. We don’t know him yet, or why we should care about him. But once the second trailer hits, and the movie is now renamed Indiana Jones and the Raiders of the Lost Ark, it’s all about Dr Jones, his fear of snakes, and those cool scenes that we all remember. This comparison is important, because most of us will be facing what Raiders of the Lost Ark faced: an audience that needs to have something else to latch on to than what you want them to be excited about. An audience that doesn’t know Dr Jones yet and will have to discover him on their own. The same goes for our pitch — you need to treat your audience (the stakeholders) to something that excites them . If you can do that, you’ve won half the battle! Convince the party you are pitching to that a thing is worth making. A value proposition . Also convince them that you are the ones who should make it. Establish your credibility . Cater both messages specifically to the people you are pitching to. The packaging . Financial security: If your project fails miserably, maybe doesn’t even get released, they will usually soak up the loss while you simply move on. This part is rarely said out loud, but it’s a key ask nonetheless. Funding advances : The most obvious ask towards publishers and investors: getting a chunk of change that pays for development. Just make sure to make it realistic and not low- or high-balling your numbers. Say how much you need and why. Commitment: Asking for your team to commit to your project or to key changes. This can sometimes be the hardest pitching you’ll do. Even more so if your team has low morale. Marketing help : Getting help marketing your game. Be specific with this ask, since some partners may only do the minimum if you don’t have a concrete marketing spend that is contractually obligated. Be clear with your ask. It’s not uncommon to match marketing spend with development spend. Technical support: Server architecture, cloud infrastructure, console porting, and other elements of development that are beyond your capabilities as a developer and therefore included in your ask. Cash : Unless you are unable to afford asked rates or you are making a very big ask from a busy partner, you will rarely have to pitch as much if you have cash to spend. But it’s definitely a gain to be considered. If you pay for something upfront, you will rarely have to part with things you’d rather keep. Revenue : There are more ways to share revenue than there are stars in the sky. It may be time-limited or permanent, the percentage may shrink or grow over time, and the share may or may not be taken from one party to compensate another (advance on royalties). If you want to offer revenue share, you should provide a revenue projection based on real data. One that shows how large the earning potential is in multiple scenarios, for example based on number of copies sold. Just be realistic. Equity : Instead of potential future profits you can offer ownership. It can be ownership in your company, or a newly founded company that handles this specific project under mutual conditions with investors. Equity allows someone to have a bigger stake in what you are doing and will of course mean that they get a chunk of future profits also . Just be careful to part with too much equity. Property : You can offer up the intellectual property (IP) you are creating. Your game, its characters and likenesses, assets, etc. Usually including everything related to it, such as potential sequels, merchandise, tie-ins, and more. I’ve been told you shouldn’t accept deals like this today, but it may be on the table and you may not have much authority to say no. Exclusivity : Something that will often be heavily implied anyway, but not always formalised, is exclusivity: that you won’t be launching on more platforms or making more than one game at the same time. This is less relevant today, where platform holders seem less inclined to pay for exclusivity, but depending on who you are talking to, it’s still something worth considering. It can be forever, it can be time-limited, or have other restrictions. But offering exclusivity can be valuable. Delays : explain how you plan to deliver your project on schedule. Almost every game project suffers from some kind of delays (for no good reasons). Sometimes those could’ve been foreseen, planned for, or even mitigated. This is really where your credibility comes in. Convince people why you won’t suffer those delays. Complexity : this is where many systemic designs fail stakeholder scrutiny. They will look or sound complex or complicated, and may imply scale that is not actually required. Inexperience : if there are things in the project that you haven’t done before, or technologies you need to evaluate and research properly, you have to be very transparent about it. If your whole team hasn’t delivered a game before, this is a major risk that you must be able to address. Time constraints : release windows, budget timeframes; there can be multiple reasons why your time is constrained. Perhaps you can’t start fulltime until you get the third programmer onboard, and that won’t happen until six months from now. Bring out a concrete timeline that you can safely commit to. Non-Compliance : the game may become something else than what you agreed on, for creative or financial reasons. Smaller, larger, or styled differently than intended. This is where most creative differences will come from, since many signed deals will be commitments and you’ve just chosen to break them. This is the main reason you’re likely to have milestones and other deliverables along the way, so that any accident about to happen can be course corrected. DOA : the game may be dead on arrival, missing its target audience or simply failing to gain traction against other launches in the same window. In the best of worlds, this doesn’t just kill your studio, but provides at least six months to a year of time where you can do your best to salvage or improve the situation.

0 views
Filippo Valsorda 2 weeks ago

Inspecting the Source of Go Modules

Go has indisputably the best package integrity story of any programming language ecosystem. The Go Checksum Database guarantees that every Go client in the world is using the same source for a given Go module and version, forever. It works despite the decentralized nature of Go modules, which can be fetched directly from their origin based on the import path. (For example, you can fetch v1.2.3 of by cloning the git repository and exporting the v1.2.3 tag. forces this. 1 ) The Checksum Database stores the cryptographic hash of a module version the first time it is used across the ecosystem, and then provides that same checksum to every Go client going forward. If e.g. a git tag were force-pushed or a code host were to try to serve targeted versions to some clients, the tool would notice the mismatch and fail the fetch. This is vastly more practical than requiring module authors to manage keys, but provides comparable security, because the author themselves can verify the checksum in the Checksum Database matches the one they developed. Moreover, the Checksum Database is a transparency log , which prevents even the database operator (i.e. Google) from falsifying or hiding entries. However, any time we read code directly from the code host we introduce a weak link in this chain. For example, there is no guarantee that the code displayed at is the actual contents of from v1.2.3 of module : GitHub allows force-pushing git tags and even built its recommended GitHub Actions workflows on top of mutable tags. Last year this was taken advantage of to make a classic typosquatting attack harder to identify. A fake BoltDB module was published with malicious code , and then innocent code was force-pushed to GitHub. Some commenters described this as exploiting the Go Modules Mirror’s cache, but it is better understood as exploiting the natural lack of verification in the GitHub web interface, which doesn’t show the authentic (and in this case malicious) source of a module version, as used by actual Go tooling. The solution when reviewing modules locally is to use a command like to fetch the correct source. 2 We are also working on a command to verify the contents of a local git repository against the Go Checksum Database, which can also be used by module authors to check that the contents of the Checksum Database are correct. However, pkg.go.dev still links to unverified code hosts, and clicking on pkg.go.dev source links is very convenient. Russ Cox made a simple service to view the source of a Go module at go-mod-viewer.appspot.com . pkg.geomys.dev is a new similar service with optional syntax highlighting, line and line range linking , multiple fonts, automatic dark mode, and a file tree and module versions browser. You can use it manually by replacing with in any pkg.go.dev URL, or you can install the companion browser extension for Chrome and Firefox, which replaces links to code hosts in pkg.go.dev pages with links to pkg.geomys.dev. The service works by making HTTP Range requests directly to the module version’s zip file, and decompressing the file in the browser, without having to fetch the whole archive. Once proxy.golang.org fixes their CORS configuration it will work without any Geomys backend. Currently, it trusts the Google Modules Proxy to serve the correct zip files, without checking the transparency log proof. I plan to implement optional proof checking once proxy.golang.org CORS is fixed, including third-party gossip . Unfortunately, checking the proof does require fetching the whole module version’s zip archive to compute the dirhash, which is included in the Checksum Database (and in go.sum). For updates, follow me on Bluesky at @filippo.abyssdomain.expert or on Mastodon at @[email protected] . I recently went to Paris and found the Tour Eiffel elevator to be more fascinating than the tower itself. Whatever that says about me. My work is made possible by Geomys , an organization of professional Go maintainers, which is funded by Ava Labs , Teleport , Tailscale , and Sentry . Through our retainer contracts they ensure the sustainability and reliability of our open source maintenance work and get a direct line to my expertise and that of the other Geomys maintainers. (Learn more in the Geomys announcement .) Here are a few words from some of them! Teleport — For the past five years, attacks and compromises have been shifting from traditional malware and security breaches to identifying and compromising valid user accounts and credentials with social engineering, credential theft, or phishing. Teleport Identity is designed to eliminate weak access patterns through access monitoring, minimize attack surface with access requests, and purge unused permissions via mandatory access reviews. Ava Labs — We at Ava Labs , maintainer of AvalancheGo (the most widely used client for interacting with the Avalanche Network ), believe the sustainable maintenance and development of open source cryptographic protocols is critical to the broad adoption of blockchain technology. We are proud to support this necessary and impactful work through our ongoing sponsorship of Filippo and his team. I generally recommend against and actually configure to remove the direct fallback and reduce the attack surface of running git clone on potentially adversarial repositories. That’s besides the point of this article, though: whether you fetch direct or through a proxy, you will still always get the same contents authenticated by the sumdb (or an error).  ↩ If you use a code agent, you can add the following line to your AGENTS.md or CLAUDE.md:   ↩ I generally recommend against and actually configure to remove the direct fallback and reduce the attack surface of running git clone on potentially adversarial repositories. That’s besides the point of this article, though: whether you fetch direct or through a proxy, you will still always get the same contents authenticated by the sumdb (or an error).  ↩ If you use a code agent, you can add the following line to your AGENTS.md or CLAUDE.md:   ↩

0 views
Simon Willison 2 weeks ago

Introducing Showboat and Rodney, so agents can demo what they’ve built

A key challenge working with coding agents is having them both test what they’ve built and demonstrate that software to you, their overseer. This goes beyond automated tests - we need artifacts that show their progress and help us see exactly what the agent-produced software is able to do. I’ve just released two new tools aimed at this problem: Showboat and Rodney . I recently wrote about how the job of a software engineer isn't to write code, it's to deliver code that works . A big part of that is proving to ourselves and to other people that the code we are responsible for behaves as expected. This becomes even more important - and challenging - as we embrace coding agents as a core part of our software development process. The more code we churn out with agents, the more valuable tools are that reduce the amount of manual QA time we need to spend. One of the most interesting things about the StrongDM software factory model is how they ensure that their software is well tested and delivers value despite their policy that "code must not be reviewed by humans". Part of their solution involves expensive swarms of QA agents running through "scenarios" to exercise their software. It's fascinating, but I don't want to spend thousands of dollars on QA robots if I can avoid it! I need tools that allow agents to clearly demonstrate their work to me, while minimizing the opportunities for them to cheat about what they've done. Showboat is the tool I built to help agents demonstrate their work to me. It's a CLI tool (a Go binary, optionally wrapped in Python to make it easier to install) that helps an agent construct a Markdown document demonstrating exactly what their newly developed code can do. It's not designed for humans to run, but here's how you would run it anyway: Here's what the result looks like if you open it up in VS Code and preview the Markdown: Here's that demo.md file in a Gist . So a sequence of , , and commands constructs a Markdown document one section at a time, with the output of those commands automatically added to the document directly following the commands that were run. The command is a little special - it looks for a file path to an image in the output of the command and copies that image to the current folder and references it in the file. That's basically the whole thing! There's a command to remove the most recently added section if something goes wrong, a command to re-run the document and check nothing has changed (I'm not entirely convinced by the design of that one) and a command that reverse-engineers the CLI commands that were used to create the document. It's pretty simple - just 172 lines of Go. I packaged it up with my go-to-wheel tool which means you can run it without even installing it first like this: That command is really important: it's designed to provide a coding agent with everything it needs to know in order to use the tool. Here's that help text in full . This means you can pop open Claude Code and tell it: And that's it! The text acts a bit like a Skill . Your agent can read the help text and use every feature of Showboat to create a document that demonstrates whatever it is you need demonstrated. Here's a fun trick: if you set Claude off to build a Showboat document you can pop that open in VS Code and watch the preview pane update in real time as the agent runs through the demo. It's a bit like having your coworker talk you through their latest work in a screensharing session. And finally, some examples. Here are documents I had Claude create using Showboat to help demonstrate features I was working on in other projects: row-state-sql CLI Demo shows a new command I added to that same project. Change grouping with Notes demonstrates another feature where groups of changes within the same transaction can have a note attached to them. I've now used Showboat often enough that I've convinced myself of its utility. (I've also seen agents cheat! Since the demo file is Markdown the agent will sometimes edit that file directly rather than using Showboat, which could result in command outputs that don't reflect what actually happened. Here's an issue about that .) Many of the projects I work on involve web interfaces. Agents often build entirely new pages for these, and I want to see those represented in the demos. Showboat's image feature was designed to allow agents to capture screenshots as part of their demos, originally using my shot-scraper tool or Playwright . The Showboat format benefits from CLI utilities. I went looking for good options for managing a multi-turn browser session from a CLI and came up short, so I decided to try building something new. Claude Opus 4.6 pointed me to the Rod Go library for interacting with the Chrome DevTools protocol. It's fantastic - it provides a comprehensive wrapper across basically everything you can do with automated Chrome, all in a self-contained library that compiles to a few MBs. All Rod was missing was a CLI. I built the first version as an asynchronous report prototype , which convinced me it was worth spinning out into its own project. I called it Rodney as a nod to the Rod library it builds on and a reference to Only Fools and Horses - and because the package name was available on PyPI. You can run Rodney using or install it like this: (Or grab a Go binary from the releases page .) Here's a simple example session: Here's what that looks like in the terminal: As with Showboat, this tool is not designed to be used by humans! The goal is for coding agents to be able to run and see everything they need to know to start using the tool. You can see that help output in the GitHub repo. Here are three demonstrations of Rodney that I created using Showboat: After being a career-long skeptic of the test-first, maximum test coverage school of software development (I like tests included development instead) I've recently come around to test-first processes as a way to force agents to write only the code that's necessary to solve the problem at hand. Many of my Python coding agent sessions start the same way: Telling the agents how to run the tests doubles as an indicator that tests on this project exist and matter. Agents will read existing tests before writing their own so having a clean test suite with good patterns makes it more likely they'll write good tests of their own. The frontier models all understand that "red/green TDD" means they should write the test first, run it and watch it fail and then write the code to make it pass - it's a convenient shortcut. I find this greatly increases the quality of the code and the likelihood that the agent will produce the right thing with the smallest amount of prompts to guide it. But anyone who's worked with tests will know that just because the automated tests pass doesn't mean the software actually works! That’s the motivation behind Showboat and Rodney - I never trust any feature until I’ve seen it running with my own eye. Before building Showboat I'd often add a “manual” testing step to my agent sessions, something like: Both Showboat and Rodney started life as Claude Code for web projects created via the Claude iPhone app. Most of the ongoing feature work for them happened in the same way. I'm still a little startled at how much of my coding work I get done on my phone now, but I'd estimate that the majority of code I ship to GitHub these days was written for me by coding agents driven via that iPhone app. I initially designed these two tools for use in asynchronous coding agent environments like Claude Code for the web. So far that's working out really well. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . Proving code actually works Showboat: Agents build documents to demo their work Rodney: CLI browser automation designed to work with Showboat Test-driven development helps, but we still need manual testing I built both of these tools on my phone shot-scraper: A Comprehensive Demo runs through the full suite of features of my shot-scraper browser automation tool, mainly to exercise the command. sqlite-history-json CLI demo demonstrates the CLI feature I added to my new sqlite-history-json Python library. row-state-sql CLI Demo shows a new command I added to that same project. Change grouping with Notes demonstrates another feature where groups of changes within the same transaction can have a note attached to them. krunsh: Pipe Shell Commands to an Ephemeral libkrun MicroVM is a particularly convoluted example where I managed to get Claude Code for web to run a libkrun microVM inside a QEMU emulated Linux environment inside the Claude gVisor sandbox. Rodney's original feature set , including screenshots of pages and executing JavaScript. Rodney's new accessibility testing features , built during development of those features to show what they could do. Using those features to run a basic accessibility audit of a page . I was impressed at how well Claude Opus 4.6 responded to the prompt "Use showboat and rodney to perform an accessibility audit of https://latest.datasette.io/fixtures " - transcript here .

0 views
Simon Willison 3 weeks ago

Distributing Go binaries like sqlite-scanner through PyPI using go-to-wheel

I've been exploring Go for building small, fast and self-contained binary applications recently. I'm enjoying how there's generally one obvious way to do things and the resulting code is boring and readable - and something that LLMs are very competent at writing. The one catch is distribution, but it turns out publishing Go binaries to PyPI means any Go binary can be just a call away. sqlite-scanner is my new Go CLI tool for scanning a filesystem for SQLite database files. It works by checking if the first 16 bytes of the file exactly match the SQLite magic number sequence . It can search one or more folders recursively, spinning up concurrent goroutines to accelerate the scan. It streams out results as it finds them in plain text, JSON or newline-delimited JSON. It can optionally display the file sizes as well. To try it out you can download a release from the GitHub releases - and then jump through macOS hoops to execute an "unsafe" binary. Or you can clone the repo and compile it with Go. Or... you can run the binary like this: By default this will search your current directory for SQLite databases. You can pass one or more directories as arguments: Add for JSON output, to include file sizes or for newline-delimited JSON. Here's a demo: If you haven't been uv-pilled yet you can instead install using and then run . To get a permanent copy with use . The reason this is worth doing is that , and PyPI will work together to identify the correct compiled binary for your operating system and architecture. This is driven by file names. If you visit the PyPI downloads for sqlite-scanner you'll see the following files: When I run or on my Apple Silicon Mac laptop Python's packaging magic ensures I get that variant. Here's what's in the wheel , which is a zip file with a extension. In addition to the the most important file is which includes the following: That method - also called from - locates the binary and executes it when the Python package itself is executed, using the entry point defined in the wheel. Using PyPI as a distribution platform for Go binaries feels a tiny bit abusive, albeit there is plenty of precedent . I’ll justify it by pointing out that this means we can use Go binaries as dependencies for other Python packages now. That's genuinely useful! It means that any functionality which is available in a cross-platform Go binary can now be subsumed into a Python package. Python is really good at running subprocesses so this opens up a whole world of useful tricks that we can bake into our Python tools. To demonstrate this, I built datasette-scan - a new Datasette plugin which depends on and then uses that Go binary to scan a folder for SQLite databases and attach them to a Datasette instance. Here's how to use that (without even installing anything first, thanks ) to explore any SQLite databases in your Downloads folder: If you peek at the code you'll see it depends on sqlite-scanner in and calls it using against in its own scan_directories() function . I've been exploring this pattern for other, non-Go binaries recently - here's a recent script that depends on static-ffmpeg to ensure that is available for the script to use. After trying this pattern myself a couple of times I realized it would be useful to have a tool to automate the process. I first brainstormed with Claude to check that there was no existing tool to do this. It pointed me to maturin bin which helps distribute Rust projects using Python wheels, and pip-binary-factory which bundles all sorts of other projects, but did not identify anything that addressed the exact problem I was looking to solve. So I had Claude Code for web build the first version , then refined the code locally on my laptop with the help of more Claude Code and a little bit of OpenAI Codex too, just to mix things up. The full documentation is in the simonw/go-to-wheel repository. I've published that tool to PyPI so now you can run it using: The package you can see on PyPI was built using like this: This created a set of wheels in the folder. I tested one of them like this: When that spat out the correct version number I was confident everything had worked as planned, so I pushed the whole set of wheels to PyPI using like this: I had to paste in a PyPI API token I had saved previously and that was all it took. is very clearly meant as a proof-of-concept for this wider pattern - Python is very much capable of recursively crawling a directory structure looking for files that start with a specific byte prefix on its own! That said, I think there's a lot to be said for this pattern. Go is a great complement to Python - it's fast, compiles to small self-contained binaries, has excellent concurrency support and a rich ecosystem of libraries. Go is similar to Python in that it has a strong standard library. Go is particularly good for HTTP tooling - I've built several HTTP proxies in the past using Go's excellent handler. I've also been experimenting with wazero , Go's robust and mature zero dependency WebAssembly runtime as part of my ongoing quest for the ideal sandbox for running untrusted code. Here's my latest experiment with that library. Being able to seamlessly integrate Go binaries into Python projects without the end user having to think about Go at all - they and everything Just Works - feels like a valuable addition to my toolbox. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options .

0 views
Justin Duke 3 weeks ago

Brief notes on migrating to Postgres-backed jobs

It seems premature to talk about a migration that is only halfway done, even if it's the hard half that's done — but I think there's something useful in documenting the why and how of a transition while you're still in the thick of it, before the revisionist history of completion sets in. Early last year, we built out a system for running background jobs directly against Postgres within Django. This very quickly got abstracted out into a generic task runner — shout out to Brandur and many other people who have been beating this drum for a while. And as far as I can tell, this concept of shifting away from Redis and other less-durable caches for job infrastructure is regaining steam on the Rails side of the ecosystem, too. The reason we did it was mostly for ergonomics around graceful batch processing. It is significantly easier to write a poller in Django for stuff backed by the ORM than it is to try and extend RQ or any of the other task runner options that are Redis-friendly. Django gives you migrations, querysets, admin visibility, transactional guarantees — all for free, all without another moving part. And as we started using it and it proved stable, we slowly moved more and more things over to it. At the time of this writing, around half of our jobs by quantity — which represent around two-thirds by overall volume — have been migrated over from RQ onto this system. This is slightly ironic given that we also last year released django-rq-cron , a library that, if I have my druthers, we will no longer need. Fewer moving parts is the watchword. We're removing spindles from the system and getting closer and closer to a simple, portable, and legible stack of infrastructure.

1 views
Stone Tools 4 weeks ago

Scala Multimedia on the Commodore Amiga

The ocean is huge. It's not only big enough to separate landmasses and cultures, but also big enough to separate ideas and trends. Born and raised in the United States, I couldn't understand why the UK was always eating so much pudding. Please forgive my pre-internet cultural naiveté. I should also be kind to myself for thinking the Video Toaster was the be-all-end-all for video production and multimedia authoring on the Amiga. Search Amiga World metadata on Internet Archive for "toaster" and "scala" and you'll see my point. "Toaster" brings up dozens of top-level hits, and "Scala" gets zero. The NTSC/PAL divide was as vast as the ocean. From the States, cross either ocean and Scala was everywhere, including a full, physical-dongle-copy-protection-removed, copy distributed on the cover disk of CU Amiga Magazine , issue 96. Listening to Scala founder, Jon Bøhmer, speak of Scala 's creation in an interview on The Retro Hour , his early intuition on the Amiga's potential in television production built Scala into an omnipresent staple across multiple continents. Intuition alone can't build an empire. Bøhmer also had gladiatorial-like aggression to maintain his dominance in that market. As he recounted, "A Dutch company tried to make a Scala clone, and they made a mistake of putting...the spec sheet on their booth and said all those different things that Scala didn't have yet. So I took that spec sheet back to my developers (then, later) lo and behold before those guys had a bug free version out on the street, we had all their features and totally eradicated their whole proposal." Now, of course I understand that it would have been folly to ignore the threat. Looked at from another angle, Scala had apparently put themselves in a position where their dominance could face a legitimate threat from a disruptor. Ultimately, that's neither here nor there as in the end, Scala had early momentum and could swing the industry their direction. Scala (the software) remains alive and well even now, in the digital signage authoring and playback software arena. You know the stuff, like interactive touchscreens at restaurant checkouts, or animated displays at retail stores. As with the outliner/PIM software in the ThinkTank article , the world of digital signage is likewise shockingly crowded. Discovering this felt like catching a glimpse of a secondary, invisible world just below the surface of conscious understanding. Scala didn't find success without good reason. It solved some thorny broadcast production issues on hardware that was alone in its class for a time. A unique blend of software characteristics (multitasking, IFF, ARexx) turned an Amiga running Scala into more than the sum of its parts. Scala by itself would have made rumbles. Scala on the Amiga was seismic. At heart, I'm a print guy. Like anyone, I enjoy watching cool video effects, and I once met Kiki Stockhammer in person. But my brain has never been wired for animation or motion design. My 3D art was always static; my designs were committed to ink on paper. I liked holding a physical artifact in my hands at the end of the design process. Considering the sheer depths of my video naivete, for this investigation I will need a lot of help from the tutorials. I'll build the demo stuff from the manual, and try to push myself further and see where my explorations take me. CU Amiga Magazine issues 97 - 102 contain Scala MM300 tutorials as well, so I'll check those out for a man-on-the-streets point of view. The first preconception I need to shed is thinking Scala is HyperCard for the Amiga. It flirts with certain concepts, but building Myst with this would be out of reach for most people. I'll never say it's "impossible," as I don't like tempting the Fates that way, but it would need considerable effort and development skills. A little terminology is useful before we really dig in. I usually start an exploration of GUI applications by checking out the available menus. With Scala , there aren't any. I don't mean the menubar is empty, I mean there isn't a menubar, period. It does not exist. I am firmly in Scala Land and Scala 's vision of how multimedia work gets done. As with PaperClip , I find its opinionated interface comforting. I have serious doubts about common assumptions of interface homogeneity being a noble goal, but that's a discussion for a future post. Despite its plain look, what we see when the program launches is richly complex. Anything in purple (or whatever your chosen color scheme uses) is clickable, and if it has its own boundaries it does its own thing. Across the top we have the Scala logo, program title bar, and the Amiga Workbench "depth gadget." Clicking the logo is how we save our project and/or exit the program. Then we have what is clearly a list, and judging from interface cues its a list of pages. This list ("script") is akin to a HyperCard stack with transitions ("wipes") between cards ("pages"). Each subsection of any given line item is its own button for interfacing with that specific aspect of the page. It's approachable and nonthreatening, and to my mind encourages me to just click on things and see what happens. The bottom-sixth holds an array of buttons that would normally be secreted away under standard Amiga GUI menus. On the one hand, this means if you see it, it's available; no poking through dozens of greyed-out items. On the other hand, keyboard shortcuts and deeper tools aren't exposed. There's no learning through osmosis here. Following the tutorial, the first thing to do is define my first "page." Click "New," choose a background as a visual starting point if you like, click "OK", choose a resolution and color depth (this is per-screen, not per-project), and click "OK" to finish. The program steps me through the process; it is clear how to proceed. The design team for Scala really should be commended for the artistic craftsmanship of the product. It is easy to put something professional together with the included backgrounds, images, and music. Everything is tasteful and (mostly) subdued, if aesthetically "of its time" occasionally. Thanks to IFF support, if you don't like the built-in assets, you can create your own in one of the Amiga's many paint or music programs. That visual care extends to the included fonts, which are a murderer's row of well-crafted classics. All the big stars are here! Futura, Garamond, Gill Sans, Compact, and more. Hey, is that Goudy I see coming down the red carpet? And behind them? Why it's none other than Helvetica, star of their own hit movie that has the art world buzzing! And, oh no! Someone just threw red paint all over Franklin Gothic. What a shame, because I'm pretty sure that's a pleather dress. The next screen is where probably 85% of my time will be spent. One thing I've noticed with the manual is a lack of getting the reader up to speed on the nomenclature of the program. This screen contains the "Edit Menu" but is that what I should call this screen? The "Edit Menu" screen? Screen layouts are called "pages." Is this the "Page Edit" screen? Anyway, the "Edit Menu" gives a lot of control, both fine and coarse, for text styling, shape types, creating buttons, setting the color palette, coordinating object reveals, and more. Buttons with hide extra options, for styling or importing other resources, and it could be argued the interface works against itself a bit. As Scala has chosen to eschew typical Amiga GUI conventions, they walk a delicate line of showing as much as possible, while avoiding visual confusion. It never feels overwhelming, but only just and could stand to borrow from the MacOS playbook's popup menus, rather than cycling of options. Entering text is simple; click anywhere on the screen and begin typing. Where it gets weird is how Scala treats all text as one continuous block. Every line is ordered by Y-position on screen, but every line is connected to the next. Typing too much on a given line will spill over into the next line down, wherever it may be, and however it may be styled. 0:00 / 0:17 1× Text weirdness in the Edit Screen. (I think I had a Trapper Keeper in that pattern.) The unobtrusive buttons "IN" and "OUT" on the left define how the currently selected object will transition into or out of the screen. Doing this by mouse selection is kind of a drag, as there is no visible selection border for the object being modified. There is an option to draw boxes around objects, but there is no differentiation of selected vs. unselected objects, except when there is. It's a bit inconsistent. The "List" button reveals a method for assigning transitions and rearranging object timings precisely. It is quickly my preferred method for anything more complex than "a simple piece of text flies into view." As a list we can define only a pure sequence. Do a thing. Do a second thing. Do a third thing. The end. Multiple items can be "chained" to perform precisely the same wipe as the parent object, with no variation. It's a grouping tool, not a timing tool. 0:00 / 0:04 1× "List" editing of text effect timings. Stay tuned for the sequel: "celeriac and jicama" I'm having a lot of fun exploring these tools, and have immediately wandered off the tutorial path just to play around. Everything works like I'd expect, and I don't need to consult the manual much at all. There are no destructive surprises nor wait times. I click buttons and see immediate results; my inquisitiveness is rewarded. Pages with animation are all good and well, but it is interactivity which elevates a Scala page over the stoicism of a PowerPoint slide. That means it's time for the go-to interaction metaphor: the good ole' button. Where HyperCard has the concept of buttons as objects, in Scala a button is just a region of the screen. It accepts two events: and , though it burdens these simple actions with the confusing names and . I mix up these terms constantly in my mind. To add a button, draw a box. Alternately, click something you've drawn and a box bound to that object's dimensions will be auto-generated. Don't be fooled! That box is not tethered to the object. It just happens to be sized precisely to the object's current dimensions and position on screen, as a helpful shortcut to generate the most-likely button for your needs. Button interactions can do a few things. First, it can adjust colors within its boundaries. Amiga palettes use indexed color, so color swaps are trivial and pixel-perfect. Have some white text that should highlight in red when the mouse enters it? Set the "mark" (mouse enter) palette to remap white to red. Same for "select" (mouse click), a separate palette remap could turn the white to yellow on click. Why am I talking about this when I can just show you? 0:00 / 0:24 1× I intentionally drew the button to be half the text height to illustrate that the button has no relation to the text itself. Color remapping occurs within button boundaries. The double palettes represent the current palette (top), and the remapped palette (bottom). Buttons can also contain simple logic, setting or reading global variable states to determine how to behave at any given moment. IF-THEN statements can likewise be embedded to route presentation order based on those variables. So, a click could add +1 to a global counter, then if the counter is a certain value it could transition to a corresponding page. 0:00 / 0:03 1× If we feel particularly clever with index color palette remapping, it is possible to give the illusion of complete image replacement. Buttons do not need any visible attributes, nor do they need to be mouse-clicked to perform their actions. If "Function Keys" are enabled at the Scala "System" level, the first 10 buttons on a page are automatically linked to F1 - F10. A sample script which ships with Scala demonstrates F-Key control over a page in real-time, altering the values of sports scores by set amounts. This is a clever trick, and with deeper thought opens up interesting possibilities. If every page in a script were to secretly contain such a set of buttons, a makeshift control panel could function like a "video soundboard" of sorts. F-Keys could keep a presentation dynamic, perhaps reacting to live audience participation. I mention this for no particular reason and it is not a setup for a later reveal. ahem Once we've made some pages, its time to stitch them together into a proper presentation, a "script" in Scala parlance. This all happens in the "Main Menu" which works similarly to the "List" view when editing page text elements, with a few differences. "Wipe" is the transition from the previous page to the selected page. If you want to wipe "out" from a page with transition X, then wipe "in" to next page with transition Y, a page must be added in-between to facilitate that. The quality of the real-time wipe effects surprises me. Again, my video naivete is showing, because I always thought the Amiga needed specialized hardware to do stuff like this, especially when there is video input. The wipes are fun, if perhaps a little staid compared to the Toaster 's. In Scala 's defense, they remain a bit more timeless in their simplicity. "Pause" controls, by time or frame count, how long to linger on a page before moving on to the next one. Time can be relative to the start of the screen reveal, or absolute so as to coordinate Scala animations with known timestamps on a pre-recorded video source. A mouse click can also be assigned as the "pause," waiting for a click to continue. "Sound" attaches a sound effect, or a MOD music file, to the reveal. There are rudimentary tools for adjusting pitch and timing, and even for trimming sounds to fit. An in-built sampler makes quick, crunchy, low-fidelity voice recordings, for when you need to add a little extra pizazz in a pinch, or to rough out an idea to see how it works. Sometimes the best tool for the job is the one you have with you. There are hidden tools on the Main Menu. Like many modern GUI table views, the gap between columns is draggable. Narrowing the "Name" column reveals two hidden options to the right: Variables and Execute. Now I'm finally getting a whiff of HyperCard . Unlike HyperCard , these tools are rather opaque and non-intuitive. Right off the bat, there is no built-in script editor. Rather, Scala is happy to position itself as one tool in your toolbox, not to provide every tool you need out of the box. It's going to take some time to get to know how these work, perhaps more than I have allocated for this project, but I'll endeavor to at least come to grips with these. The Scala manual says, "The Scala definition of variables (closely resembles) ARexx, since all variable operators are performed by ARexx." After 40 years, I guess it's time to finally learn about ARexx. ARexx is the Amiga implementation of the REXX scripting language . From ARexx User's Reference Manual, "ARexx is particularly well suited as a command language. Command programs, sometimes called "scripts" or "macros", are widely used to extend the predefined commands of an operating system or to customize an applications program." This is essentially the Amiga's AppleScript equivalent, a statement which surely has a pedant somewhere punching their 1084 monitor at my ignorance. Indeed, the Amiga had ARexx before Apple had AppleScript, but not before Apple had HyperCard . Amiga Magazine , August 1989, described it thusly, "Amiga's answer to HyperCard is found in ARexx, a programming and DOS command language, macro processor, and inter-process controller, all rolled into one easy-to-use command language." "Easy-to-use" you say? Commodore had their heart in the right place, but the "Getting Acquainted" section of the ARexx manual immediately steers hard into programmer-speak. From the jump we're hit with stuff like, "(ARexx) uses the double-precision math library called "mathieeedoubbas.library" that is supplied with the Amiga WorkBench disk, so make sure that this file is present in your LIBS: directory. The distribution disk includes the language system, some example programs, and a set of the INCLUDE files required for integrating ARexx with other software packages." I know exactly what I'd have thought back in the day. What is a "mathieeedoubbas?" What is a "library?" Is "LIBS" and "library" the same thing? What is "double-precision?" What is "INCLUDE"? What is a "language system?" You, manual, said yourself on page 2, "If you are new to the REXX language, or perhaps to programming itself, you should review chapters 1 through 4." So far, that ain't helpin'. Luckily for young me, now me knows a thing or two about programming and can make sense of this stuff. Well, "sense" in the broadest definition only. What this means for Scala is that we have lots of options for handling variables and logic in our project. The manual says, "Any ARexx operators and functions can be used (in the variable field)." However, a function like "Say," which outputs text to console, doesn't make any sense in a Scala context, so I'm not always 100% clear where lie the boundaries of useful operators and functions. In addition to typical math functions and simple string concatentation, ARexx gives us boolean and equality checks, bitwise operators, random number generation, string to digit conversion, string filtering and trimming, the current time, and a lot more. Even checking for file existence works, which possibly carried over from Scala 's roots as a modem-capable automated remote video-titler. Realistically, there's only so much we can do given the tiny tiny OMG it's so small interface into which we type our expressions. My aspirations are scoped by the interface design. This is not necessarily a bad thing , IMHO. " Small, sharp tools " is a handy mental scoping model. Variables are global, starting from the page on which they're defined. So page 1 cannot reach variables defined on page 2. A page can display the value of any currently defined variable by using the prefix in the on-screen text, as in . 0:00 / 0:08 1× I was trying to do Cheifet's melt effect, but I couldn't get animated brushes to work in Scala . Still, I was happy to get even this level of control over genlock/ Scala interplay. "Execution" in the Main Menu means "execute a script." Three options are available: Workbench, CLI, and ARexx. For a feature that gets two pages in the manual with extra-wide margins, this is a big one, but I get why it only receives a brief mention. The only other recourse would be to include hundreds of pages of training material. "It exists. Have fun." is the basic thrust here. "Workbench" can launch anything reachable via the Workbench GUI, the same as double-clicking it. This is useful for having a script set up the working environment with helper apps, so an unpaid intern doesn't forget to open them. For ARexx stuff, programs must be running to receive commands, for example. "CLI" does the same thing as Workbench, except for AmigaDOS programs; programs that don't have a GUI front-end. Maybe open a terminal connection or monitor a system resource. "ARexx" of course runs ARexx scripts. For a program to accept ARexx commands, it must have an active REXX port open. Scala can send commands, and even its own variable data, to a target program to automate it in interesting ways. I saw an example of drawing images in a paint program entirely through ARexx scripting. Scala itself has an open REXX port, meaning its own tools can be controlled by other programs. In this way, data can flow between software, even from different makers, to form a little self-enclosed, automation ecosystem. One unusually powerful option is that Scala can export its own presentation script, which includes information for all pages, wipes, timings, sound cues, etc, as a self-contained ARexx script. Once in that format, it can be extended (in any text editor) with advanced ARexx commands and logic, perhaps to extract data from a database and build dynamic pages from that. Now it gets wild. That modified ARexx file can then be brought back into Scala as an "Execute" ARexx script on a page. Let me clarify this. A Scala script, which builds and runs an entire multi-page presentation, can itself be transformed into just another ARexx script assigned to a single page of a Scala project. One could imagine building a Scala front-end with a selection of buttons, each navigating on-click to a separate page which itself contains a complete, embedded presentation on a given topic. Scripts all the way down. There's one more scripting language Scala supports, and that's its own. Dubbed Scala Lingo (or is it Lingua?), when we save a presentation script we're saving in Lingo. It's human-readable and ARexx-friendly, which is what made it possible to save a presentation as an ARexx script in the previous section. Here's pure Lingo. This is a 320x200x16 (default palette) page, solid blue background with fade in. It displays one line of white text with anti-aliasing. The text slides in from the left, pauses 3 seconds, then slides out to the right. Here's the same page as an ARexx script. Looks like all we have to do is wrap each line of Lingo in single quotes, and add a little boilerplate. So, we have Scala on speaking terms with the Amiga and its applications, already a thing that could only be done on this particular platform at the time. Scala's choice of platform was further benefited by one of the Amiga's greatest strengths. That was thanks to the "villain" of the PaperClip article , Electronic Arts. The hardware and software landscape of the 70s and 80s was a real Wild West, anything goes, invent your own way, period of experimentation. Ideas could grow and bloom and wither on the vine multiple times over the course of a decade. Why, enough was going on a guy could devote an entire blog to it all. ahem While this was fun for the developers who had an opportunity to put their own stamp on the industry, for end-users it could create a bit of a logistical nightmare. Specifically, apps tended to be siloed, self-contained worlds which read and wrote their own private file types. Five different art programs? Five different file formats. Data migration was occasionally supported, as with VisiCalc's use of DIF (data interchange format) to store its documents. DIF was not a "standard" per se, but rather a set of guidelines for storing document data in ASCII format. Everyone using DIF could roll their own flavor and still call it DIF, like Lotus did in extending (but not diverging from) VisiCalc's original. Microsoft's DIF variant broke with everyone else, a fact we'll just let linger in the air like a fart for a moment. Let's really breathe it in, especially those of us on Windows 11 . More often than not, especially in the case of graphics and sound, DIF-like options were simply not available. Consider The Print Shop on the Apple 2. When its sequel, The New Print Shop , arrived it couldn't even open graphics from the immediately previous version of itself . A converter program was included to bring original Print Shop graphics into New Print Shop . On the C64, the Koala file format became semi-standard for images, simply by virtue of its popularity. Even so, there was a market for helping users move graphics across applications on the exact same hardware. While other systems struggled, programs like Deluxe Video on the Amiga were bringing in Deluxe Music and Deluxe Paint assets without fuss. A cynic will say, "Well yeah, those were all EA products so of course they worked together." That would be true in today's "silos are good, actually" regression of computing platforms into rent extractors. But, I will reiterate once more, there was genuinely a time when EA was good to its users. They didn't just treat developers as artists, they also empowered users in their creative pursuits. EA had had enough of the file format wars. They envisioned a brighter future and proposed an open file standard to achieve precisely that. According Dave Parkinson's article "A bit IFFy," in Amiga Computing Magazine , issue 7, "The origins of IFF are to be found in the (Apple) Macintosh's clipboard, and the file conventions which allow data to be cut and pasted between different Mac applications. The success of this led Electronic Arts to wonder — why not generalize this?" Why not, indeed! In 1985, working directly in conjunction with Commodore, the Electronic Arts Interchange File Format 1985 was introduced; IFF for short. It cannot be overstated how monumental it was to unlocking the Amiga's potential as a creative workhorse. From the Scala manual, "Unlike other computers, the Amiga has very standardized file formats for graphics and sound. This makes it easy to exchange data between different software packages. This is why you can grab a video image in one program, modify it in another, and display it in yet another." I know it's hard for younger readers to understand the excitement this created, except to simply say that everything in computing has its starting point. EA and the Amiga led the charge on this one. So, what is it? From "A Quick Introduction to IFF" by Jerry Morrison of Electronic Arts, "IFF is a 2-level standard. The first layer is the "wrapper" or “envelope” structure for all IFF files. Technically, it’s the syntax. The second layer defines particular IFF file types such as ILBM (standard raster pictures), ANIM (animation), SMUS (simple musical score), and 8SVX (8-bit sampled audio voice)." To assist in the explanation of the IFF file format, I built a Scala presentation just for you, taken from Amiga ROM Kernel Reference Manual . This probably would have been better built in Lingo, rather than trying to fiddle with the cumbersome editing tools and how they (don't) handle overlapping objects well. What's done is done. 0:00 / 0:07 1× I used the previously mentioned "link" wipe to move objects as groups. IFF is a thin wrapper around a series of data "chunks." It begins with a declaration of what type of IFF this particular file is, known as its "FORM." Above we see the ILBM "FORM," probably the most prevalent image format on the Amiga. Each chunk has its own label, describes how many bytes long it is, and is then followed by that many data bytes. That's really all there is to it. IDs for the FORM and the expected chunks are spec'd out in the registered definition document. Commodore wanted developers to always try to use a pre-existing IFF definition for data when possible. If there was no such definition, say for ultra-specialized data structures, then a new definition should be drawn up. "To prevent conflicts, new FORM identifications must be registered with Commodore before use," says Amiga ROM Kernel Reference Manual . In Morrison's write-up on IFF, he likened it to ASCII. When ASCII data is read into a program, it is sliced, diced, mangled, and whatever else needs to be done internally to make the program go. However, the data itself is on disk in a format unrelated to the program's needs. Morrison described a generic system for storing data, of whatever type, in a standardized way which separated data from software implementations. At its heart, IFF first declares what kind of data it holds (the FORM type), then that data is stored in a series of labelled chunks. The specification of how many chunks a given FORM needs, the proper labels for those chunks, the byte order for the raw data, and so on are all in the FORM's IFF definition document. In this way, anyone could write a simple IFF reader that follows the registered definition, et voila! Deluxe Paint animations are suddenly a valid media resource for Scala to consume. It can be confusing when hearing claims of "IFF compatibility" in magazines or amongst the Amiga faithful, but this does not mean that any random Amiga program can consume any random IFF file. The burden of supporting various FORMS still rests on each individual developer. FORM definitions which are almost identical, yet slightly different, were allowed. For example, the image FORM is "almost identical to " with differences in the chunk and the requirement of a new chunk called . "Almost identical" is not "identical" and so though both RGBN and ILBM are wrapped in standardized IFF envelopes, a program must explicitly support the ones of interest. Prevalent support for any given FORM type came out of a communal interest to make it standard. Cooperation was the unsung hero of the IFF format. "Two can do something better than one," has been on infinite loop in my mind since 1974. How evergreen is that XKCD comic about standards ? Obviously, given we're not using it these days, IFF wound up being one more format on the historical pile. We can find vestiges of its DNA here and there , but not the same ubiquity. There were moves to adopt IFF across other platforms. Tom Hudson, he of DEGAS Elite and CAD-3D , published a plea in the Fall 1986 issue of START Magazine for the Atari ST development crowd to adopt IFF for graphics files. He's the type to put up, not shut up, and so he also provided an IFF implementation on the cover disk, and detailed the format and things to watch out for. Though inspired by Apple originally, Apple seemed to believe IFF only had a place within a specific niche. AIFF, audio interchange file format, essentially standardized audio on the Mac, much like ILBM did for Amiga graphics. Despite being an IFF variant registered with Commodore, Scala doesn't recognize it in my tests. So, again, IFF itself wasn't a magical panacea for all file format woes. That fact was recognized even back in the 80s. In Amazing Computing , July 1987, in an article "Is IFF Really a Standard?" by John Foust, "Although the Amiga has a standard file format, it does not mean Babel has been avoided." He noted that programs can interpret IFF data incorrectly, resulting in distorted images, or outright failure. Ah well, nevertheless . Side note: One might reasonably believe TIFF to be a successful variant of IFF. Alas, TIFF shares "IFF" in name only and stands for "tagged image file format." One more side note: Microsoft also did to IFF what they did to DIF. fart noise The last major feature of note is Scala's extensibility. In the Main Menu list view, we have columns for various page controls. The options there can be expanded by including EX modules, programs which control external systems. This feels adjacent to HyperCard's XCMDs and XFCNs, which could extend HyperCard beyond its factory settings. EX modules bundled with Scala can control Sony Laserdisc controllers, enable MIDI file playback, control advanced Genlock hardware, and more. Once installed as a "Startup" item in Scala , these show up in the Main Menu and are as simple to control as any of Scala's built-in features. As an EX module, it is also Lingo scriptable so the opportunity to coordinate complex hardware interactions all through point-and-click is abundant. I turned on WinUAE's MIDI output and set it to "Microsoft GS Wave Table." In Amiga Workbench, I enabled the MIDI EX for Scala . On launch, Scala showed a MIDI option for my pages so I loaded up Bohemian-Rhapsody-1.mid . Mamma mia, it worked! I haven't found information about how to make new EXes, nor am I clear what EXes are available beyond Scala's own. However, here at the tail end of my investigation, Scala is suddenly doing things I didn't think it could do. The potential energy for this program is crazy high. No, I'm not going to be doing that any time soon, but boy do I see the appeal. Electronic Arts's documentation quoted Alan Kay for the philosophy behind the IFF standard, "Simple things should be simple, complex things should possible." Scala upholds this ideal beautifully. Making text animate is simple. Bringing in Deluxe Paint animations is simple. Adding buttons which highlight on hover and travel to arbitrary pages on click is simple. The pages someone would typically want to build, the bread-and-butter stuff, is simple. The complex stuff though, especially ARexx scripting, is not fooling around. I tried to script Scala to speak a phrase using the Amiga's built-in voice synthesizer and utterly failed. Jimmy Maher wrote of ARexx in The Future Was Here: The Commodore Amiga , "Like AmigaOS itself, it requires an informed, careful user to take it to its full potential, but that potential is remarkable indeed." While Scala didn't make me a video convert, it did retire within me the notion that the Toaster was the Alpha and Omega of the desktop video space. Interactivity, cross-application scripting, and genlock all come together into a program that feels boundless. In isolation, Scala a not a killer app. It becomes one when used as the central hub for a broader creative workflow. A paint program is transformed into a television graphics department. A basic sampler becomes a sound booth. A database and a little Lingo becomes an editing suite. Scala really proves the old Commodore advertising slogan correct, "Only Amiga Makes it Possible." 0:00 / 1:48 1× I'm accelerating the cycle of nostalgia. Now, we long for "four months ago." The more I worked with Scala , the more I wanted to see how close I could get to emulating video workflows of the day. Piece by piece over a few weeks I discovered the following (needs WinUAE , sorry) setup for using live Scala graphics with an untethered video source in a Discord stream. Scala can't do video switching*, so I'm locked to whatever video source happens to be genlocked to WinUAE at the moment. But since when were limitations a hindrance to creativity? * ARexx and EX are super-powerful and can extend Scala beyond its built-in limitations, but I don't see an obvious way to explore this within WinUAE. This is optional, depending on your needs, but its the fun part. You can use whatever webcam you have connected just as well. Camo Camera can stream mobile phone video to a desktop computer, wirelessly no less. Camo Camera on the desktop advertises your phone as a webcam to the desktop operating system. So, install that on both the mobile device and desktop, and connect them up. WinUAE can see the "default" Windows webcam, and only the default, as a genlock source; we can't select from a list of available inputs. It was tricky getting Windows 11 to ignore my webcam and treat Camo Camera as my default, but I got it to work. When you launch WinUAE , you should see your camera feed live in Workbench as the background. So far, so good. Next, in Scala > Settings turn on Genlock. You should now see your camera feed in Scala with Scala's UI overlaid. Now that we have Scala and our phone's video composited, switch over to OBS Studio . Set the OBS "Source" to "Window Capture" on WinUAE. Adjust the crop and scale to focus in on the portion of the video you're interested in broadcasting. On the right, under "Controls" click "Start Virtual Camera." Discord, Twitch , et al are able to see OBS as the camera input for streaming. When you can see the final output in your streaming service of choice (I used Discord 's camera test to preview), design the overlay graphics of your heart's desire. Use that to help position graphics so they won't be cut off due to Amiga/Discord aspect ratio differences. While streaming, interactivity with the live Scala presentation is possible. If you build the graphics and scripts just right, interesting real-time options are possible. Combine this with what we learned about buttons and F-Keys, and you could wipe to a custom screen like "Existential Crisis - Back in 5" with a keypress. 0:00 / 0:35 1× Headline transitions were manually triggered by the F-Keys, just to pay off the threat I made earlier in the post. See? I set'em up, I knock'em down. I also wrote a short piece about Cheifet , because of course I did. Ways to improve the experience, notable deficiencies, workarounds, and notes about incorporating the software into modern workflows (if possible). WinUAE v6.0.2 (2025.12.21) 64-bit on Windows 11 Emulating an NTSC Amiga 1200 2MB Chip RAM, 8MB Z2 Fast RAM AGA Chipset 68020 CPU, 24-bit addressing, no FPU, no MMU, cycle-exact emulation Kickstart/Workbench 3.1 (from Amiga Forever ) Windows directory mounted as HD0: For that extra analog spice, I set up the video Filter as per this article Scala Multimedia MM300 Cover disk version from CU Amiga Magazine , issue 96 (no copy protection) I didn't have luck running MM400 , nor could I find a MM400 manual Also using Deluxe Paint IV and TurboText Nothing to speak of. The "stock" Amiga 1200 setup worked great. I never felt the need to speed boost it, though I did give myself as much RAM as possible. I'll go ahead and recommend Deluxe Paint IV over III as a companion to Scala , because it supports the same resolutions and color depths. If you wind up with a copy of Scala that needs the hardware dongle, WinUAE emulates that as well. Under are the "red" (MM200) and "green" (MM300 and higher) variants I'm not aware of any other emulators that offer a Genlock option. I did not encounter any crashes of the application nor emulator. One time I had an "out of chip RAM" memory warning pop up in Scala . I was unclear what triggered it, as I had maxed out the chip RAM setting in WinUAE . Never saw it again after that. I did twice have a script become corrupted. Scripts are plain text and human-readable, so I was able to open it, see what was faulting, and delete the offending line. So, -6 points for corrupting my script; +2 points for keeping things simple enough that I could fix it on my own. F-Keys stopped working in Scala 's demonstration pages. Then, it started working again. I think there might have been an insidious script error that looked visually correct but was not. Deleting button variable settings and resetting them got it working again. This happened a few times. I saw some unusual drawing errors. Once was when a bar of color touched the bottom right edge of the visible portion of the screen, extra pixels were drawn into the overscan area. Another time, I had the phrase "Deluxe Paint" in Edit Mode, but when I viewed the page it only said "Deluxe Pa". Inspecting the text in "List" mode revealed unusual characters (the infinity symbol?!) had somehow been inserted into the middle of the text. I outlined one option above under "Bonus: Streaming Like Its 1993" above. OBS recording works quite well and is what I used for this post. WinUAE has recording options, but I didn't have a chance to explore them. I don't yet know how to export Scala animations into a Windows-playable format. For 2026, it would surely be nice to have native 16:9 aspect ratio support. Temporary script changes would be useful. I'd love to be able to turn off a page temporarily to better judge before/after flow. It can be difficult to visualize an entire project flow sometimes. With page transitions, object transitions, variable changes, logic flow, and more, understanding precisely what to do to create a desired effect can get a little confusing. Scala wants to maintain a super simple interface almost to its detriment. Having less pretty, more information dense, "advanced" interface options would be welcome. I suppose that's what building a script in pure ARexx is for. I'd like to be able to use DPaint animated brushes. Then I could make my own custom "transition" effects that mix with the Scala page elements. Maybe it's possible and I haven't figured out the correct methodology? The main thing I wanted was a Genlock switch, so I could do camera transitions easily. That's more of a WinUAE wishlist item though.

0 views
Nelson Figueroa 1 months ago

Using time.Sleep() in Go

You can use to pause your program for a predetermined amount of time, similar to most programming languages. First, you should know that the package has useful constants that allow you to conveniently specify time in units. They’re accessed with the notation. (e.g. ) You can use these constants with the function. For example, if we want to pause execution for 1 second, we can write the following code: We can do some multiplication with to pause a program for 30 seconds: You can do multiplication with all of the other constants mentioned as well. Here are some more examples using the other constants. Not really sure why you would use this, but just know it’s possible. My main motivation for writing this is that I think the official documentation is way too dense and doesn’t show several examples of . I just needed some examples to understand the syntax and move on with my day. https://pkg.go.dev/time

0 views

Education is not one-size-fits-all for the latest CTO Fellows

Today, I’m honored to introduce you to the latest cohort of Now Go Build CTO Fellows. Individuals from around the world with diverse backgrounds (engineering, the humanities, business, healthcare, entrepreneurship, law) that have found their calling at the intersection of technology and education.

0 views
Xe Iaso 1 months ago

Backfilling Discord forum channels with the power of terrible code

Hey all! We've got a Discord so you can chat with us about the wild world of object storage and get any help you need. We've also set up Answer Overflow so that you can browse the Q&A from the web. Today I'm going to discuss how we got there and solved one of the biggest problems with setting up a new community or forum: backfilling existing Q&A data so that the forum doesn't look sad and empty. All the code I wrote to do this is open source in our glue repo . The rest of this post is a dramatic retelling of the thought process and tradeoffs that were made as a part of implementing, testing, and deploying this pull request . Ready? Let's begin! There's a bunch of ways you can think about this problem, but given the current hype zeitgeist and contractual obligations we can frame this as a dataset management problem. Effectively we have a bunch of forum question/answer threads on another site, and we want to migrate the data over to a new home on Discord. This is the standard "square peg to round hole" problem you get with Extract, Transform, Load (ETL) pipelines and AI dataset management (mostly taking your raw data and tokenizing it so that AI models work properly). So let's think about this from an AI dataset perspective. Our pipeline has three distinct steps: When thinking about gathering and transforming datasets, it's helpful to start by thinking about the modality of the data you're working with. Our dataset is mostly forum posts, which is structured text. One part of the structure contains HTML rendered by the forum engine. This, the "does this solve my question" flag, and the user ID of the person that posted the reply are the things we care the most about. I made a bucket for this (in typical recovering former SRE fashion it's named for a completely different project) with snapshots enabled, and then got cracking. Tigris snapshots will let me recover prior state in case I don't like my transformations. When you are gathering data from one source in particular, one of the first things you need to do is ask permission from the administrator of that service. You don't know if your scraping could cause unexpected load leading to an outage. It's a classic tragedy of the commons problem that I have a lot of personal experience in preventing. When you reach out, let the administrators know the data you want to scrape and the expected load– a lot of the time, they can give you a data dump, and you don't even need to write your scraper. We got approval for this project, so we're good to go! To get a head start, I adapted an old package of mine to assemble User-Agent strings in such a way that gives administrators information about who is requesting data from their servers along with contact information in case something goes awry. Here's an example User-Agent string: This gives administrators the following information: This seems like a lot of information, but realistically it's not much more than the average Firefox install attaches to each request: The main difference is adding the workload hostname purely to help debugging a misbehaving workload. This is a concession that makes each workload less anonymous, however keep in mind that when you are actively scraping data you are being seen as a foreign influence. Conceding more data than you need to is just being nice at that point. One of the other "good internet citizen" things to do when doing benign scraping is try to reduce the amount of load you cause to the target server. In my case the forum engine is a Rails app (Discourse), which means there's a few properties of Rails that work to my advantage. Fun fact about Rails: if you append to the end of a URL, you typically get a JSON response based on the inputs to the view. For example, consider my profile on Lobsters at https://lobste.rs/~cadey . If you instead head to https://lobste.rs/~cadey.json , you get a JSON view of my profile information. This means that a lot of the process involved gathering a list of URLs with the thread indices we wanted, then constructing the thread URLs with slapped on the end to get machine-friendly JSON back. This made my life so much easier. Now that we have easy ways to get the data from the forum engine, the next step is to copy it out to Tigris directly after ingesting it. In order to do that I reused some code I made ages ago as a generic data storage layer kinda like Keyv in the node ecosystem . One of the storage backends was a generic object storage backend. I plugged Tigris into it and it worked on the first try. Good enough for me! Either way: this is the interface I used: By itself this isn't the most useful, however the real magic comes with my adaptor type . This uses Go generics to do type-safe operations on Tigris such that you have 90% of what you need for a database replacement. When you do any operations on a adaptor, the following happens: In the future I hope to extend this to include native facilities for forking, snapshots, and other nice to haves like an in-memory cache to avoid IOPs pressure, but for now this is fine. As the data was being read from the forum engine, it was saved into Tigris. All future lookups to that data I scraped happened from Tigris, meaning that the upstream server only had to serve the data I needed once instead of having to constantly re-load and re-reference it like the latest batch of abusive scrapers seem to do . So now I have all the data, I need to do some massaging to comply both with Discord's standards and with some arbitrary limitations we set on ourselves: In general, this means I needed to take the raw data from the forum engine and streamline it down to this Go type: In order to make this happen, I ended up using a simple AI agent to do the cleanup. It was prompted to do the following: I figured this should be good enough so I sent it to my local DGX Spark running GPT-OSS 120b via llama.cpp and manually looked at the output for a few randomly selected threads. The sample was legit, which is good enough for me. Once that was done I figured it would be better to switch from the locally hosted model to a model in a roughly equivalent weight class (gpt-5-mini). I assumed that the cloud model would be faster and slightly better in terms of its output. This test failed because I have somehow managed to write code that works great with llama.cpp on the Spark but results in errors using OpenAI's production models. I didn't totally understand what went wrong, but I didn't dig too deep because I knew that the local model would probably work well enough. It ended up taking about 10 minutes to chew through all the data, which was way better than I expected and continues to reaffirm my theory that GPT-OSS 120b is a good enough generic workhorse model, even if it's not the best at coding . From here things worked, I was able to ingest things and made a test Discord to try things out without potentially getting things indexed. I had my tool test-migrate a thread to the test Discord and got a working result. To be fair, this worked way better than expected (I added random name generation and as a result our CEO Ovais, became Mr. Quinn Price for that test), but it felt like one thing was missing: avatars. Having everyone in the migrated posts use the generic "no avatar set" avatar certainly would work, but I feel like it would look lazy. Then I remembered that I also have an image generation model running on the Spark: Z-Image Turbo . Just to try it out, I adapted a hacky bit of code I originally wrote on stream while I was learning to use voice coding tools to generate per-user avatars based on the internal user ID. This worked way better than I expected when I tested how it would look with each avatar attached to their own users. In order to serve the images, I stored them in the same Tigris bucket, but set ACLs on each object so that they were public, meaning that the private data stayed private, but anyone can view the objects that were explicitly marked public when they were added to Tigris. This let me mix and match the data so that I only had one bucket to worry about. This reduced a lot of cognitive load and I highly suggest that you repeat this pattern should you need this exact adaptor between this exact square peg and round hole combination. Now that everything was working in development, it was time to see how things would break in production! In order to give the façade that every post was made by a separate user, I used a trick that my friend who wrote Pluralkit (an accessibility tool for a certain kind of neurodivergence) uses: using Discord webhooks to introduce multiple pseudo-users into one channel. I had never combined forum channels with webhook pseudo-users like this before, but it turned out to be way easier than expected . All I had to do was add the right parameter when creating a new thread and the parameter when appending a new message to it. It was really neat and made it pretty easy to associate each thread ingressed from Discourse into its own Discord thread. Then all that was left was to run the Big Scary Command™ and see what broke. A couple messages were too long (which was easy to fix by simply manually rewriting them, doing the right state layer brain surgery, deleting things on Discord, and re-running the migration tool. However 99.9% of messages were correctly imported on the first try. I had to double check a few times including the bog-standard wakefulness tests. If you've never gone deep into lucid dreaming before, a wakefulness test is where you do something obviously impossible to confirm that it does not happen, such as trying to put your fingers through your palm. My fingers did not go through my palm. After having someone else confirm that I wasn't hallucinating more than usual I found out that my code did in fact work and as a result you can now search through the archives on community.tigrisdata.com or via the MCP server ! I consider that a massive success. As someone who has seen many truly helpful answers get forgotten in the endless scroll of chats, I wanted to build a way to get that help in front of users when they need it by making it searchable outside of Discord. Finding AnswerOverflow was pure luck: I happened to know someone who uses it for the support Discord for the Linux distribution I use on my ROG Ally, Bazzite . Thanks, j0rge! AnswerOverflow also has an MCP server so that your agents can hook into our knowledge base to get the best answers. To find out more about setting it up, take a look at the "MCP Server" button on the Tigris Community page . They've got instructions for most MCP clients on the market. Worst case, configure your client to access this URL: And bam, your agent has access to the wisdom of the ancients. But none of this is helpful without the actual answers. We were lucky enough to have existing Q&A in another forum to leverage. If you don't have the luxury, you can write your own FAQs and scenarios as a start. All I can say is, thank you to the folks who asked and answered these questions– we're happy to help, and know that you're helping other users by sharing. Connect with other developers, get help, and share your projects. Search our Q&A archives or ask a new question. Join the Discord . Extracting the raw data from the upstream source and caching it in Tigris. Transforming the cached data to make it easier to consume in Discord, storing that in Tigris again. Loading the transformed data into Discord so that people can see the threads in app and on the web with Answer Overflow . The name of the project associated with the requests (tigris-gtm-glue, where gtm means "go-to-market", which is the current in-vogue buzzword translation for whatever it is we do). The Go version, computer OS, and CPU architecture of the machine the program is running on so that administrator complaints can be easier isolated to individual machines. A contact URL for the workload, in our case it's just the Tigris home page. The name of the program doing the scraping so that we can isolate root causes down even further. Specifically it's the last path element of , which contains the path the kernel was passed to the executable. The hostname where the workload is being run in so that we can isolate down to an exact machine or Kubernetes pod. In my case it's the hostname of my work laptop. Key names get prefixed automatically. All data is encoded into JSON on write and decoded from JSON on read using the Go standard library. Type safety at the compiler level means the only way you can corrupt data is by having different "tables" share the same key prefix. Try not to do that! You can use Tigris bucket snapshots to help mitigate this risk in the worst case. Discord needs Markdown, the forum engine posts are all HTML. We want to remove personally-identifiable information from those posts just to keep things a bit more anonymous. Discord has a limit of 2048 characters per message and some posts will need to be summarized to fit within that window. Convert HTML to Markdown : Okay, I could have gotten away using a dedicated library for this like html2text , but I didn't think about that at the time. Remove mentions and names : Just strip them out or replace the mentions with generic placeholders ("someone I know", "a friend", "a colleague", etc.). Keep "useful" links : This was left intentionally vague and random sampling showed that it was good enough. Summarize long text : If the text is over 1000 characters, summarize it to less than 1000 characters.

0 views
Anton Zhiyanov 1 months ago

Interfaces and traits in C

Everyone likes interfaces in Go and traits in Rust. Polymorphism without class-based hierarchies or inheritance seems to be the sweet spot. What if we try to implement this in C? Interfaces in Go  • Traits in Rust  • Toy example  • Interface definition  • Interface data  • Method table  • Method table in implementor  • Type assertions  • Final thoughts An interface in Go is a convenient way to define a contract for some useful behavior. Take, for example, the honored : Anything that can read data into a byte slice provided by the caller is a . Quite handy, because the code doesn't need to care where the data comes from — whether it's memory, the file system, or the network. All that matters is that it can read the data into a slice: We can provide any kind of reader: Go's interfaces are structural, which is similar to duck typing. A type doesn't need to explicitly state that it implements ; it just needs to have a method: The Go compiler and runtime take care of the rest: A trait in Rust is also a way to define a contract for certain behavior. Here's the trait: Unlike in Go, a type must explicitly state that it implements a trait: The Rust compiler takes care of the rest: Either way, whether it's Go or Rust, the caller only cares about the contract (defined as an interface or trait), not the specific implementation. Let's make an even simpler version of — one without any error handling (Go): Usage example: Let's see how we can do this in C! The main building blocks in C are structs and functions, so let's use them. Our will be a struct with a single field called . This field will be a pointer to a function with the right signature: To make fully dynamic, let's turn it into a struct with a function pointer (I know, I know — just bear with me): Here's the "method" implementation: The is pretty obvious: And, finally, the function: See how easy it is to turn a into a : all we need is . Pretty cool, right? Not really. Actually, this implementation is seriously flawed in almost every way (except for the definition). Memory overhead . Each instance has its own function pointers (8 bytes per function on a 64-bit system) as "methods", which isn't practical even if there are only a few of them. Regular objects should store data, not functions. Layout dependency . Converting from to like only works if both structures have the same field as their first member. If we try to implement another interface: Everything will fall apart: and have different layouts, so type conversion in ⓧ is invalid and causes undefined behavior. Lack of type safety . Using a as the receiver in means the caller can pass any type, and the compiler won't even show a warning: C isn't a particularly type-safe language, but this is just too much. Let's try something else. A better way is to store a reference to the actual object in the interface: We could have the method in the interface take a instead of a , but that would make the implementation more complicated without any real benefits. So, I'll keep it as . Then will only have its own fields: We can make the method type-safe: To make this work, we add a method that returns the instance wrapped in a interface: The and functions remain quite simple: This approach is much better than the previous one: Since our type now knows about the interface (through the method), our implementation is more like a basic version of a Rust trait than a true Go interface. For simplicity, I'll keep using the term "interface". There is one downside, though: each instance has its own function pointer for every interface method. Since only has one method, this isn't an issue. But if an interface has a dozen methods and the program uses a lot of these interface instances, it can become a problem. Let's fix this. Let's extract interface methods into a separate strucute — the method table. The interface references its methods though the field: and don't change at all: The method initializes the static method table and assigns it to the interface instance: The only difference in is that it calls the method on the interface indirectly using the method table ( instead of ): stays the same: Now the instance always has a single pointer field for its methods. So even for large interfaces, it only uses 16 bytes ( + fields). This approach also keeps all the benefits from the previous version: We can even add a separate helper so the client doesn't have to worry about implementation detail: There's another approach I've seen out there. I don't like it, but it's still worth mentioning for completeness. Instead of embedding the method table in the interface, we can place it in the implementation ( ): We initialize the method table in the constructor: now takes a pointer: And converts to with a simple type cast: This keeps pretty lightweight, only adding one extra field. But the cast only works because is the first field in . If we try to implement a second interface, things will break — just like in the very first solution. I think the "method table in the interface" approach is much better. Go has an function that copies data from a source (a reader) to a destination (a writer): There's an interesting comment in its documentation: If implements , the copy is implemented by calling . Otherwise, if implements , the copy is implemented by calling . Here's what the function looks like: is a type assertion that checks if the reader is not just a , but also implements the interface. The Go runtime handles these kinds of dynamic type checks. Can we do something like this in C? I'd prefer not to make it fully dynamic, since trying to recreate parts of the Go runtime in C probably isn't a good idea. What we can do is add an optional method to the interface: Then we can easily check if a given is also a : Still, this feels a bit like a hack. I'd rather avoid using type assertions unless it's really necessary. Interfaces (traits, really) in C are possible, but they're not as simple or elegant as in Go or Rust. The method table approach we discussed is a good starting point. It's memory-efficient, as type-safe as possible given C's limitations, and supports polymorphic behavior. Here's the full source code if you are interested: The struct is lean and doesn't have any interface-related fields. The method takes a instead of a . The cast from to is handled inside the method. We can implement multiple interfaces if needed. Lightweight structure. Easy conversion from to . Supports multiple interfaces.

0 views

Model-Market Fit

In June 2007, Marc Andreessen published what became the defining essay on startup strategy. “The Only Thing That Matters” argued that of the three elements of a startup—team, product, and market— market matters most . A great market pulls the product out of the startup. The product doesn’t need to be great; it just has to basically work. Andreessen’s insight has guided a generation of founders. But nineteen years later, something has changed. A new variable has entered the equation. One that determines whether the market can pull anything at all. That variable is the model. For AI startups, there is a prerequisite layer beneath product-market fit: the degree to which current model capabilities can satisfy what a market demands . I call it Model-Market Fit, or MMF . When MMF exists, Andreessen’s framework applies perfectly. The market pulls the product out. When it doesn’t, no amount of brilliant UX, go-to-market strategy, or engineering can make customers adopt a product whose core AI task doesn’t solve their job to be done. The pattern is unmistakable once you see it. A model crosses a capability threshold. Within months, a vertical that had been dormant for years suddenly explodes with activity. For years, legal tech AI was stuck below scale. There were plenty of companies but none broke through. Document review tools that required more human oversight than they saved. Contract analysis that missed critical clauses. Every legal startup before 2023 struggled to cross $100M ARR. I remember this firsthand. I founded Doctrine in 2016, which grew to become the leading AI legal platform in Europe. But it was incredibly hard to raise money because all companies were sub-scale and the market wasn’t hot at all. Investors saw legal AI as a niche with limited upside. The market existed. Law firms desperately wanted automation. But the state-of-the-art models couldn’t handle the core tasks lawyers needed. BERT and similar transformer models excelled at classification like sorting documents, identifying contract types, flagging potential issues. But legal work requires generation and reasoning: drafting memos that synthesize complex case law, summarizing depositions while preserving nuanced arguments, generating discovery requests tailored to specific fact patterns. Traditional ML could categorize a contract as “employment” or “NDA,” but it couldn’t write a coherent brief explaining why a non-compete clause was unenforceable under California law. Then GPT-4 arrived in March 2023. Within eighteen months, Silicon Valley startups raised over hundreds of millions. Doctrine’s business is on fire. Thomson Reuters acquired Casetext for $650 million. Dozens of legal AI startups emerged. The legal AI market minted more unicorns in 12 months than in the previous 10 years combined. The market hadn’t changed. The model capability threshold had been crossed. Similarly, coding assistants existed before Sonnet. GitHub Copilot had millions of users. But there’s a difference between autocomplete that occasionally helps and an AI that genuinely understands your codebase and creates high-quality code for you. I experienced this firsthand. I tried Cursor early on, before Sonnet. It was meh. I installed it, tested it for a few days, deleted it. Did the same thing again a month later. Same result… interesting demo, not a workflow. Then Claude 3.5 Sonnet dropped! Within a week, I couldn’t work without Cursor. Neither could anyone on my team. The product became the workflow. We weren’t “using an AI assistant,” we were pair programming with something that understood our entire codebase. Cursor’s growth went vertical. Not because they shipped some brilliant new feature. Because the underlying model crossed the threshold that made their product actually work . They got Model Market Fit. The most important thing is MMF. The startups that won weren’t necessarily first, but they were prepared when the model capability threshold was finally crossed. So far in coding or legal, none of the incumbents won. It was always new players. Today’s leading legal startups had spent months understanding exactly how lawyers work like what output formats they need, what compliance requirements exist, how associates actually research cases. The race doesn’t go to the first mover. It goes to the first to product-market fit after model-market fit exists. The corollary is equally important: when MMF doesn’t exist, the market cannot pull. The demand is there. The willingness to pay is there. But the core task doesn’t work. Let’s review some examples. Mathematicians would love an AI that could prove novel theorems. The market is real, research institutions, defense contractors, and tech companies would pay millions for genuine mathematical reasoning. But even the most advanced models can’t do it consistently. They can verify known proofs. They can assist with mechanical steps. They can occasionally produce insights on bounded problems. But originating novel proofs on open problems? The capability threshold remains uncrossed. GPT-5, o1, o3... each generation improves incrementally, but we’re not at the point where you can feed an AI an open conjecture and expect a rigorous proof. Yet. Investment banks and hedge funds desperately want AI that can perform comprehensive financial analysis. The market is massive; a single successful trade or M&A deal can generate hundreds of millions in fees. But AI remains surprisingly bad at the core tasks that matter most. Excel output is still unreliable when dealing with complex financial models. More critically, AI struggles to combine quantitative analysis with qualitative insights from 200-page documents... exactly what analysts spend their days doing. A human analyst reads through earnings calls, regulatory filings, and industry reports, then synthesizes that qualitative intelligence with spreadsheet models to make investment recommendations. AI can handle pieces of this workflow, but the end-to-end reasoning that justifies million-dollar positions? The capability gap is wide today. This will obviously change soon. But for now, the human remains in the loop not as oversight, but as the primary decision-maker. The difference between verticals with MMF and those without is stark. Compare two benchmarks from Vals.ai: LegalBench (legal reasoning tasks): Top models hit 87% accuracy . Gemini 3 Pro leads at 87.04%, with multiple models clustered above 85%. This is production-grade performance. Accurate enough that lawyers can trust the output with light review. Finance Agent (core financial analyst tasks): Top models hit 56.55% accuracy . Even GPT-5.1, the current leader, barely crosses the halfway mark. Claude Sonnet 4.5 with extended thinking sits at 55.32%. That’s a 30-point gap. Legal has MMF. Finance doesn’t. The benchmarks reveal what intuition suggests: models have crossed the threshold for legal reasoning but remain fundamentally unreliable for financial analysis. You can ship a legal AI product today. A finance AI product that does the actual job of an analyst? Very soon but not now. The pharmaceutical industry has invested billions in AI-driven drug discovery. The market is enormous because a single successful drug is worth tens of billions. Yet the breakthroughs remain elusive. AI can accelerate certain steps: identifying candidate molecules, predicting protein structures (AlphaFold was transformative here), optimizing clinical trial design. But the end-to-end autonomous discovery that would justify the valuations? It doesn’t exist. The human remains in the loop not because the workflow is designed that way, but because the AI can’t actually do the job. There’s a reliable signal for missing MMF: examine how “human-in-the-loop” is positioned . When MMF exists, human-in-the-loop is a feature. It maintains quality, builds trust, handles edge cases. The AI does the work; the human provides oversight. When MMF doesn’t exist, human-in-the-loop is a crutch. It hides the fact that the AI can’t perform the core task. The human isn’t augmenting, they’re compensating. Strip away the human, and the product doesn’t work. The test is simple: if all human correction were removed from this workflow, would customers still pay? If the answer is no, there’s no MMF. There’s only a demo. This creates a brutal strategic dilemma. Do you build for current MMF or anticipated MMF? If MMF doesn’t exist today, building a startup around it means betting on model improvements that are on someone else’s roadmap. You don’t control when or whether the capability arrives. You’re burning runway while Anthropic and OpenAI decide your fate. Worse, you might be wrong about what capability is needed. Models might scale differently than you expect. The 80% to 99% accuracy gap that your vertical requires might be five years away, or it might never close in the way you imagined. Of course, if you believe in Artificial General Intelligence, then you know that models will eventually be able to do pretty much anything. But “eventually” is doing a lot of work in that sentence. The question isn’t whether AI will solve the problem; it’s when, and whether your startup survives long enough to see it (which is a function of your runway). But there’s a counterargument often shared at Ycombinator, and it’s compelling. When MMF unlocks, you need more than just model capability. You need: - Domain-specific data pipelines - Regulatory relationships - Customer trust built over years - Deep workflow integration - Understanding of how professionals actually work Legal startups didn’t just plug in GPT-4. They had already built the scaffolding. When the model arrived, they were ready to run. There’s also the question of influence. The teams closest to the problem shape how models get evaluated, fine-tuned, and deployed. They’re not passively waiting for capability; they’re defining what capability means in their vertical. The question isn’t whether to be early. It’s how early, and what you’re building while you wait. The dangerous zone is the middle: MMF that’s 24 to 36 months away. Close enough to seem imminent. Far enough to burn through multiple funding rounds waiting. This is where conviction and runway become everything. If you’re betting on MMF that’s 2+ years out, you better be in a gigantic market worth the wait. Consider healthcare and financial services. These markets are so massive that even Anthropic and OpenAI are going all-in despite very mixed current results. The potential upside justifies positioning early, even if the models aren’t quite there yet. When you’re targeting trillion dollar markets, the risk-reward calculation changes entirely. The math is simple: expected value = probability of MMF arriving × market size × your likely share . Product-market fit has famously resisted precise measurement. Andreessen described it qualitatively: “ You can always feel when product/market fit isn’t happening... And you can always feel product/market fit when it’s happening. ” MMF is similarly intuitive, but we can be more specific. Can the model, given the same inputs a human expert receives, produce output that a customer would pay for without significant human correction? This test has three components: 1. Same inputs : The model gets what the human would get—documents, data, context. No magical preprocessing that a real workflow couldn’t provide. 2. Output a customer would pay for : Not a demo. Not a proof of concept. Production-quality work that solves a real problem. 3. Without significant human correction : The human might review, refine, or approve. But if they’re rewriting 50% of the output, the model isn’t doing the job. In unregulated verticals, 80% accuracy might be enough. An AI that writes decent first drafts of marketing copy creates value even if humans edit heavily. In regulated verticals—finance, legal, healthcare—80% accuracy is often useless. A contract review tool that misses 20% of critical clauses isn’t augmenting lawyers; it’s creating liability. A medical diagnostic that’s wrong one time in five isn’t a product; it’s a lawsuit haha! The gap between 80% and 99% accuracy is often infinite in practice. It’s the difference between “promising demo” and “production system.” Many AI startups are stuck in this gap, raising money on demos while waiting for the capability that would make their product actually work. There’s a second capability frontier that most discussions of MMF miss: the ability to work autonomously over extended periods . Current MMF examples (legal document review, coding assistance) are fundamentally short-horizon tasks today. Prompt in, output out, maybe a few tool calls. The model does something useful in seconds or minutes. But the highest-value knowledge work isn’t like that. A financial analyst doesn’t answer one question; they spend days building a model, stress-testing assumptions, and synthesizing information across dozens of sources. A strategy consultant doesn’t produce a single slide; they iterate through weeks of research, interviews, and analysis. A drug discovery researcher doesn’t run one experiment; they design and execute campaigns spanning months. These workflows require something models can’t yet do reliably: sustained autonomous operation . The agentic threshold isn’t just “can the model use tools.” It’s: - Persistence : Can it maintain goals and context across hours or days? - Recovery : Can it recognize failures, diagnose problems, and try alternative approaches? - Coordination : Can it break complex objectives into subtasks and execute them in sequence? - Judgment : Can it know when to proceed versus when to stop and ask for guidance? Today’s agents can handle tasks measured in minutes. Tomorrow’s need to handle tasks measured in days. That’s not an incremental improvement—it’s a phase change in capability. This is why finance doesn’t have MMF despite models being “good at reading documents.” Reading a 10-K is a 30-second task. Building an investment thesis is a multi-day workflow requiring the agent to gather data, build models, test scenarios, and synthesize conclusions—all while maintaining coherent reasoning across the entire process. The next wave of MMF unlocks will come from smarter models AND models that can work for days on the same task. Andreessen’s core insight was that market matters more than team or product because a great market pulls the product out of the startup. The market creates the gravitational force. The AI corollary: model capability is the prerequisite for that gravitational pull to begin . No market, however large and hungry, can pull a product that doesn’t work. And in AI, “doesn’t work” is determined by the model, not by your engineering or design. You can build the most beautiful interface, the most elegant workflow, the most sophisticated data pipeline… and if the underlying model can’t perform the core task, none of it matters. MMF → PMF → Success. Skip the first step, and the second becomes impossible. This is both constraint and opportunity. For founders, it means being ruthlessly honest about where capability actually is versus where you hope it will be. For investors, it means evaluating not just market size and team quality, but the gap between current model capability and what the market requires. And for everyone building in AI: the question isn’t just whether the market wants what you’re building. It’s whether the models can deliver it. That’s the only thing that matters.

1 views