Building Barista—Part 2: Formal Verification and TLA+
☕ Welcome to The Coder Cafe! Part 1 of Building Barista covered the architecture. Today, we explore formal verification through a real bug my tests missed. You’ll learn what model checking is, how to read a bit of TLA+, and why it catches the bugs we never think about. I spent a lot of time on formal verification at Google; this time, I pointed it at my own system. Get cozy, grab a coffee, and let’s begin! One Email a Day A quick summary of Barista : it’s a system I built that fetches hundreds of sources, gets a unique prompt per user, and sends that user one email per day with the best sources for them. Making sure Barista sends a single email per user per day is something important to me. If Alice receives the same email twice on a random day, that’s a terrible user experience. As discussed in Part 1 , Barista is a set of independent stages coordinating through Postgres. The last one, , runs every 5 minutes, looks for the users whose send time has passed, and emails each of them the posts picked by the stage. The nominal scenario: The stage starts and notices an email has to be sent to Alice. Barista calls the email provider and receives an ACK. ✅ Barista writes back in Postgres that the email to Alice was sent that day. Now the worst-case scenario: The stage starts and notices an email has to be sent to Alice. Barista calls the email provider and receives an ACK. ❌ Before writing back in Postgres that the email was sent, Barista crashes. The stage starts again and notices an email has to be sent to Alice. Barista calls the email provider again and receives an ACK. ← Duplicate email ✅ Barista writes back in Postgres that the email to Alice was sent that day. How do you prevent that? I picked my email provider for it: Plunk. Why Plunk? Because it accepts an idempotency key. When calling Plunk, I pass a key like : the user and the day. If Plunk receives the same key again, it doesn’t send anything; it replies with a 409. My code reads that 409 as “these posts were already sent” and marks them as sent. No duplicate. It was a conscious decision. Plunk is 10x more expensive per email than AWS SES, which has no idempotency key. I was happy to pay for it. So here was the state: A business decision: no duplicates, even if it costs more. A sound design, with an email provider deduplicating for me. Unit tests, integration tests, end-to-end tests. All green. Great mental model of the code. Pure intellectual control . Now, what if I told you my code had a bug? One that none of my tests caught. I only managed to find it with something called formal verification. Let’s see how. Formal Verification Before becoming an SRE, I worked in Air Traffic Management (I talked about it in Reliability Explained ). It’s a so-called safety-critical domain. You can easily imagine why. In a safety-critical system, green tests aren’t enough. That’s the blind spot of intellectual control: tests only check the scenarios we thought about. And the bugs that hurt are, by definition, the ones we didn’t think about. In domains like this, green tests aren’t the end of the conversation. Some teams go further and prove that certain bugs can’t happen. Airbus, for example, used a static analyzer called Astrée to prove that the flight control software of the A340 and A380 is free of a whole class of runtime errors. So instead of picking scenarios one by one, what if we could check all of them? That’s the idea behind formal verification: describe the system precisely, write down the properties it must respect, and let a tool check that no possible execution breaks them. Note the word: verification . It answers “did we build the thing right?”, meaning does the design respect our properties. Not to be confused with validation : “did we build the right thing?”, meaning are these the properties we actually need. A tool can do the first. The second stays on us. There are several ways to verify a system, from mathematical proofs to running real code on generated inputs. The one I used is model checking . A model checker sees a system as states (a value for every variable) and actions (a step from one state to the next). It starts from the initial state, tries every possible action, then every possible action from each new state, and so on, until it has visited every reachable state. A test follows one path. A model checker walks all of them. That’s why it finds bugs nobody writes a test for. AWS described it in 2015, in a paper on how they use formal methods: one of their DynamoDB bugs had survived design reviews, code reviews, and testing. The shortest sequence of events to trigger it was 35 steps long. At each state, the model checker checks our properties. They come in two flavors: Safety : Something bad never happens (e.g., Alice never receives the same email twice) Liveness : Something good eventually happens (e.g., Alice eventually receives her daily email) When a property breaks, we get a counterexample : the exact sequence of steps that led there. The model checker I used is TLC. It reads models written in TLA+, a language created by Leslie Lamport to describe a system as states and actions. In other words: we describe the system in TLA+, and TLC explores every state it can reach. A TLA+ spec doesn’t look like code. There are no instructions executed one after the other. Instead, each action is a formula that says two things: when the step is allowed (a condition on the current state), and what the state looks like after it (the values in the next state). TLC then tries every allowed action, in every possible order. A few symbols are enough to follow the examples below: means “is defined as” is the value of in the next state and mean “for all” and “there exists” Let’s see it on a simple example. Imagine a cafe with one croissant left and two baristas. When a customer asks for a croissant, a barista does two things: first, they check the display case to see if there’s one left; then, they sell it. The property we want is a safety one (something bad never happens): the cafe never sells a croissant it doesn’t have. In other words, the stock never goes below zero. To model this cafe, we need two variables: : how many croissants are left. It starts at 1. : where each barista is. Each one starts , moves to after checking, and ends . Now, the two actions. The first one is checking the display case: The first two lines are conditions: can only happen if barista is idle and there’s a croissant left. The last two lines describe the next state: is now selling ( means “every other barista stays where they are”), and the stock stays the same. The second action is selling: And the property, written as an invariant that must hold in every state: That’s the whole model. TLC starts from the initial state (one croissant, two idle baristas) and tries every action, in every possible order. In most orders, everything is fine. For example, barista 1 checks and sells, then barista 2 can’t check anymore because the stock is zero. But TLC finds one order that breaks the invariant: Both baristas looked at the display case before either one sold. Both saw a croissant. Both sold it. That’s how to read a counterexample: find the step where it broke (5), then walk back to the decision that made it inevitable (3). The fix is to make checking and selling a single action, so nobody can slip in between. Re-run TLC, and it explores every state without finding a violation. In a cafe, we can merge two steps. Across a network call and a database write, we can’t, and that’s exactly why idempotency keys exist. Now, let’s do the same with Barista. Back to Barista. Years after leaving Air Traffic Management, I wondered: what if I applied the same rigor here? So I wrote a TLA+ spec of the stage. The model was tiny: one user, two posts, a clock with eight slots per day, and two days. The key idea is that it tracks two things separately: what was really delivered to the user, and what Barista believes was delivered. If Barista works, both always agree. The full spec also has a few actions I’m leaving out here: the clock ticks, Rank picks new posts, and moves the next send time forward. Here are the variables (simplified): : the clock. : when user ’s next email is due. : the posts picked for ’s next email. : the idempotency keys Plunk has already seen. : the list of emails really delivered. The truth. : the posts Barista has marked as sent to . What Barista believes. And a simplified version of the action at the time: A few new pieces: defines a local name (here, the key). is a pair, our . adds to a set, and adds to the end of a list. And is a record: an email to containing these posts. Read it line by line. can only run once the send time has passed. If Plunk already knows the key, it replies with a 409 and nothing is delivered. Otherwise, the email goes out. Either way, Plunk remembers the key, and Barista marks the posts as sent. That last line is the one to watch. The real action can also end badly: Plunk delivers the email, then Barista crashes before recording anything. TLC tries that outcome too. Now the property, in the same spirit as the cafe’s : In plain English: Barista’s belief must match the truth. If a post is marked as sent, an email really contained it. I ran TLC. Guess what? Despite all the tests, despite the idempotency key I was paying 10x for, it failed on the first run . Every configuration failed, even the simplest one: no crash, no settings change. Here’s one of the counterexamples, in real times. Alice gets her email at 22:00 UTC: An outage on May 20 eats Alice’s May 21 email. The catch-up send at 01:00 UTC uses key , so the regular send at 22:00 gets a 409, and its posts are marked as sent without being delivered. Alice never gets her May 21 email. Its posts are marked as sent, so Barista will never offer them again. No error, no alert. And it happens 21 hours after everything recovered. The fun part is that I never modeled an outage. Nothing in the spec says Barista can be down. But nothing forces to run on time either: in TLA+, the clock is just another action, and TLC is free to let it tick while waits. An outage, a slow deploy, a job stuck behind another one: for the model, it’s all the same. TLC tried every possible delay and found the one crossing midnight. So what went wrong? Plunk did exactly what it promised. A 409 doesn’t mean “these posts were sent.” It means “something was already sent under this key.” My key was the day of the attempt, not the day the email was for. As long as a send ran on the day it was due, it didn’t matter. The moment one slipped past midnight, it did. Should Barista skip the catch-up, then? No. Alice should still get her May 20 email. The catch-up was right; my reading of the 409 was wrong. Ironically, I was obsessed with sending too many emails, and the very thing I was paying 10x for made me send too few, silently. Would my tests have caught it? Honestly, no. I tested a crash after the ACK, and the idempotency key handled it. I would never have written a test where runs late, past midnight UTC, followed by a regular send the next evening. And that was exactly my fear in the first place. The fix? One key per email instead of one per day. I re-ran TLC. Green. In December 2025, Martin Kleppmann, the author of Designing Data-Intensive Applications , published a post with a bold prediction: AI will make formal verification go mainstream . His argument is mostly about cost. Formal verification has always been expensive. Proving that seL4, an OS kernel of about 8,700 lines of C, was correct took around 20 person-years. For most systems, a bug was cheaper than the proof that would have prevented it. But coding agents are getting good at writing proofs. And a wrong proof isn’t dangerous: the checker rejects it, and the agent tries again. One of the issues with TLA+ has always been making sure the spec is accurate. A TLA+ spec is a model, and a model is only as good as its abstraction. TLC checks the spec, not the code. The hard part is making sure the spec describes what the code actually does. But with the rise of coding agents, especially over the last few months, I feel like this gap is shrinking. An agent can read the code and draft the spec, keep both in sync when the code changes, and turn a counterexample into a failing test against the real code. At Google, I spent a lot of time exploring formal verification, especially on convoluted logic: loops with several paths, retries, error codes. On many occasions, I found bugs despite significant test coverage. Back then, though, writing and maintaining a spec was a real investment, so it was reserved for the parts that truly justified it. That’s what’s changing. When a spec costs an afternoon instead of weeks, the question is no longer “is this part critical enough to deserve a spec?” It becomes “why not?” Does it mean software will be bug-free? No. As Kleppmann also points out, the hard part moves to the specification itself: are we checking the properties we actually care about? That’s the validation question from earlier. But think about what it took to find Barista’s bug: a property of a few lines, and a tool patient enough to try every delay, every order, every midnight. Now extend that. Maybe that’s what software engineering becomes: less time on how the code achieves something, more time on what must hold no matter what. Alice never receives the same email twice. Alice eventually receives her daily email. We write those down; agents write the code and the spec; tools check that the properties hold. Kleppmann compares it to compilers: we don’t read the machine code they produce, and maybe one day, we won’t read the code agents produce either. Will we get there? I don’t know. But it might be worth exploring in your own system: write down its invariants, and see whether a model can show you where they break. If you want to see what this pipeline picks for you, give Barista a try. Describe what you like to read, and your first email lands in your inbox in seconds. It’s free (one post a day), and every new account gets 7 days of the paid plan to try more. Resources Sources How Amazon Web Services Uses Formal Methods, Newcombe et al., CACM, 2015 Systems Correctness Practices at AWS, Brooker and Desai, ACM Queue, 2024 The Astrée Static Analyzer Prediction: AI will make formal verification go mainstream How we continuously validate formal specifications in production - Datadog One Email a Day A quick summary of Barista : it’s a system I built that fetches hundreds of sources, gets a unique prompt per user, and sends that user one email per day with the best sources for them. Making sure Barista sends a single email per user per day is something important to me. If Alice receives the same email twice on a random day, that’s a terrible user experience. As discussed in Part 1 , Barista is a set of independent stages coordinating through Postgres. The last one, , runs every 5 minutes, looks for the users whose send time has passed, and emails each of them the posts picked by the stage. The nominal scenario: The stage starts and notices an email has to be sent to Alice. Barista calls the email provider and receives an ACK. ✅ Barista writes back in Postgres that the email to Alice was sent that day. The stage starts and notices an email has to be sent to Alice. Barista calls the email provider and receives an ACK. ❌ Before writing back in Postgres that the email was sent, Barista crashes. The stage starts again and notices an email has to be sent to Alice. Barista calls the email provider again and receives an ACK. ← Duplicate email ✅ Barista writes back in Postgres that the email to Alice was sent that day. A business decision: no duplicates, even if it costs more. A sound design, with an email provider deduplicating for me. Unit tests, integration tests, end-to-end tests. All green. Safety : Something bad never happens (e.g., Alice never receives the same email twice) Liveness : Something good eventually happens (e.g., Alice eventually receives her daily email) means “is defined as” is the value of in the next state and mean “for all” and “there exists” : how many croissants are left. It starts at 1. : where each barista is. Each one starts , moves to after checking, and ends . : the clock. : when user ’s next email is due. : the posts picked for ’s next email. : the idempotency keys Plunk has already seen. : the list of emails really delivered. The truth. : the posts Barista has marked as sent to . What Barista believes. An outage on May 20 eats Alice’s May 21 email. The catch-up send at 01:00 UTC uses key , so the regular send at 22:00 gets a 409, and its posts are marked as sent without being delivered. Alice never gets her May 21 email. Its posts are marked as sent, so Barista will never offer them again. No error, no alert. And it happens 21 hours after everything recovered. The fun part is that I never modeled an outage. Nothing in the spec says Barista can be down. But nothing forces to run on time either: in TLA+, the clock is just another action, and TLC is free to let it tick while waits. An outage, a slow deploy, a job stuck behind another one: for the model, it’s all the same. TLC tried every possible delay and found the one crossing midnight. So what went wrong? Plunk did exactly what it promised. A 409 doesn’t mean “these posts were sent.” It means “something was already sent under this key.” My key was the day of the attempt, not the day the email was for. As long as a send ran on the day it was due, it didn’t matter. The moment one slipped past midnight, it did. Should Barista skip the catch-up, then? No. Alice should still get her May 20 email. The catch-up was right; my reading of the 409 was wrong. Ironically, I was obsessed with sending too many emails, and the very thing I was paying 10x for made me send too few, silently. Would my tests have caught it? Honestly, no. I tested a crash after the ACK, and the idempotency key handled it. I would never have written a test where runs late, past midnight UTC, followed by a regular send the next evening. And that was exactly my fear in the first place. The fix? One key per email instead of one per day. I re-ran TLC. Green. The Future of Software Engineering? In December 2025, Martin Kleppmann, the author of Designing Data-Intensive Applications , published a post with a bold prediction: AI will make formal verification go mainstream . His argument is mostly about cost. Formal verification has always been expensive. Proving that seL4, an OS kernel of about 8,700 lines of C, was correct took around 20 person-years. For most systems, a bug was cheaper than the proof that would have prevented it. But coding agents are getting good at writing proofs. And a wrong proof isn’t dangerous: the checker rejects it, and the agent tries again. One of the issues with TLA+ has always been making sure the spec is accurate. A TLA+ spec is a model, and a model is only as good as its abstraction. TLC checks the spec, not the code. The hard part is making sure the spec describes what the code actually does. But with the rise of coding agents, especially over the last few months, I feel like this gap is shrinking. An agent can read the code and draft the spec, keep both in sync when the code changes, and turn a counterexample into a failing test against the real code. At Google, I spent a lot of time exploring formal verification, especially on convoluted logic: loops with several paths, retries, error codes. On many occasions, I found bugs despite significant test coverage. Back then, though, writing and maintaining a spec was a real investment, so it was reserved for the parts that truly justified it. That’s what’s changing. When a spec costs an afternoon instead of weeks, the question is no longer “is this part critical enough to deserve a spec?” It becomes “why not?” Does it mean software will be bug-free? No. As Kleppmann also points out, the hard part moves to the specification itself: are we checking the properties we actually care about? That’s the validation question from earlier. But think about what it took to find Barista’s bug: a property of a few lines, and a tool patient enough to try every delay, every order, every midnight. Now extend that. Maybe that’s what software engineering becomes: less time on how the code achieves something, more time on what must hold no matter what. Alice never receives the same email twice. Alice eventually receives her daily email. We write those down; agents write the code and the spec; tools check that the properties hold. Kleppmann compares it to compilers: we don’t read the machine code they produce, and maybe one day, we won’t read the code agents produce either. Will we get there? I don’t know. But it might be worth exploring in your own system: write down its invariants, and see whether a model can show you where they break. If you want to see what this pipeline picks for you, give Barista a try. Describe what you like to read, and your first email lands in your inbox in seconds. It’s free (one post a day), and every new account gets 7 days of the paid plan to try more. Resources Sources How Amazon Web Services Uses Formal Methods, Newcombe et al., CACM, 2015 Systems Correctness Practices at AWS, Brooker and Desai, ACM Queue, 2024 The Astrée Static Analyzer Prediction: AI will make formal verification go mainstream How we continuously validate formal specifications in production - Datadog
Tutorial
Backend
DevOps
Performance
Rails
Ruby
Sql
Docker
Php