The latest Thoughtworks TechRadar is out. Here are some of the more data-related ‘blips’ (as they’re called on the radar) that I noticed. Each item links to the blip’s entry where you can read more information about Thoughtwork’s usage and opinions on it. Databricks Assistant Apache Paimon Delta Sharing Naive API-to-MCP conversion Standalone data engineering teams Text to SQL
Nextcloud. I really want to like it, but it’s making it really difficult. I like what Nextcloud offers with its feature set and how easily it replaces a bunch of services under one roof (files, calendar, contacts, notes, to-do lists, photos etc.), but no matter how hard I try and how much I optimize its resources on my home server, it feels slow to use, even on hardware that is ranging from decent to good. Then I opened developer tools and found the culprit. It’s the Javascript. On a clean page load, you will be downloading about 15-20 MB of Javascript, which does compress down to about 4-5 MB in transit, but that is still a huge amount of Javascript. For context, I consider 1 MB of Javascript to be on the heavy side for a web page/app. Yes, that Javascript will be cached in the browser for a while, but you will still be executing all of that on each visit to your Nextcloud instance, and that will take a long time due to the sheer amount of code your browser now has to execute on the page. A significant contributor to this heft seems to be the bundle, which based on its name seems to provide some common functionality that’s shared across different Nextcloud apps that one can install. It’s coming in at 4.71 MB at the time of writing. Then you want notifications, right? is here to cover you, at 1.06 MB . Then there are the app-specific views. The Calendar app is taking up 5.94 MB to show a basic calendar view. Files app includes a bunch of individual scripts, such as ( 1.77 MB ), ( 1.17 MB ), ( 1.09 MB ), ( 0.9 MB which I’ve never used!) and many smaller ones. Notes app with its basic bare-bones editor? 4.36 MB for the ! This means that even on an iPhone 13 mini, opening the Tasks app (to-do list), will take a ridiculously long time. Imagine opening your shopping list at the store and having to wait 5-10 seconds before you see anything, even with a solid 5G connection. Sounds extremely annoying, right? I suspect that a lot of this is due to how Nextcloud is architected. There’s bound to be some hefty common libraries and tools that allow app developers to provide a unified experience, but even then there is something seriously wrong with the end result, the functionality to bundle size ratio is way off. As a result, I’ve started branching out some things from Nextcloud, such as replacing the Tasks app with using a private Vikunja instance, and Photos to a private Immich instance. Vikunja is not perfect, but its 1.5 MB of Javascript is an order of magnitude smaller compared to Nextcloud, making it feel incredibly fast in comparison. However, with other functionality I have to admit that the convenience of Nextcloud is enough to dissuade me from replacing it elsewhere, due to the available feature set comparing well to alternatives. I’m sure that there are some legitimate reasons behind the current state, and overworked development teams and volunteers are unfortunately the norm in the industry, but it doesn’t take away the fact that the user experience and accessibility suffers as a result. I’d like to thank Alex Russell for writing about web performance and why it matters, with supporting evidence and actionable advice, it has changed how I view websites and web apps and has pushed me to be better in my own work. I highly suggest reading his content, starting with the performance inequality gap series. It’s educational, insightful and incredibly irritating once you learn how crap most things are and how careless a lot of development teams are towards performance and accessibility.
This blog post explains how to effectively file abuse reports against cloud providers to stop malicious traffic. Key points: Two IP Types : Residential (ineffective to report) vs. Commercial (targeted reports) Why Cloud Providers : Cloud customers violate provider terms, making abuse reports actionable Effective Abuse Reports Should Include : Note on "Free VPNs" : Often sell your bandwidth as part of botnets, not true public infrastructure The goal is to make scraping the cloud provider's problem, forcing them to address violations against their terms of service. Two IP Types : Residential (ineffective to report) vs. Commercial (targeted reports) Why Cloud Providers : Cloud customers violate provider terms, making abuse reports actionable Effective Abuse Reports Should Include : Time of abusive requests IP/User-Agent identifiers robots.txt status System impact description Service context Process : Use whois to find abuse contacts (look for "abuse-c" or "abuse-mailbox") Send detailed reports with all listed emails Expect response within 2 business days Note on "Free VPNs" : Often sell your bandwidth as part of botnets, not true public infrastructure
“ The Kafka community is currently seeing an unprecedented situation with three KIPs ( KIP-1150 , KIP-1176 , KIP-1183) simultaneously addressing the same challenge of high replication costs when running Kafka across multiple cloud availability zones. ” — Luke Chen, The Path Forward for Saving Cross-AZ Replication Costs KIPs At the time of writing the Kafka project finds itself at a fork in the road where choosing the right path forward for implementing S3 topics has implications for the long-term success of the project. Not just the next couple of years, but the next decade. Open-source projects live and die by these big decisions and as a community, we need to make sure we take the right one. This post explains the competing KIPs, but goes further and asks bigger questions about the future direction of Kafka. Before comparing proposals, we should step back and ask what kind of system we want Kafka to become. Kafka now faces two almost opposing forces. One force is stabilizing: the on-prem deployments and the low latency workloads that depend on local disks and replication. Kafka must continue to serve those use cases. The other force is disrupting: the elastic, cloud-native workloads that favor stateless compute and shared object storage. Relaxed-latency workloads such as analytics have seen a shift in system design with durability increasingly delegated to shared object storage, freeing the compute layer to be stateless, elastic, and disposable. Many systems now scale by adding stateless workers rather than rebalancing stateful nodes. In a stateless compute design, the bottleneck shifts from data replication to metadata coordination. Once durability moves to shared storage, sequencing and metadata consistency become the new limits of scalability. That brings us to the current moment, with three competing KIPs defining how to integrate object storage directly into Kafka. While we evaluate these KIPs, it’s important to consider the motivations for building direct-to-S3 topics. Cross-AZ charges are typically what are on people’s minds, but it’s a mistake to think of S3 simply as a cheaper disk or a networking cheat. The shift is also architectural, providing us an opportunity to achieve those operational benefits such as elastic stateless compute. The devil is in the details: how each KIP enables Kafka to leverage object storage while also retaining Kafka’s soul and what made it successful in the first place. With that in mind, while three KIPs have been submitted, it comes down to two different paths: Revolutionary : Choose a direct-to-S3 topic design that maximizes the benefits of an object-storage architecture, with greater elasticity and lower operational complexity. However, in doing so, we may increase the implementation cost and possibly the long-term code maintenance too by maintaining two very different topic-models in the same project (leader-based replication and direct-to-S3). Evolutionary : Shoot for an evolutionary design that makes use of existing components to reduce the need for large refactoring or duplication of logic. However, by coupling to the existing architecture, we forfeit the extra benefits of object storage, focusing primarily on networking cost savings (in AWS and GCP). Through this coupling, we also run the risk of achieving the opposite: harder to maintain code by bending and contorting a second workload into an architecture optimized for something else. In this post I will explain the two paths in this forked road, how the various KIPs map onto those paths, and invite the whole community to think through what they want for Apache Kafka for the next decade. Note that I do not include KIP-1183 as it looks dead in the water, and not a serious contender. The KIP proposes AutoMQ’s storage abstractions without the accompanying implementation. Which perhaps cynically, seems to benefit AutoMQ were it ever adopted, leaving the community to rewrite the entire storage subsystem again. If you want a quick summary of the three KIPs (including KIP-1183), you can read Luke Chen’s The Path Forward for Saving Cross-AZ Replication Costs KIPs or Anton Borisov’s summary of the three KIPs . This post is structured as follows: The term “Diskless” vs “Direct-to-S3” The Common Parts. Some approaches are shared across multiple implementations and proposals. Revolutionary: KIPs and real-world implementations Evolutionary: Slack’s KIP-1176 The Hybrid: balancing revolution with evolution Deciding Kafka’s future I used the term “diskless” in the title as that is the current hype word. But it is clear that not all designs are actually diskless in the same spirit as “serverless”. Serverless implies that users no longer need to consider or manage servers at all, not that there are no servers. In the world of open-source, where you run stuff yourself, diskless would have to mean literally “no disks”, else you will be configuring disks as part of your deployment. But all the KIPs (in their current state) depend on disks to some extent, even KIP-1150 which was proposed as diskless. In most cases, disk behavior continues to influence performance and therefore correct disk provisioning will be important. So I’m not a fan of “diskless”, I prefer “direct-to-S3”, which encompasses all designs that treat S3 (and other object stores) as the only source of durability. The main commonality between all Direct-to-S3 Kafka implementations and design proposals is the uploading of objects that combine the data of multiple topics. The reasons are two-fold: Avoiding the small file problem . Most designs are leaderless for producer traffic, allowing for any server to receive writes to any topic. To avoid uploading a multitude of tiny files, servers accumulate batches in a buffer until ready for upload. Before upload, the buffer is sorted by topic id and partition, to make compaction and some reads more efficient by ensuring that data of the same topic and same partition are in contiguous byte ranges. Pricing . The pricing of many (but not all) cloud object storage services penalize excessive requests, so it can be cheaper to roll-up whatever data has been received in the last X milliseconds and upload it with a single request. In the leader-based model, the leader determines the order of batches in a topic partition. But in the leaderless model, multiple brokers could be simultaneously receiving produce batches of the same topic partition, so how do we order those batches? We need a way of establishing a single order for each partition and we typically use the word “sequencing” to describe that process. Usually there is a central component that does the sequencing and metadata storage, but some designs manage the sequencing in other ways. WarpStream was the first to demonstrate that you could hack the metadata step of initiating a producer to provide it with broker information that would align the producer with a zone-local broker for the topics it is interested in. The Kafka client is leader-oriented, so we just pass it a zone-local broker and tell the client “this is the leader”. This is how all the leaderless designs ensure producers write to zone-local brokers. It’s not pretty, and we should make a future KIP to avoid the need for this kind of hack. Consumer zone-alignment heavily depends on the particular design, but two broad approaches exist: Leaderless: The same way that producer alignment works via metadata manipulation or using KIP-392 (fetch from follower) which can be used in a leaderless context. Leader-based: Zone-aware consumer group assignment as detailed in KIP-881: Rack-aware Partition Assignment for Kafka Consumers . The idea is to use consumer-to-partition assignment to ensure consumers are only assigned zone-local partitions (where the partition leader is located). KIP-392 (fetch-from-follower) , which is effective for designs that have followers (which isn’t always the case). Given almost all designs upload combined objects, we need a way to make those mixed objects more read optimized. This is typically done through compaction, where combined objects are ultimately separated into per-topic or even per-partition objects. Compaction could be one-shot or go through multiple rounds. The “revolutionary” path draws a new boundary inside Kafka by separating what can be stateless from what must remain stateful. Direct-to-S3 traffic is handled by a lightweight, elastic layer of brokers that simply serve producers and consumers. The direct-to-S3 coordination (sequencing/metadata) is incorporated into the stateful side of regular brokers where coordinators, classic topics and KRaft live. I cover three designs in the “revolutionary” section: WarpStream (as a reference, a kind of yardstick to compare against) KIP-1150 revision 1 Aiven Inkless (a Kafka-fork) Before we look at the KIPs that describe possible futures for Apache Kafka, let’s look at a system that was designed from scratch with both cross-AZ cost savings and elasticity (from object storage) as its core design principles. WarpStream was unconstrained by an existing stateful architecture, and with this freedom, it divided itself into: Leaderless, stateless and diskless agents that handle Kafka clients, as well as compaction/cleaning work. Coordination layer : A central metadata store for sequencing, metadata storage and housekeeping coordination. Fig WS-A. The WarpStream stateless/stateful split architecture. As per the Common Parts section, the zone alignment and sorted combined object upload strategies are employed. Fig WS-B. The WarpStream write path. On the consumer side, which again is leaderless and zone-local, a per-zone shared cache is implemented (which WarpStream dubbed distributed mmap). Within a zone, this shared cache assigns each agent a portion of the partition-space. When a consumer fetch arrives, an agent will download the object byte range itself if it is responsible for that partition, else it will ask the responsible agent to do that on its behalf. That way, we ensure that multiple agents per zone are not independently downloading the same data, thus reducing S3 costs. Fig WS-C. Shared per-zone read cache to reduce S3 throughput and request costs. WarpStream implements agent roles (proxy, job) and agent groups to separate the work of handling producer, consumer traffic from background jobs such as compaction, allowing for independent scaling for each workload. The proxy role (producer/consumer Kafka client traffic) can be further divided into proxy-producer and proxy-consumer. Agents can be deployed as dedicated agent groups, which allows for further separation of workloads. This is useful for avoiding noisy neighbor issues, running different groups in different VPCs and scaling different workloads that hit the same topics. For example, you could use one proxy group for microservices, and a separate proxy-consumer group for an analytics workload. Fig WS-D. Agent roles and groups can be used for shaping traffic, independent scaling and deploying different workloads into separate VPCs. Being a pure Direct-to-S3 system allowed WarpStream to choose a design that clearly separates traffic serving work into stateless agents and the coordination logic into one central metadata service. The traffic serving layer is highly elastic, with relatively simple agents that require no disks at all. Different workloads benefit from independent and flexible scaling via the agent roles and groups. The point of contention is the metadata service, which needs to be carefully managed and scaled to handle the read/write metadata volume of the stateless agents. Confluent Freight Clusters follow a largely similar design principle of splitting stateless brokers from central coordination. I will write about the Freight design sometime soon in the future. Apache Kafka is a stateful distributed system, but next we’ll see how KIP-1150 could fulfill much of the same capabilities as WarpStream. KIP-1150 has continued to evolve since it was first proposed, undergoing two subsequent major revisions between the KIP itself and mailing list discussion . This section describes the first version of the KIP, created in April 2025. KIP-1150 revision 1 uses a leaderless architecture where any broker can serve Kafka producers and consumers of any topic partition. Batch Coordinators (replicated state machines like Group and Transaction Coordinators), handle coordination (object sequencing and metadata storage). Brokers accumulate Kafka batches and upload shared objects to S3 (known as Shared Log Segment Objects, or SLSO). Metadata that maps these blocks of Kafka batches to SLSOs (known as Batch Coordinates) is then sent to a Batch Coordinator (BC) that sequences the batches (assigning offsets) to provide a global order of those batches per partition. The BC acts as sequencer and batch coordinate database for later lookups, allowing for the read path to retrieve batches from SLSOs. The BCs will also apply idempotent and transactional producer logic (not yet finalized). Fig KIP-1150-rev1-A. Leaderless brokers upload shared objects, then commit and sequence them via the Batch Coordinator. The Batch Coordinator (BC) is the stateful component akin to WarpStream’s metadata service. Kafka has other coordinators such as the Group Coordinator (for the consumer group protocol), the Transaction Coordinator (for reliable 2PC) and Share Coordinator (for queues aka share groups). The coordinator concept, for centralizing some kind of coordination, has a long history in Kafka. The BC is the source of truth about uploaded objects, Direct-to-S3 topic partitions, and committed batches. This is the component that contains the most complexity, with challenges around scaling, failovers, reliability as well as logic for idempotency, transactions, object compaction coordination and so on. A Batch Coordinator has the following roles: Sequencing . Chooses the total ordering for writes, assigning offsets without gaps or duplicates. Metadata storage . Stores all metadata that maps partition offset ranges to S3 object byte ranges. Serving lookup requests . Serving requests for log offsets. Serving requests for batch coordinates (S3 object metadata). Partition CRUD operations . Serving requests for atomic operations (creating partitions, deleting topics, records, etc.) Data expiration Managing data expiry and soft deletion. Coordinating physical object deletion (performed by brokers). The Group and Transaction Coordinators use internal Kafka topics to durably store their state and rely on the KRaft Controller for leader election (coordinators are highly available with failovers). This KIP does not specify whether the Batch Coordinator will be backed by a topic, or use some other option such as a Raft state machine (based on KRaft state machine code). It also proposes that it could be pluggable. For my part, I would prefer all future coordinators to be KRaft state machines, as the implementation is rock solid and can be used like a library to build arbitrary state machines inside Kafka brokers. When a producer starts, it sends a Metadata request to learn which brokers are the leaders of each topic partition it cares about. The Metadata response contains a zone-local broker. The producer sends all Produce requests to this broker. The receiving broker accumulates batches of all partitions and uploads a shared log segment object (SLSO) to S3. The broker commits the SLSO by sending the metadata of the SLSO (the metadata is known as the batch coordinates) to the Batch Coordinator. The coordinates include the S3 object metadata and the byte ranges of each partition in the object. The Batch Coordinator assigns offsets to the written batches, persists the batch coordinates, and responds to the broker. The broker sends all associated acknowledgements to the producers. Brokers can parallelize the uploading SLSOs to S3, but commit them serially to the Batch Coordinator. When batches are first written, a broker does not assign offsets as would happen with a regular topic, as this is done after the data is written, when it is sequenced and indexed by the Batch Coordinator. In other words, the batches stored in S3 have no offsets stored within the payloads as is normally the case. On consumption, the broker must inject the offsets into the Kafka batches, using metadata from the Batch Coordinator. The consumer sends a Fetch request to the broker. The broker checks its cache and on a miss, the broker queries the Batch Coordinator for the relevant batch coordinates. The broker downloads the data from object storage. The broker injects the computed offsets and timestamps into the batches (the offsets being part of the batch coordinates). The broker constructs and sends the Fetch response to the Consumer. Fig KIP-1150-rev1-B. The consume path of KIP-1150 (ignoring any caching logic here). The KIP notes that broker roles could be supported in the future. I believe that KIP-1150 revision 1 starts becoming really powerful with roles akin to WarpStream. That way we can separate out direct-to-S3 topic serving traffic and object compaction work on proxy brokers , which become the elastic serving and background job layer. Batch Coordinators would remain hosted on standard brokers, which are already stateful. With broker roles, we can see how Kafka could implement its own WarpStream-like architecture, which makes full use of disaggregated storage to enable better elasticity. Fig KIP-1150-rev1-C. Broker roles would bring the stateless/stateful separation that would unlock elasticity in the high throughput workload of many Direct-to-S3 deployments. Things like supporting idempotency, transactions, caching and object compaction are left to be decided later. While taxing to design, these things look doable within the basic framework of the design. But as I already mentioned in this post, this will be costly in effort to develop but also may come with a long term code maintenance overhead if complex parts such as transactions are maintained twice. It may also be possible to refactor rather than do wholesale rewrites. Inkless is Aiven’s direct-to-S3 fork of Apache Kafka. The Inkless design shares the combined object upload part and metadata manipulation that all designs across the board are using. It is also leaderless for direct-to-S3 topics. Inkless is firmly in the revolutionary camp. If it were to implement broker roles, it would make this Kafka-fork much closer to a WarpStream-like implementation (albeit with some issues concerning the coordination component as we’ll see further down). While Inkless is described as a KIP-1150 implementation, we’ll see that it actually diverges significantly from KIP-1150, especially the later revisions (covered later). Inkless eschewed the Batch Coordinator of KIP-1150 in favor of a Postgres instance, with coordination being executed through Table Valued Functions (TVF) and row locking where needed. Fig Inkless-A. The write path. On the read side, each broker must discover what batches exist by querying Postgres (using a TVF again), which returns the next set of batch coordinates as well as the high watermark. Now the broker knows where the next batches are located, it requests those batches via a read-through cache. On a cache miss, it fetches the byte ranges from the relevant objects in S3. Fig Inkless-B. The read path. Inkless bears some resemblance to KIP-1150 revision 1, except the difficult coordination bits are delegated to Postgres. Postgres does all the sequencing, metadata storage, as well as coordination for compaction and file cleanup. For example, compaction is coordinated via Postgres, with a TVF that is periodically called by each broker which finds a set of files which together exceed a size threshold and places them into a merge work order (tables file_merge_work_items and file_merge_work_item_files ) that the broker claims. Once carried out, the original files are marked for deletion (which is another job that can be claimed by a broker). Fig Inkless-C. Direct-to-S3 traffic uses a leaderless broker architecture with coordination owned by Postgres. Inkless doesn’t implement transactions, and I don’t think Postgres could take on the role of transaction coordinator, as the coordinator does more than sequencing and storage. Inkless will likely have to implement some kind of coordinator for that. The Postgres data model is based on the following tables: logs . Stores the Log Start Offset and High Watermark of each topic partition, with the primary key of topic_id and partition. files . Lists all the objects that host the topic partition data. batches . Maps Kafka batches to byte ranges in Files. producer_state . All the producer state needed for idempotency. Some other tables for housekeeping, and the merge work items. Fig Inkless-D. Postgres data model. The Commit File TVF , which sequences and stores the batch coordinates, works as follows: A broker opens a transaction and submits a table as an argument, containing the batch coordinates of the multiple topics uploaded in the combined file. The TVF logic creates a temporary table (logs_tmp) and fills it via a SELECT on the logs table, with the FOR UPDATE clause which obtains a row lock on each topic partition row in the logs table that matches the list of partitions being submitted. This ensures that other brokers that are competing to add batches to the same partition(s) queue up behind this transaction. This is a critical barrier that avoids inconsistency. These locks are held until the transaction commits or aborts. Next it, inside a loop, partition-by-partition, the TVF: Updates the producer state. Updates the high watermark of the partition (a row in the logs table). Inserts the batch coordinates into the batches table (sequencing and storing them). Commits the transaction. Apache Kafka would not accept a Postgres dependency of course, and KIP-1150 has not proposed centralizing coordination in Postgres either. But the KIP has suggested that the Batch Coordinator be pluggable, which might leave it open for using Postgres as a backing implementation. As a former database performance specialist, the Postgres locking does concern me a bit. It blocks on the logs table rows scoped to the topic id and partition. An ORDER BY prevents deadlocks, but given the row locks are maintained until the transaction commits, I imagine that given enough contention, it could cause a convoy effect of blocking. This blocking is fair, that is to say, First Come First Serve (FCFS) for each individual row. For example, with 3 transactions: T1 locks rows 11–15, T2 wants to lock 6-11, but only manages 6-10 as it blocks on row 11. Meanwhile T2 wants to lock 1-6, but only manages 1-5 as it blocks on 6. We now have a dependency tree where T1 blocks T2 and T2 blocks T3. Once T1 commits, the others get unblocked, but under sustained load, this kind of locking and blocking can quickly cascade, such that once contention starts, it rapidly expands. This contention is sensitive to the number of concurrent transactions and the number of partitions per commit. A common pattern with this kind of locking is that up until a certain transaction throughput everything is fine, but at the first hint of contention, the whole thing slows to a crawl. Contention breeds more contention. I would therefore caution against the use of Postgres as a Batch Coordinator implementation. The following is a very high-level look at Slack’s KIP-1176 , in the interests of keeping this post from getting too detailed. There are three key points to this KIP’s design: Maintain leader-based topic partitions (producers continue to write to leaders), but replace Kafka replication protocol with a per-broker S3-based write-ahead-log (WAL). Try to preserve existing partition replica code for idempotency and transactions. Reuse existing tiered storage for long-term S3 data management. Fig Slack-KIP-1176-A. Leader-based architecture retained, replication replaced by an S3 WAL. Tiered storage manages long-term data. The basic idea is to preserve the leader-based architecture of Kafka, with each leader replica continuing to write to an active local log segment file, which it rotates periodically. A per-broker write-ahead-log (WAL) replaces replication. A WAL Combiner component in the Kafka broker progressively (and aggressively) tiers portions of the local active log segment files (without closing them), combining them into multi-topic objects uploaded to S3. Once a Kafka batch has been written to the WAL, the broker can send an acknowledgment to its producer. This active log segment tiering does not change how log segments are rotated. Once an active log segment is rotated out (by closing it and creating a new active log segment file), it can be tiered by the existing tiered storage component, for the long-term. Fig Slack-KIP-1176-B. Produce batches are written to the page cache as usual, but active log segment files are aggressively tiered to S3 (possibly Express One Zone) in combined log segment files. The WAL acts as write-optimized S3 storage and the existing tiered storage uploads closed log segment files for long-term storage. Once all data of a given WAL object has been tiered, it can be deleted. The WAL only becomes necessary during topic partition leader-failovers, where the new leader replica bootstraps itself from the WAL. Alternatively, each topic partition can have one or more followers which actively reconstruct local log segments from the WAL, providing a faster failover. The general principle is to keep as much of Kafka unchanged as possible, only changing from the Kafka replication protocol to an S3 per-broker WAL. The priority is to avoid the need for heavy rework or reimplementation of logic such as idempotency, transactions and share groups integration. But it gives up elasticity and the additional architectural benefits that come from building on disaggregated storage. Having said all of the above. There are a lot of missing or hacky details that currently detract from the evolutionary goal. There is a lot of hand-waving when it comes to correctness too. It is not clear that this KIP will be able to deliver a low-disruption evolutionary design that is also correct, highly available and durable. Discussion in the mailing list is ongoing. Luke Chen remarked: “ the current availability story is weak… It’s not clear if the effort is still small once details on correctness, cost, cleanness are figured out. ”, and I have to agree. The second revision of KIP-1150 replaces the future object compaction logic by delegating long-term storage management to the existing tiered storage abstraction (like Slack’s KIP-1176). The idea is to: Remove Batch Coordinators from the read path. Avoid separate object compaction logic by delegating long-term storage management to tiered storage (which already exists). Rebuild per-partition log segments from combined objects in order to: Submit them for long-term tiering (works as a form of object compaction too). Serve consumer fetch requests. The entire write path becomes a three stage process: Stage 1 – Produce path, synchronous . Uploads multi-topic WAL Segments to S3 and sequences the batches, acknowledging to producers once committed. This is unchanged except SLSOs are now called WAL Segments. Stage 2 – Per-partition log segment file construction, asynchronous . Each broker is assigned a subset of topic partitions. The brokers download WAL segment byte ranges that host these assigned partitions and append to on-disk per-partition log segment files. Stage 3 – Tiered storage, asynchronous . The tiered storage architecture tiers the locally cached topic partition log segments files as normal. Stage 1 – The produce path The produce path is the same, but SLSOs are now called WAL Segments. Fig KIP-1150-rev2-A. Write path stage 1 (the synchronous part). Leaderless brokers upload multi-topic WAL Segments, then commit and sequence them via the Batch Coordinator. Stage 2 – Local per-partition segment caching The second stage is preparation for both: Stage 3, segment tiering (tiered storage). The read pathway for tailing consumers Fig KIP-1150-rev2-B. Stage 2 of the write-path where assigned brokers download WAL segments and append to local log segments. Each broker is assigned a subset of topic partitions. Each broker polls the BC to learn of new WAL segments. Each WAL segment that hosts any of the broker’s assigned topic partitions will be downloaded (at least the byte range of its assigned partitions). Once the download completes, the broker will inject record offsets as determined by the batch coordinator, and append the finalized batch to a local (per topic partition) log segment on-disk. At this point, a log segment file looks like a classic topic partition segment file. The difference is that they are not a source of durability, only a source for tiering and consumption. WAL segments remain in object storage until all batches of a segment have been tiered via tiered storage. Then WAL segments can be deleted. Stage 3 – Tiered storage Tiered storage continues to work as it does today (KIP-405), based on local log segments. It hopefully knows nothing of the Direct-to-S3 components and logic. Tiered segment metadata is stored in KRaft which allows for WAL segment deletion to be handled outside of the scope of tiered storage also. Fig KIP-1150-rev2-C. Tiered storage works as-is, based on local log segment files. Data is consumed from S3 topics from either: The local segments on-disk, populated from stage 2. Tiered log segments (traditional tiered storage read pathway) End-to-end latency of any given batch is therefore based on: Produce batch added to buffer. WAL Segment containing that batch written to S3. Batch coordinates submitted to the Batch Coordinator for sequencing. Producer request acknowledged Tail, untiered (fast path for tailing consumers) -> Replica downloads WAL Segment slice. Replica appends the batch to a local (per topic partition) log segment. Replica serves a consumer fetch request from the local log segment. Tiered (slow path for lagging consumers) -> Remote Log Manager downloads tiered log segment Replica serves a consumer fetch request from the downloaded log segment. Avoiding excessive reads to S3 will be important in stage 2 (when a broker is downloading WAL segment files for its assigned topic partitions). This KIP should standardize how topic partitions are laid out inside every WAL segment and perform partition-broker assignments based on that same order: Pick a single global topic partition order (based on a permutation of a topic partition id). Partition that order into contiguous slices, giving one slice per broker as its assignment ( (a broker may get multiple topic partitions, but they must be adjacent in the global order). Lay out every WAL segment in that same global order. That way, each broker’s partition assignment will occupy one contiguous block per WAL Segment, so each read broker needs only one byte-range read per WAL segment object (possibly empty if none of its partitions appear in that object). This reduces the number of range reads per broker when reconstructing local log segments. By integrating with tiered storage and reconstructing log segments, revision 2 moves to a more diskful design, where disks form a step in the write path to long term storage. It is also more stateful and sticky than revision 1, given that each topic partition is assigned a specific broker for log segment reconstruction, tiering and consumer serving. Revision 2 remains leaderless for producers, but leaderful for consumers. Therefore to avoid cross-AZ traffic for consumer traffic, it will rely on KIP-881: Rack-aware Partition Assignment for Kafka Consumers to ensure zone-local consumer assignment. This makes revision 2 a hybrid. By delegating responsibility to tiered storage, more of the direct-to-S3 workload must be handled by stateful brokers. It is less able to benefit from the elasticity of disaggregated storage. But it reuses more of existing Kafka. The third revision, which at the time of writing is a loose proposal in the mailing list, ditches the Batch Coordinator (BC) altogether. Most of the complexity of KIP-1150 centers around Batch Coordinator efficiency, failovers, scaling as well as idempotency, transactions and share groups logic. Revision 3 proposes to replace the BCs with “classic” topic partitions. The classic topic partition leader replicas will do the work of sequencing and storing the batch coordinates of their own data. The data itself would live in SLSOs (rev1)/WAL Segments (rev2) and ultimately, as tiered log segments (tiered storage). To make this clear, if as a user you create the topic Orders with one partition, then an actual topic Orders will be created with one partition. However, this will only be used for the sequencing of the Orders data and the storage of its metadata. The benefit of this approach is that all the idempotency and transaction logic can be reused in these “classic-ish” topic partitions. There will be code changes but less than Batch Coordinators. All the existing tooling of moving partitions around, failovers etc works the same way as classic topics. So replication continues to exist, but only for the metadata. Fig KIP-1150-rev3-A. Replicated partitions continue to exist, acting as per-partition sequencers and metadata stores (replacing batch coordinators). One wrinkle this adds is that there is no central place to manage the clean-up of WAL Segments. Therefore a WAL File Manager component would have responsibility for background cleanup of those WAL segment files. It would periodically check the status of tiering to discover when a WAL Segment can get deleted. Fig KIP-1150-rev3-B. The WAL File Manager is responsible for WAL Segment clean up The motivation behind this change to remove Batch Coordinators is to simplify implementation by reusing existing Kafka code paths (for idempotence, transactions, etc.). However, it also opens up a whole new set of challenges which must be discussed and debated, and it is not clear this third revision solves the complexity. Revision 3 now depends on the existing classic topics, with leader-follower replication. It moves a little further again towards the evolutionary path. It is curious to see Aiven productionizing its Kafka fork “Inkless”, which falls under the “revolutionary” umbrella, while pushing towards a more evolutionary stateful/sticky design in these later revisions of KIP-1150. Apache Kafka is approaching a decision point with long-term implications for its architecture and identity. The ongoing discussions around KIP-1150 revisions 1-3 and KIP-1176 are nominally framed around replication cost reduction, but the underlying issue is broader: how should Kafka evolve in a world increasingly shaped by disaggregated storage and elastic compute? At its core, the choice comes down to two paths. The evolutionary path seeks to fit S3 topics into Kafka’s existing leader-follower framework, reusing current abstractions such as tiered storage to minimize disruption to the codebase. The revolutionary path instead prioritizes the benefits of building directly on object storage. By delegating to shared object storage, Kafka can support an S3 topic serving layer which is stateless, elastic, and disposable. Scaling coming by adding and removing stateless workers rather than rebalancing stateful nodes. While maintaining Kafka’s existing workloads with classic leader-follower topics. While the intentions and goals of the KIPs clearly fall on a continuum of revolutionary to evolutionary, the reality in the mailing list discussions makes everything much less clear. The devil is in the details, and as the discussion advances, the arguments of “simplicity through reuse” start to strain. The reuse strategy is a retrofitting strategy which ironically could actually make the codebase harder to maintain in the long term. Kafka’s existing model is deeply rooted in leader-follower replication, with much of its core logic built around that assumption. Retrofitting direct-to-S3 into this model forces some “unnatural” design choices. Choices that would not be made otherwise (if designing a cloud-native solution). My own view aligns with the more revolutionary path in the form of KIP-1150 revision 1 . It doesn’t simply reduce cross-AZ costs, but fully embraces the architectural benefits of building on object storage. With additional broker roles and groups, Kafka could ultimately achieve a similar elasticity to WarpStream (and Confluent Freight Clusters). The approach demands more upfront engineering effort, may increase long-term maintenance complexity, but avoids tight coupling to the existing leader-follower architecture. Much depends on what kind of refactoring is possible to avoid the duplication of idempotency, transactions and share group logic. I believe the benefits justify the upfront cost and will help keep Kafka relevant in the decade ahead. In theory, both directions are defensible, ultimately it comes down to the specifics of each KIP. The details really matter. Goals define direction, but it’s the engineering details that determine the system’s actual properties. We know the revolutionary path involves big changes, but the evolutionary path comes with equally large challenges, where retrofitting may ultimately be more costly while simultaneously delivering less. The committers who maintain Kafka are cautious about large refactorings and code duplication, but are equally wary of hacks and complex code serving two needs. We need to let the discussions play out. My aim with this post has been to take a step back from "how to implement direct-to-S3 topics in Kafka", and think more about what we want Kafka to be. The KIPs represent the how, the engineering choices. Framed that way, I believe it is easier for the wider community to understand the KIPs, the stakes and the eventual decision of the committers, whichever way they ultimately decide to go. Revolutionary : Choose a direct-to-S3 topic design that maximizes the benefits of an object-storage architecture, with greater elasticity and lower operational complexity. However, in doing so, we may increase the implementation cost and possibly the long-term code maintenance too by maintaining two very different topic-models in the same project (leader-based replication and direct-to-S3). Evolutionary : Shoot for an evolutionary design that makes use of existing components to reduce the need for large refactoring or duplication of logic. However, by coupling to the existing architecture, we forfeit the extra benefits of object storage, focusing primarily on networking cost savings (in AWS and GCP). Through this coupling, we also run the risk of achieving the opposite: harder to maintain code by bending and contorting a second workload into an architecture optimized for something else. The term “Diskless” vs “Direct-to-S3” The Common Parts. Some approaches are shared across multiple implementations and proposals. Revolutionary: KIPs and real-world implementations Evolutionary: Slack’s KIP-1176 The Hybrid: balancing revolution with evolution Deciding Kafka’s future Avoiding the small file problem . Most designs are leaderless for producer traffic, allowing for any server to receive writes to any topic. To avoid uploading a multitude of tiny files, servers accumulate batches in a buffer until ready for upload. Before upload, the buffer is sorted by topic id and partition, to make compaction and some reads more efficient by ensuring that data of the same topic and same partition are in contiguous byte ranges. Pricing . The pricing of many (but not all) cloud object storage services penalize excessive requests, so it can be cheaper to roll-up whatever data has been received in the last X milliseconds and upload it with a single request. Leaderless: The same way that producer alignment works via metadata manipulation or using KIP-392 (fetch from follower) which can be used in a leaderless context. Leader-based: Zone-aware consumer group assignment as detailed in KIP-881: Rack-aware Partition Assignment for Kafka Consumers . The idea is to use consumer-to-partition assignment to ensure consumers are only assigned zone-local partitions (where the partition leader is located). KIP-392 (fetch-from-follower) , which is effective for designs that have followers (which isn’t always the case). WarpStream (as a reference, a kind of yardstick to compare against) KIP-1150 revision 1 Aiven Inkless (a Kafka-fork) Leaderless, stateless and diskless agents that handle Kafka clients, as well as compaction/cleaning work. Coordination layer : A central metadata store for sequencing, metadata storage and housekeeping coordination. Sequencing . Chooses the total ordering for writes, assigning offsets without gaps or duplicates. Metadata storage . Stores all metadata that maps partition offset ranges to S3 object byte ranges. Serving lookup requests . Serving requests for log offsets. Serving requests for batch coordinates (S3 object metadata). Partition CRUD operations . Serving requests for atomic operations (creating partitions, deleting topics, records, etc.) Data expiration Managing data expiry and soft deletion. Coordinating physical object deletion (performed by brokers). When a producer starts, it sends a Metadata request to learn which brokers are the leaders of each topic partition it cares about. The Metadata response contains a zone-local broker. The producer sends all Produce requests to this broker. The receiving broker accumulates batches of all partitions and uploads a shared log segment object (SLSO) to S3. The broker commits the SLSO by sending the metadata of the SLSO (the metadata is known as the batch coordinates) to the Batch Coordinator. The coordinates include the S3 object metadata and the byte ranges of each partition in the object. The Batch Coordinator assigns offsets to the written batches, persists the batch coordinates, and responds to the broker. The broker sends all associated acknowledgements to the producers. The consumer sends a Fetch request to the broker. The broker checks its cache and on a miss, the broker queries the Batch Coordinator for the relevant batch coordinates. The broker downloads the data from object storage. The broker injects the computed offsets and timestamps into the batches (the offsets being part of the batch coordinates). The broker constructs and sends the Fetch response to the Consumer. logs . Stores the Log Start Offset and High Watermark of each topic partition, with the primary key of topic_id and partition. files . Lists all the objects that host the topic partition data. batches . Maps Kafka batches to byte ranges in Files. producer_state . All the producer state needed for idempotency. Some other tables for housekeeping, and the merge work items. A broker opens a transaction and submits a table as an argument, containing the batch coordinates of the multiple topics uploaded in the combined file. The TVF logic creates a temporary table (logs_tmp) and fills it via a SELECT on the logs table, with the FOR UPDATE clause which obtains a row lock on each topic partition row in the logs table that matches the list of partitions being submitted. This ensures that other brokers that are competing to add batches to the same partition(s) queue up behind this transaction. This is a critical barrier that avoids inconsistency. These locks are held until the transaction commits or aborts. Next it, inside a loop, partition-by-partition, the TVF: Updates the producer state. Updates the high watermark of the partition (a row in the logs table). Inserts the batch coordinates into the batches table (sequencing and storing them). Commits the transaction. Maintain leader-based topic partitions (producers continue to write to leaders), but replace Kafka replication protocol with a per-broker S3-based write-ahead-log (WAL). Try to preserve existing partition replica code for idempotency and transactions. Reuse existing tiered storage for long-term S3 data management. Remove Batch Coordinators from the read path. Avoid separate object compaction logic by delegating long-term storage management to tiered storage (which already exists). Rebuild per-partition log segments from combined objects in order to: Submit them for long-term tiering (works as a form of object compaction too). Serve consumer fetch requests. Stage 1 – Produce path, synchronous . Uploads multi-topic WAL Segments to S3 and sequences the batches, acknowledging to producers once committed. This is unchanged except SLSOs are now called WAL Segments. Stage 2 – Per-partition log segment file construction, asynchronous . Each broker is assigned a subset of topic partitions. The brokers download WAL segment byte ranges that host these assigned partitions and append to on-disk per-partition log segment files. Stage 3 – Tiered storage, asynchronous . The tiered storage architecture tiers the locally cached topic partition log segments files as normal. Stage 3, segment tiering (tiered storage). The read pathway for tailing consumers The local segments on-disk, populated from stage 2. Tiered log segments (traditional tiered storage read pathway) Produce batch added to buffer. WAL Segment containing that batch written to S3. Batch coordinates submitted to the Batch Coordinator for sequencing. Producer request acknowledged Tail, untiered (fast path for tailing consumers) -> Replica downloads WAL Segment slice. Replica appends the batch to a local (per topic partition) log segment. Replica serves a consumer fetch request from the local log segment. Tiered (slow path for lagging consumers) -> Remote Log Manager downloads tiered log segment Replica serves a consumer fetch request from the downloaded log segment. Pick a single global topic partition order (based on a permutation of a topic partition id). Partition that order into contiguous slices, giving one slice per broker as its assignment ( (a broker may get multiple topic partitions, but they must be adjacent in the global order). Lay out every WAL segment in that same global order.
Yesterday (Oct 20, 2025), AWS had a major incident . A lot of companies had a really bad day. I’ve been chatting with friends at impacted companies and a few had similar stories. Their teams were calm while their execs were not. There was a lot of “DO SOMETHING!” or “WHY AREN’T YOU FREAKING OUT?!” Incidents happen, but panic doesn’t do anything to solve them. Fires aren’t emergencies for the fire department and incidents shouldn’t be for engineering responders either. In this case, very little was likely in their control. For all but the most trivial of systems, you’re not going to migrate off of AWS in the middle of an incident and expect a good outcome. But the principle stands: the worst thing you can do is freak out, especially when the incident is self-inflicted . Calm leadership is going to be better than panic every time. Imagine you lit your stove on fire and called the fire department to put it out. Which of these two responses would you prefer after the fire chief hops out of the truck: I’m sure the first response would have you questioning who hired this person to be the fire chief and would not give you a lot of confidence that your whole house wasn’t going to burn down. Similarly, the second response might let you be able to exhale and leave things to the professionals. Like fires, not all incidents are created equally. A pan on fire on the stove is much more routine than a large wildfire threatening a major city. I would argue the need for calm leadership gets MORE important the bigger the impact of the incident. A small incident might not attract much attention, while a big incident certainly will. In yesterday’s AWS incident it would most definitely not be reassuring to see whoever was in charge of the response making an announcement about how bad it was and how they were on the verge of a panic attack. Same goes for wildfires, and same goes for your incidents. When you’re in a leadership position during an incident, it serves no useful purpose to yell and scream or jump up and down or tear your hair out. Your responders want normal operations to be restored as much as you do, especially if it’s off-hours and they’ve had their free time or sleep interrupted. Shouting isn’t going to make them solve the problems any faster. Now and then someone else will react negatively to someone in the Incident Commander or the engineering management seat behaving in a calm and collected way during an incident. Usually it’s someone who is in the position to get yelled at by a customer, like an account rep or a non-technical executive. They’re under a lot of pressure because incidents can cost your customers money, can result in churn or lost sales opportunities, and all of those things can affect the bottom line or career prospects of people who have no control over the actual technical issue. So they’re stressed about a lot of factors that are not the incident per se, but might be downstream consequences of it, much in the same way you would be after setting your kitchen on fire. Sometimes this comes out as anger or frustration that you, the technology person responsible for this, are not similarly stressed. This is understandable. Keep in mind too that to them, they are utterly dependent on you and the tech team in this moment. They don’t know how to fix whatever’s broken, but they’re still going to hear about it from customers and prospects who might not be nice about it. This can cause a feeling of powerlessness that results in expressions of frustration. This is not a reason to change your demeanour. The best thing you can do is keep the updates flowing, even if there’s no new news. People find reassurance in knowing the team is still working the problem. Keep the stakeholder updates jargon-free and high-level. Show empathy that you understand the customer impact and the potential consequences of that, but reassure them that the focus right now is restoring normal operations and, if appropriate, you’ll be available to talk through what happened after the dust settles. A negative reaction to the calm approach is usually born out of a fear that ‘nothing is happening’. The truth is often a big part of incident response is waiting. Without the technical context to understand what’s really going on, some people just want to see something happening to reassure themselves. You can mitigate this with updates: Use your tools to broadcast this as often as you can and you’ll tamp down on a lot of that anxiety. Big cloud provider incidents are fairly rare. Self-inflicted incidents are not. The responders trying to fix a self-inflicted problem are very likely to be the ones who took an action that caused or contributed to the outage. Even in a healthy, blameless culture this is going to cause some stress and shame, and in a blameful, scapegoating culture it’s going to be incredibly difficult. Having their manager, manager’s manager, or department head freaking out is only going to compound the problem. Highly stressed people can’t think straight and make mistakes. Set a tone of calm, methodical response and you’ll get out from under the problem a lot faster. This might be culturally hard to achieve if you have very hands-on execs who want to be very involved, even when their involvement is counterproductive (hint: they don’t see it that way even if you do). But to the extent you can, have one communication channel for the responders and a separate one for updates for stakeholders. Mixing the two just distracts the responders and lengthens the time-to-resolution. The key to this being successful is not neglecting the updates. I recommend using a timer and giving an update every 10 minutes whether there’s news or not. People will appreciate it and your responders can focus on responding. I don’t want this to be a “how to run a good incident” post. There are plenty of comprehensive materials out there that do a better job than I would here. I do want to focus on how to be a good leader in the context of incidents. Like many things, it comes down to culture and practice. Firefighters train to stay calm when responding to fires. They’re methodical and follow well-known good practices. To make this as easy as possible on yourself: If you can, work “fires aren’t emergencies for the fire department” into your company lexicon and remind people that in the context of incidents, the engineering team are the fire department. " Wildfire " by USFWS/Southeast is marked with Public Domain Mark 1.0 . Like this? Please feel free to share it on your favourite social media or link site! Share it with friends! Hit subscribe to get new posts delivered to your inbox automatically. Feedback? Questions? Get in touch ! [Screaming] “Oh dear! A fire! Somebody get the hose! Agggh!” “We see the problem, we know how to deal with it, we’ll handle it and give you an update as soon as we can.” Here’s what we’ve done. Here’s what we’re doing (or waiting on the results of). Here’s what we’ll likely do next. Set a good example when you’re in the incident commander seat (whether in an official or ad-hoc capacity). Run practice incidents where the stakes are low. Build up runbooks and checklists to make response activities automatic. Define your response process and roles. Define your communication channels so everyone knows where to find updates.
So, I originally planned for this to be on my premium newsletter, but decided it was better to publish on my free one so that you could all enjoy it. If you liked it, please consider subscribing to support my work. Here’s $10 off the first year of annual . I’ve also recorded an episode about this on my podcast Better Offline ( RSS feed , Apple , Spotify , iHeartRadio ), it’s a little different but both handle the same information, just subscribe and it'll pop up. Over the last two years I have written again and again about the ruinous costs of running generative AI services, and today I’m coming to you with real proof. Based on discussions with sources with direct knowledge of their AWS billing, I am able to disclose the amounts that AI firms are spending, specifically Anthropic and AI coding company Cursor, its largest customer . I can exclusively reveal today Anthropic’s spending on Amazon Web Services for the entirety of 2024, and for every month in 2025 up until September, and that that Anthropic’s spend on compute far exceeds that previously reported. Furthermore, I can confirm that through September, Anthropic has spent more than 100% of its estimated revenue (based on reporting in the last year) on Amazon Web Services, spending $2.66 billion on compute on an estimated $2.55 billion in revenue. Additionally, Cursor’s Amazon Web Services bills more than doubled from $6.2 million in May 2025 to $12.6 million in June 2025, exacerbating a cash crunch that began when Anthropic introduced Priority Service Tiers, an aggressive rent-seeking measure that begun what I call the Subprime AI Crisis , where model providers begin jacking up the prices on their previously subsidized rates. Although Cursor obtains the majority of its compute from Anthropic — with AWS contributing a relatively small amount, and likely also taking care of other parts of its business — the data seen reveals an overall direction of travel, where the costs of compute only keep on going up . Let’s get to it. In February of this year, The information reported that Anthropic burned $5.6 billion in 2024, and made somewhere between $400 million and $600 million in revenue: While I don’t know about prepayment for services, I can confirm from a source with direct knowledge of billing that Anthropic spent $1.35 billion on Amazon Web Services in 2024, and has already spent $2.66 billion on Amazon Web Services through the end of September. Assuming that Anthropic made $600 million in revenue, this means that Anthropic spent $6.2 billion in 2024, leaving $4.85 billion in costs unaccounted for. The Information’s piece also brings up another point: Before I go any further, I want to be clear that The Information’s reporting is sound, and I trust that their source (I have no idea who they are or what information was provided) was operating in good faith with good data. However, Anthropic is telling people it spent $1.5 billion on just training when it has an Amazon Web Services bill of $1.35 billion, which heavily suggests that its actual compute costs are significantly higher than we thought, because, to quote SemiAnalysis, “ a large share of Anthropic’s spending is going to Google Cloud .” I am guessing, because I do not know, but with $4.85 billion of other expenses to account for, it’s reasonable to believe Anthropic spent an amount similar to its AWS spend on Google Cloud. I do not have any information to confirm this, but given the discrepancies mentioned above, this is an explanation that makes sense. I also will add that there is some sort of undisclosed cut that Amazon gets of Anthropic’s revenue, though it’s unclear how much. According to The Information , “Anthropic previously told some investors it paid a substantially higher percentage to Amazon [than OpenAI’s 20% revenue share with Microsoft] when companies purchase Anthropic models through Amazon.” I cannot confirm whether a similar revenue share agreement exists between Anthropic and Google. This also makes me wonder exactly where Anthropic’s money is going. Anthropic has, based on what I can find, raised $32 billion in the last two years, starting out 2023 with a $4 billion investment from Amazon from September 2023 (bringing the total to $37.5 billion), where Amazon was named its “primary cloud provider” nearly eight months after Anthropic announced Google was Anthropic’s “cloud provider.,” which Google responded to a month later by investing another $2 billion on October 27 2023 , “involving a $500 million upfront investment and an additional $1.5 billion to be invested over time,” bringing its total funding from 2023 to $6 billion. In 2024, it would raise several more rounds — one in January for $750 million, another in March for $884.1 million, another in May for $452.3 million, and another $4 billion from Amazon in November 2024 , which also saw it name AWS as Anthropic’s “primary cloud and training partner,” bringing its 2024 funding total to $6 billion. In 2025 so far, it’s raised a $1 billion round from Google , a $3.5 billion venture round in March, opened a $2.5 billion credit facility in May, and completed a $13 billion venture round in September, valuing the company at $183 billion . This brings its total 2025 funding to $20 billion. While I do not have Anthropic’s 2023 numbers, its spend on AWS in 2024 — around $1.35 billion — leaves (as I’ve mentioned) $4.85 billion in costs that are unaccounted for. The Information reports that costs for Anthropic’s 521 research and development staff reached $160 million in 2024 , leaving 394 other employees unaccounted for (for 915 employees total), and also adding that Anthropic expects its headcount to increase to 1900 people by the end of 2025. The Information also adds that Anthropic “expects to stop burning cash in 2027.” This leaves two unanswered questions: An optimist might argue that Anthropic is just growing its pile of cash so it’s got a warchest to burn through in the future, but I have my doubts. In a memo revealed by WIRED , Anthropic CEO Dario Amodei stated that “if [Anthropic wanted] to stay on the frontier, [it would] gain a very large benefit from having access to this capital,” with “this capital” referring to money from the Middle East. Anthropic and Amodei’s sudden willingness to take large swaths of capital from the Gulf States does not suggest that it’s not at least a little desperate for capital, especially given Anthropic has, according to Bloomberg , “recently held early funding talks with Abu Dhabi-based investment firm MGX” a month after raising $13 billion . In my opinion — and this is just my gut instinct — I believe that it is either significantly more expensive to run Anthropic than we know, or Anthropic’s leaked (and stated) revenue numbers are worse than we believe. I do not know one way or another, and will only report what I know. So, I’m going to do this a little differently than you’d expect, in that I’m going to lay out how much these companies spent, and draw throughlines from that spend to its reported revenue numbers and product announcements or events that may have caused its compute costs to increase. I’ve only got Cursor’s numbers from January through September 2025, but I have Anthropic’s AWS spend for both the entirety of 2024 and through September 2025. So, this term is one of the most abused terms in the world of software, but in this case , I am sticking to the idea that it means “month times 12.” So, if a company made $10m in January, you would say that its annualized revenue is $120m. Obviously, there’s a lot of (when you think about it, really obvious) problems with this kind of reporting — and thus, you only ever see it when it comes to pre-IPO firms — but that’s besides the point. I give you this explanation because, when contrasting Anthropic’s AWS spend with its revenues, I’ve had to work back from whatever annualized revenues were reported for that month. Anthropic’s 2024 revenues are a little bit of a mystery, but, as mentioned above, The Information says it might be between $400 million and $600 million. Here’s its monthly AWS spend. I’m gonna be nice here and say that Anthropic made $600 million in 2024 — the higher end of The Information’s reporting — meaning that it spent around 226% of its revenue ($1.359 billion) on Amazon Web Services. [Editor's note: this copy originally had incorrect maths on the %. Fixed now.] Thanks to my own analysis and reporting from outlets like The Information and Reuters, we have a pretty good idea of Anthropic’s revenues for much of the year. That said, July, August, and September get a little weirder, because we’re relying on “almosts” and “approachings,” as I’ll explain as we go. I’m also gonna do an analysis on a month-by-month basis, because it’s necessary to evaluate these numbers in context. In this month, Anthropic’s reported revenue was somewhere from $875 million to $1 billion annualized , meaning either $72.91 million or $83 million for the month of January. In February, as reported by The Information , Anthropic hit $1.4 billion annualized revenue, or around $116 million each month. In March, as reported by Reuters , Anthropic hit $2 billion in annualized revenue, or $166 million in revenue. Because February is a short month, and the launch took place on February 24 2025, I’m considering the launches of Claude 3.7 Sonnet and Claude Code’s research preview to be a cost burden in the month of March. And man, what a burden! Costs increased by $59.1 million, primarily across compute categories, but with a large ($2 million since January) increase in monthly costs for S3 storage. I estimate, based on a 22.4% compound growth rate, that Anthropic hit around $2.44 billion in annualized revenue in April, or $204 million in revenue. Interestingly, this was the month where Anthropic launched its $100 and $200 dollar a month “Max” plan s, and it doesn’t seem to have dramatically increased its costs. Then again, Max is also the gateway to things like Claude Code, which I’ll get to shortly. In May, as reported by CNBC , Anthropic hit $3 billion in annualized revenue, or $250 million in monthly average revenue. This was a big month for Anthropic, with two huge launches on May 22 2025 — its new, “more powerful” models Claude Sonnet and Opus 4, as well as the general availability of its AI coding environment Claude Code. Eight days later, on May 30 2025, a page on Anthropic's API documentation appeared for the first time: " Service Tiers ": Accessing the priority tier requires you to make an up-front commitment to Anthropic , and said commitment is based on a number of months (1, 3, 6 or 12) and the number of input and output tokens you estimate you will use each minute. As I’ll get into in my June analysis, Anthropic’s Service Tiers exist specifically for it to “guarantee” your company won’t face rate limits or any other service interruptions, requiring a minimum spend, minimum token throughput, and for you to pay higher rates when writing to the cache — which is, as I’ll explain, a big part of running an AI coding product like Cursor. Now, the jump in costs — $65.1 million or so between April and May — likely comes as a result of the final training for Sonnet and Opus 4, as well as, I imagine, some sort of testing to make sure Claude Code was ready to go. In June, as reported by The Information, Anthropic hit $4 billion in annualized revenue, or $333 million. Anthropic’s revenue spiked by $83 million this month, and so did its costs by $34.7 million. I have, for a while, talked about the Subprime AI Crisis , where big tech and companies like Anthropic, after offering subsidized pricing to entice in customers, raise the rates on their customers to start covering more of their costs, leading to a cascade where businesses are forced to raise their prices to handle their new, exploding costs. And I was god damn right. Or, at least, it sure looks like I am. I’m hedging, forgive me. I cannot say for certain, but I see a pattern. It’s likely the June 2025 spike in revenue came from the introduction of service tiers, which specifically target prompt caching, increasing the amount of tokens you’re charged for as an enterprise customer based on the term of the contract, and your forecast usage. Per my reporting in July : Cursor, as Anthropic’s largest client (the second largest being Github Copilot), represents a material part of its revenue, and its surging popularity meant it was sending more and more revenue Anthropic’s way. Anysphere, the company that develops Cursor, hit $500 million annualized revenue ($41.6 million) by the end of May , which Anthropic chose to celebrate by increasing its costs. On June 16 2025, Cursor launched a $200-a-month “Ultra” plan , as well as dramatic changes to its $20-a-month Pro pricing that, instead of offering 500 “fast” responses using models from Anthropic and OpenAI, now effectively provided you with “at least” whatever you paid a month (so $20-a-month got at least $20 of credit), massively increasing the costs for users , with one calling the changes a “rug pull” after spending $71 in a single day . As I’ll get to later in the piece, Cursor’s costs exploded from $6.19 million in May 2025 to $12.67 million in June 2025, and I believe this is a direct result of Anthropic’s sudden and aggressive cost increases. Similarly, Replit, another AI coding startup, moved to “Effort-Based Pricing” on June 18 2025 . I have not got any information around its AWS spend. I’ll get into this a bit later, but I find this whole situation disgusting. In July, as reported by Bloomberg , Anthropic hit $5 billion in annualized revenue, or $416 million. While July wasn’t a huge month for announcements, it was allegedly the month that Claude Code was generating “nearly $400 million in annualized revenue,” or $33.3 million ( according to The Information , who says Anthropic was “approaching” $5 billion in annualized revenue - which likely means LESS than that - but I’m going to go with the full $5 billion annualized for sake of fairness. There’s roughly an $83 million bump in Anthropic’s revenue between June and July 2025, and I think Claude Code and its new rates are a big part of it. What’s fascinating is that cloud costs didn’t increase too much — by only $1.8 million, to be specific. In August, according to Anthropic, its run-rate “ reached over $5 billion ,” or in or around $416 million. I am not giving it anything more than $5 billion, especially considering in July Bloomberg’s reporting said “about $5 billion.” Costs grew by $60.5 this month, potentially due to the launch of Claude Opus 4.1 , Anthropic’s more aggressively expensive model, though revenues do not appear to have grown much along the way. Yet what’s very interesting is that Anthropic — starting August 28 — launched weekly rate limits on its Claude Pro and Max plans. I wonder why? Oh fuck! Look at that massive cost explosion! Anyway, according to Reuters, Anthropic’s run rate is “approaching $7 billion” in October , and for the sake of fairness , I am going to just say it has $7 billion annualized, though I believe this number to be lower. “Approaching” can mean a lot of different things — $6.1 billion, $6.5 billion — and because I already anticipate a lot of accusations of “FUD,” I’m going to err on the side of generosity. If we assume a $6.5 billion annualized rate, that would make this month’s revenue $541.6 million, or 95.8% of its AWS spend. Nevertheless, Anthropic’s costs exploded in the space of a month by $135.2 million (35%) - likely due to the fact that users, as I reported in mid-July, were costing it thousands or tens of thousands of dollars in compute , a problem it still faces to this day, with VibeRank showing a user currently spending $51,291 in a calendar month on a $200-a-month subscription . If there were other costs, they likely had something to do with the training runs for the launches of Sonnet 4.5 on September 29 2025 and Haiku 4.5 in October 2025 . While these costs only speak to one part of its cloud stack — Anthropic has an unknowable amount of cloud spend on Google Cloud, and the data I have only covers AWS — it is simply remarkable how much this company spends on AWS, and how rapidly its costs seem to escalate as it grows. Though things improved slightly over time — in that Anthropic is no longer burning over 200% of its revenue on AWS alone — these costs have still dramatically escalated, and done so in an aggressive and arbitrary manner. So, I wanted to visualize this part of the story, because I think it’s important to see the various different scenarios. THE NUMBERS I AM USING ARE ESTIMATES CALCULATED BASED ON 25%, 50% and 100% OF THE AMOUNTS THAT ANTHROPIC HAS SPENT ON AMAZON WEB SERVICES THROUGH SEPTEMBER. I apologize for all the noise, I just want it to be crystal clear what you see next. As you can see, all it takes is for Anthropic to spend (I am estimating) around 25% of its Amazon Web Services bills (for a total of around $3.33 billion in compute costs through the end of September) to savage any and all revenue ($2.55 billion) it’s making. Assuming Anthropic spends half of its AWS spend on Google Cloud, this number climbs to $3.99 billion, and if you assume - and to be clear, this is an estimate - that it spends around the same on both Google Cloud and AWS, Anthropic has spent $5.3 billion on compute through the end of September. I can’t tell you which it is, just that we know for certain that Anthropic is spending money on Google Cloud, and because Google owns 14% of the company — rivalling estimates saying Amazon owns around 15-19% — it’s fair to assume that there’s a significant spend. I have sat with these numbers for a great deal of time, and I can’t find any evidence that Anthropic has any path to profitability outside of aggressively increasing the prices on their customers to the point that its services will become untenable for consumers and enterprise customers alike. As you can see from these estimated and reported revenues, Anthropic’s AWS costs appear to increase in a near-linear fashion with its revenues, meaning that the current pricing — including rent-seeking measures like Priority Service Tiers — isn’t working to meet the burden of its costs. We do not know its Google Cloud spend, but I’d be shocked if it was anything less than 50% of its AWS bill. If that’s the case, Anthropic is in real trouble - the cost of the services underlying its business increase the more money they make. It’s becoming increasingly apparent that Large Language Models are not a profitable business. While I cannot speak to Amazon Web Services’ actual costs, it’s making $2.66 billion from Anthropic, which is the second largest foundation model company in the world. Is that really worth $105 billion in capital expenditures ? Is that really worth building a giant 1200 acre data center in Indiana with 2.2GW of electricity? What’s the plan, exactly? Let Anthropic burn money for the foreseeable future until it dies, and then pick up the pieces? Wait until Wall Street gets mad at you and then pull the plug? Who knows. But let’s change gears and talk about Cursor — Anthropic’s largest client and, at this point, a victim of circumstance. Amazon sells Anthropic’s models through Amazon Bedrock , and I believe that AI startups are compelled to spend some of their AI model compute costs through Amazon Web Services. Cursor also sends money directly to Anthropic and OpenAI, meaning that these costs are only one piece of its overall compute costs. In any case, it’s very clear that Cursor buys some degree of its Anthropic model spend through Amazon. I’ll also add that Tom Dotan of Newcomer reported a few months ago that an investor told him that “Cursor is spending 100% of its revenue on Anthropic.” Unlike Anthropic, we lack thorough reporting of the month-by-month breakdown of Cursor’s revenues. I will, however, mention them in the month I have them. For the sake of readability — and because we really don’t have much information on Cursor’s revenues beyond a few months — I’m going to stick to a bullet point list. As discussed above, Cursor announced (along with their price change and $200-a-month plan) several multi-year partnerships with xAI, Anthropic, OpenAI and Google, suggesting that it has direct agreements with Anthropic itself versus one with AWS to guarantee “this volume of compute at a predictable price.” Based on its spend with AWS, I do not see a strong “minimum” spend that would suggest that they have a similar deal with Amazon — likely because Amazon handles more than its infrastructure than just compute, but incentivizes it to spend on Anthropic’s models through AWS by offering discounts, something I’ve confirmed with a source. In any case, here’s what Cursor spent on AWS. When I wrote that Anthropic and OpenAI had begun the Subprime AI Crisis back in July, I assumed that the increase in costs was burdensome, but having the information from its AWS bills, it seems that Anthropic’s actions directly caused Cursor’s costs to explode by over 100%. While I can’t definitively say “this is exactly what did it,” the timelines match up exactly, the costs have never come down, Amazon offers provisioned throughput , and, more than likely, Cursor needs to keep a standard of uptime similar to that of Anthropic’s own direct API access. If this is what happened, it’s deeply shameful. Cursor, Anthropic’s largest customer , in the very same month it hit $500 million in annualized revenue, immediately had its AWS and Anthropic-related costs explode to the point that it had to dramatically reduce the value of its product just as it hit the apex of its revenue growth. It’s very difficult to see Service Tiers as anything other than an aggressive rent-seeking maneuver. Yet another undiscussed part of the story is that the launch of Claude 4 Opus and Sonnet — and the subsequent launch of Service Tiers — coincided with the launch of Claude Code , a product that directly competes with Cursor, without the burden of having to pay itself for the cost of models or, indeed, having to deal with its own “Service Tiers.” Anthropic may have increased the prices on its largest client at the time it was launching a competitor, and I believe that this is what awaits any product built on top of OpenAI or Anthropic’s models. I realize this has been a long, number-stuffed article, but the long-and-short of it is simple: Anthropic is burning all of its revenue on compute, and Anthropic will willingly increase the prices on its customers if it’ll help it burn less money, even though that doesn’t seem to be working. What I believe happened to Cursor will likely happen to every AI-native company, because in a very real sense, Anthropic’s products are a wrapper for its own models, except it only has to pay the (unprofitable) costs of running them on Amazon Web Services and Google Cloud. As a result, both OpenAI and Anthropic can (and may very well!) devour the market of any company that builds on top of their models. OpenAI may have given Cursor free access to its GPT-5 models in August, but a month later on September 15 2025 it debuted massive upgrades to its competitive “Codex” platform. Any product built on top of an AI model that shows any kind of success can be cloned immediately by OpenAI and Anthropic, and I believe that we’re going to see multiple price increases on AI-native companies in the next few months. After all, OpenAI already has its own priority processing product, which it launched shortly after Anthropic’s in June . The ultimate problem is that there really are no winners in this situation. If Anthropic kills Cursor through aggressive rent-seeking, that directly eats into its own revenues. If Anthropic lets Cursor succeed, that’s revenue , but it’s also clearly unprofitable revenue . Everybody loses, but nobody loses more than Cursor’s (and other AI companies’) customers. I’ve come away from this piece with a feeling of dread. Anthropic’s costs are out of control, and as things get more desperate, it appears to be lashing out at its customers, both companies like Cursor and Claude Code customers facing weekly rate limits on their more-powerful models who are chided for using a product they pay for. Again, I cannot say for certain, but the spike in costs is clear, and it feels like more than a coincidence to me. There is no period of time that I can see in the just under two years of data I’ve been party to that suggests that Anthropic has any means of — or any success doing — cost-cutting, and the only thing this company seems capable of doing is increasing the amount of money it burns on a monthly basis. Based on what I have been party to, the more successful Anthropic becomes, the more its services cost. The cost of inference is clearly increasing for customers , but based on its escalating monthly costs, the cost of inference appears to be high for Anthropic too, though it’s impossible to tell how much of its compute is based on training versus running inference. In any case, these costs seem to increase with the amount of money Anthropic makes, meaning that the current pricing of both subscriptions and API access seems unprofitable, and must increase dramatically — from my calculations, a 100% price increase might work, but good luck retaining every single customer and their customers too! — for this company to ever become sustainable. I don’t think that people would pay those prices. If anything, I think what we’re seeing in these numbers is a company bleeding out from costs that escalate the more that its user base grows. This is just my opinion, of course. I’m tired of watching these companies burn billions of dollars to destroy our environment and steal from everybody. I’m tired that so many people have tried to pretend there’s a justification for burning billions of dollars every year, clinging to empty tropes about how this is just like Uber or Amazon Web Services , when Anthropic has built something far more mediocre. Mr. Amodei, I am sure you will read this piece, and I can make time to chat in person on my show Better Offline. Perhaps this Friday? I even have some studio time on the books. I do not have all the answers! I am going to do my best to go through the information I’ve obtained and give you a thorough review and analysis. This information provides a revealing — though incomplete — insight into the costs of running Anthropic and Cursor, but does not include other costs, like salaries and compute obtained from other providers. I cannot tell you (and do not have insight into) Anthropic’s actual private moves. Any conclusions or speculation I make in this article will be based on my interpretations of the information I’ve received, as well as other publicly-available information. I have used estimates of Anthropic’s revenue based on reporting across the last ten months. Any estimates I make are detailed and they are brief. These costs are inclusive of every product bought on Amazon Web Services, including EC2, storage and database services (as well as literally everything else they pay for). Anthropic works with both Amazon Web Services and Google Cloud for compute. I do not have any information about its Google Cloud spend. The reason I bring this up is that Anthropic’s revenue is already being eaten up by its AWS spend. It’s likely billions more in the hole from Google Cloud and other operational expenses. I have confirmed with sources that every single number I give around Anthropic and Cursor’s AWS spend is the final cash paid to Amazon after any discounts or credits. While I cannot disclose the identity of my source, I am 100% confident in these numbers, and have verified their veracity with other sources. Where is the rest of Anthropic’s money going? How will it “stop burning cash” when its operational costs explode as its revenue increases? January 2024 - $52.9 million February 2024 - $60.9 million March 2024 - $74.3 million April 2024 - $101.1 million May 2024 - $100.1 million June 2024 - $101.8 million July 2024 - $118.9 million August 2024 - $128.8 million September 2024 - $127.8 million October 2024 - $169.6 million November 2024 - $146.5 million December 2024 - $176.1 million January 2025 - $1.459 million This, apparently, is the month that Cursor hit $100 million annualized revenue — or $8.3 million, meaning it spent 17.5% of its revenue on AWS. February 2025 - $2.47 million March 2025 - $4.39 million April 2025 - $4.74 million Cursor hit $200 million annualized ($16.6 million) at the end of March 2025 , according to The Information, working out to spending 28% of its revenue on AWS. May 2025 - $6.19 million June 2025 - $12.67 million So, Bloomberg reported that Cursor hit $500 million on June 5 2025 , along with raising a $900 million funding round. Great news! Turns out it’d need to start handing a lot of that to Anthropic. This was, as I’ve discussed above, the month when Anthropic forced it to adopt “Service Tiers”. I go into detail about the situation here , but the long and short of it is that Anthropic increased the amount of tokens you burned by writing stuff to the cache (think of it like RAM in a computer), and AI coding startups are very cache heavy, meaning that Cursor immediately took on what I believed would be massive new costs. As I discuss in what I just linked, this led Cursor to aggressively change its product, thereby vastly increasing its customers’ costs if they wanted to use the same service. That same month, Cursor’s AWS costs — which I believe are the minority of its cloud compute costs — exploded by 104% (or by $6.48 million), and never returned to their previous levels. It’s conceivable that this surge is due to the compute-heavy nature of the latest Claude 4 models released that month — or, perhaps, Cursor sending more of its users to other models that it runs on Bedrock. July 2025 - $15.5 million As you can see, Cursor’s costs continue to balloon in July, and I am guessing it’s because of the Service Tiers situation — which, I believe, indirectly resulted in Cursor pushing more users to models that it runs on Amazon’s infrastructure. August 2025 - $9.67 million So, I can only guess as to why there was a drop here. User churn? It could be the launch of GPT-5 on Cursor , which gave users a week of free access to OpenAI’s new models. What’s also interesting is that this was the month when Cursor announced that its previously free “auto” model (where Cursor would select the best available premium model or its own model) would now bill at “ competitive token rates ,” by which I mean it went from charging nothing to $1.25 per million input and $6 per million output tokens. This change would take effect on September 15 2025. On August 10 2025 , Tom Dotan of Newcomer reported that Cursor was “well above” $500 million in annualized revenue based on commentary from two sources. September 2025 - $12.91 million Per the above, this is the month when Cursor started charging for its “auto” model.
Most of us use infrastructure provided by companies like DigitalOcean and AWS. Some of us choose to work on that infrastructure. And some of us are really built different and choose to build all that infrastructure from scratch . This post is a real treat for me to bring you. I met Diana through a friend of mine, and I've gotten some peeks behind the curtain as she builds a new hosting provider . So I was thrilled that she agreed to an interview to let me share some of that with you all. So, here it is: a peek behind the curtain of a new hosting provider, in a very early stage. This is the interview as transcribed (any errors are mine), with a few edits as noted for clarity. Nicole: Hi, Diana! Thanks for taking the time to do this. Can you start us off by just telling us a little bit about who you are and what your company does? Diana: So I'm Diana, I'm trans, gay, AuDHD and I like to create, mainly singing and 3D printing. I also have dreams of being the change I want to see in the world. Since graduating high school, all infrastructure has become a passion for me. Particularly networking and computer infrastructure. From your home internet connection to data centers and everything in between. This has led me to create Andromeda Industries and the dba Gigabit.Host. Gigabit.Host is a hosting service where the focus is affordable and performant host for individuals, communities, and small businesses. Let's start out talking about the business a little bit. What made you decide to start a hosting company? The lack of performance for a ridiculous price. The margins on hosting is ridiculous, it's why the majority of the big tech companies' revenue comes from their cloud offerings. So my thought has been why not take that and use it more constructively. Instead of using the margins to crush competition while making the rich even more wealthy, use those margins for good. What is the ethos of your company? To use the net profits from the company to support and build third spaces and other low return/high investment cost ventures. From my perspective, these are the types of ideas that can have the biggest impact on making the world a better place. So this is my way of adopting socialist economic ideas into the systems we currently have and implementing the changes. How big is the company? Do you have anyone else helping out? It’s just me for now, though the plan is to make it into a co-op or unionized business. I have friends and supporters of the project, giving feedback and suggesting improvements. What does your average day-to-day look like? I go to my day job during the week, and work on the company in my spare time. I have alerts and monitors that warn me when something needs addressing, overall operations are pretty hands off. You're a founder, and founders have to wear all the hats. How have you managed your work-life balance while starting this? At this point it’s more about balancing my job, working on the company, and taking care of my cat. It's unfortunately another reason that I started this endeavor, there just aren't spaces I'd rather be than home, outside of a park or hiking. All of my friends are online and most say the same, where would I go? Hosting businesses can be very capital intensive to start. How do you fund it? Through my bonuses and stocks currently, also through using more cost effective brands that are still reliable and performant. What has been the biggest challenge of operating it from a business perspective? Getting customers. I'm not a huge fan of marketing and have been using word of mouth as the primary method of growing the business. Okay, my part here then haha. If people want to sign up, how should they do that? If people are interested in getting service, they can request an invite through this link: https://portal.gigabit.host/invite/request . What has been the most fun part of running a hosting company? Getting to actually be hands on with the hardware and making it as performant as possible. It scratches an itch of eking out every last drop of performance. Also not doing it because it's easy, doing it because I thought it would be easy. What has been the biggest surprise from starting Gigabit.Host? How both complex and easy it has been at the same time. Also how much I've been learning and growing through starting the company. What're some of the things you've learned? It's been learning that wanting it to be perfect isn't realistic, taking the small wins and building upon and continuing to learn as you go. My biggest learning challenge was how to do frontend work with Typescript and styling, the backend code has been easy for me. The frontend used to be my weakness, now it could be better, and as I add new features I can see it continuing to getting better over time. Now let's talk a little bit about the tech behind the scenes. What does the tech stack look like? Next.js and Typescript for the front and backend. Temporal is used for provisioning and task automation. Supabase is handling user management Proxmox for the hardware virtualization How do you actually manage this fleet of VMs? For the customer side we only handle the initial provisioning, then the customer is free to use whatever tool they choose. The provisioning of the VMs is handled using Go and Temporal. For our internal services we use Ansible and automation scripts. [Nicole: the code running the platform is open source, so you can take a look at how it's done in the repository !] How do your technical choices and your values as a founder and company work together? They are usually in sync, the biggest struggle has been minimizing cost of hardware. While I would like to use more advanced networking gear, it's currently cost prohibitive. Which choices might you have made differently? [I would have] gathered more capital before getting started. Though that's me trying to be a perfectionist, when the reality is buy as little as possible and use what you have when able. This seems like a really hard business to be in since you need reliability out of the gate. How have you approached that? Since I've been self-funding this endeavor, I've had to forgo high availability for now due to costs. To work around that I've gotten modern hardware for the critical parts of the infrastructure. This so far has enabled us to achieve 90%+ uptime, with the current goal to add redundancy as able to do so. What have been the biggest technical challenges you've run into? Power and colocation costs. Colocation is expensive in Seattle. Around 8x the cost of my previous colo in Atlanta, GA. Power has been the second challenge, running modern hardware means higher power requirements. Most data centers outside of hyperscalers are limited to 5 to 10 kW per rack. This limits the hardware and density, thankfully for now it [is] a future struggle. Huge thanks to Diana for taking the time out of her very busy for this interview! And thank you to a few friends who helped me prepare for the interview.
In the previous post, I explored a Parquet on S3 design with tombstones for constant time deletes and a CAS updated manifest for snapshot isolation. This post extends that design. The focus is in file delete operations where we replace a Parquet row group and publish a new footer using S3 Multipart Upload (MPU) and UploadPartCopy without having to download and rebuild unchanged bytes. We preserve the same system model with immutable data files and a manifest pointer that serializes visibility.
Parquet is excellent for analytical workloads. Columnar layout, aggressive compression, predicate pushdown, but deletes require rewriting entire files. Systems like Apache Iceberg and Delta Lake solve this by adding metadata layers that track delete files separately from data files. But what if, for fun, we built something (arguably) simpler? S3 now has conditional writes (If-Match, If-None-Match) that enable atomic operations without external coordination. Let’s explore how we might build a columnar table format on S3 that gets most of Parquet’s benefits while supporting constant-time deletes.
PostgreSQL handles large JSON payloads reasonably well until you start updating or deleting them frequently. Once payloads cross the 8 KB TOAST threshold and churn becomes high, autovacuum can dominate your I/O budget and cause other issues. I have been exploring the idea of moving older JSON data (read: cold data) to Parquet on S3 while keeping recent data hot in PostgreSQL daily partitions, then simply dropping those partitions instead of running expensive DELETE operations and subsequent vacuum cycles.
This week on the People and Blogs series we have an interview with Kris Howard, whose blog can be found at web-goddess.org . Tired of RSS? Read this in your browser or sign up for the newsletter . The People and Blogs series is supported by Ilja Panić and the other 120 members of my "One a Month" club. If you enjoy P&B, consider becoming one for as little as 1 dollar a month. Heya! I'm Kris Howard, and as of September 2025, I've been blogging continuously for 25 years. 😳 I grew up in rural Indiana, had a couple of brief working stints in London and Munich, and have lived nearly all of my adult life in Sydney, Australia with my husband Rodd (aka The Snook). I started my career as a web developer in the dotcom boom and eventually went on to manage projects and lead teams. I ended up in Developer Relations for AWS, where I travelled all over telling folks why they should use the cloud. It was a lot of fun until everything became all about AI. In 2024 I joined the Snook in early retirement (at age 47), and for the past year I've been reading books (remember books?!), hanging out with friends, indulging in my craft hobbies (knitting, sewing, and now weaving), volunteering, travelling around Australia, and generally trying to spend a lot less time in front of a computer. I was on work-study in university, and they assigned me to work in the dining hall. That sucked , so I scrambled to find something better. I ended up working in the Computer Department just as my university was connecting everyone to the Internet. I learned HTML and built websites for myself and my dorm, and in 1996 I launched a fan site for my favourite author that's been running ever since. Fun trivia: I hosted that on roalddahl.org until the Dahl estate reached out about acquiring the domain name. I happily transferred it to them, and in return I got to meet Felicity Dahl and visit his writing hut in Buckinghamshire! By that time I had left uni and was working as a web developer in London. I started testing out Blogger (which back then would actually FTP static files to your own webhosting!) in September 2000 and launched my blog properly a couple months later at web-goddess.co.uk. The name was a bit of a joke, a nickname that a Scottish friend gave me because of how much time I spent online. Originally my goal was just to document my life overseas for friends and family, and to share silly things I found on the Internet. When the dotcom crash hit at the end of 2001, I ended up moving to Sydney with my Australian boyfriend and changed the domain to web-goddess.org instead. For a long time I ran off my own custom PHP CMS (which I even distributed ) before moving to Wordpress ten years ago. My blogging energy has waxed and waned over the years. There were long stretches when I just syndicated content from places like Instagram and Google Reader to the blog rather than write. Since retiring from full-time work I've really renewed my focus on it as the home for all my content online. I've migrated all my posts from Twitter, Instagram, and Facebook to the blog, and I now syndicate out from the blog to Mastodon and Bluesky. Every day I go back through my archives and clean up broken links. (So much linkrot.) I miss the Web of the early 2000s and I'm trying to keep my little corner of it thriving. My process is fairly casual, and I don't usually write extensive drafts before publishing. Back in the olden days, I used it basically like Twitter - just dashing off a thought or sharing a link multiple times a day. Nowadays I tend to be a bit more deliberate with what I share. Occasionally I'll get Rodd to review something, especially if it touches on our personal life (like my posts about our retirement finances ). I've occasionally dabbled with creating a series of posts on a theme, like when we tried cooking our way through the Jamie's 30 Minute Meals cookbook (still by far my most popular posts ever!). When we were living in Munich, I made a point of documenting all of our travels around Europe, knowing I'd want to look back on those trips later. I tend to write my posts in the Wordpress classic editor in a web browser. I'm comfortable with that workflow, but I've also used the Gutenberg block editor on occasion. I also occasionally post from the Wordpress app on my phone. Last year I figured out that I could post via Apple Shortcuts using the Wordpress API , which has made it really easy for me to share photos when I'm away from the computer. Definitely. We renovated our house last year and I took the opportunity to design the office-slash-craft-room of my dreams. I've got a ridiculously wide curved monitor, a super clicky Keychron mechanical keyboard , and a very silly NES mouse . I've got a comfy chair and a desk I can adjust up and down with the press of a button. I've got everything I need within arm's reach whether I'm blogging, knitting, or sewing. Having a pegboard with lots of tools on it fills a part of my soul I didn't realise was empty! I also tend to listen to music when I'm at the computer (Apple Music via Echo Dot speaker). I started out with my own hand-coded PHP-based CMS, which I used for fifteen years on various shared hosting providers. Eventually I realised that my rudimentary coding skills weren't meeting the security bar anymore, and I reluctantly let a friend persuade me try out Wordpress . To my surprise, I loved it and ended up migrating roalddahlfans.com over to it as well. It's not without its occasional headaches, but I haven't had any major problems in the last decade. During the initial months of the pandemic, I decided to migrate both sites from their shared webhosting over to Amazon Lightsail (since I was working for AWS at the time and wanted to learn more about the service). Both sites are hosted in the US and use the Wordpress blueprint on Linux (2GB memory, 2 vCPUs, 60GB SSD) with CloudFront as a CDN. The domain names were registered with Google Domains, which have since been migrated to Squarespace. For a couple years now I've been toying with the idea of converting one or both the sites to be static, served off S3 or similar. It would be a lot cheaper, faster, and more secure. The challenge is that I'm not starting from scratch; I have thousands of posts and pages that I've written over the years. I also really like being able to post from anywhere, including my phone. My husband Rodd was an SRE at Google and is a much better coder than me, so he's been working on a script that would allow us to run Wordpress from our home server and generate static HTML. We'll try it out with roalddahlfans.com and then decide where to go from there. Haha, I'd definitely choose a different name! I realised pretty early on that if you typed my domain incorrectly, nearly every other permutation was adult content. Whoops. I'd probably go with a static site today, just to keep things as simple as possible. I love the old school tiny websites that folks are building these days, and I'd go full IndieWeb - write on my site and syndicate everywhere else. I would be a bit more... considerate with how I write. I realised this year as I've been tidying up the broken links in my archives that I wasn't always kind in how I wrote about some people or topics. The early 2000s were a much snarkier time, I guess. I've chosen to leave those posts up, but I've added a disclaimer that they don't necessarily reflect who I am today. (Similarly, I've been adding a disclaimer on all my posts about Harry Potter. Rowling sucks, and it makes me sad.) I held off on monetising for many years, especially on roalddahlfans.com since most of the audience are kids. Eventually I added some hand-picked Google Ads to the site and to my blog archives, restricting the placement and ad categories. I also added Amazon affiliate links to the Dahl site, since I figured a lot of parents would be likely to buy the books. These more than covered the hosting costs over the years. (In fact, Rodd had to declare my websites when he joined Google in case there was a conflict of interest!) When we were living in Munich a few years back, I discovered that Germany has stringent tax laws about website monetisation and would require me to register as a side business. It was all too complicated, so the easiest thing was to just remove the ads. I didn't bother putting them back when we got to Australia, so for several years now there's been no income. Hosting is generally $40-50 AUD a month (for both sites), with an additional $125 AUD per site annually for Jetpack (which gives me automated backups, comment spam protection, and some other security features). This is honestly way overkill, and I reckon I can reduce that a lot in the future (especially if I move to a static site). I don't like the idea that everything on the web needs to be monetised or that everybody needs to have a side hustle. I love it when people share things online just because they want to. That's how both my sites started, and that's why I keep them going. That said, I financially support a couple bloggers and my Mastodon instance admin via Patreon, because I want them to continue. So many sites disappear, and it's just sad. One of the people I think of when I think of blogging is Matt Haughey from A Whole Lotta Nothing: a.wholelottanothing.org . He was the founder of Metafilter.com , and I think I've been reading his site for literally decades now. He was there when blogging was invented. Definitely someone worth talking to! Does a newsletter count as a blog these days? The person whose words have given me the most hope for humanity in 2025 is Mike Monteiro . Every time he writes it's just so, so good and I end up blogging it or sending it around to all my friends. (Archives are here .) I've also been really enjoying the blog of Robb Knight: rknight.me . He built Echofeed , the service I use to syndicate my blog posts out to Mastodon and Bluesky. He's always building something nerdy, or geeking out over pens and stationery. This is so absolutely dorky, but I am currently reading the Aubrey-Maturin "Master and Commander" books . I always figured they were quintessential "Dad books" (given they are historical fiction about naval voyages in the Napoleonic Wars) and therefore Not For Me, but this amazing essay convinced me to give them a shot. I've discovered that the books themselves are fantastic, and there's this whole fandom subculture I never knew about! They refer to reading the series as a "circumnavigation," and there are charming map websites and podcasts and subreddits with fans all talking (in character!) about the books. I'm loving it. It's been a welcome escape from the news over the past few months, and I can highly recommend the series. Now that you're done reading the interview, go check the blog and subscribe to the RSS feed . If you're looking for more content, go read one of the previous 108 interviews . Make sure to also say thank you to Markus Heurung and the other 120 supporters for making this series possible.
As many of you know, I am skeptical of the concept of relying on someone else’s computer , especially when a service grows to the point where it becomes an oligopoly, or worse, a monopoly. Cloudflare is, in my view, on track to becoming precisely that. As a result, I would argue they are a net negative for the internet and society at large. Besides the frustration they cause to VPN and Tor users through incessant captchas, Cloudflare’s infamous one more step pages have dulled users' vigilance, making them more vulnerable to even the most blatant malware attacks . Moreover, under the guise of iNnOvAtIvE cLoUd InFrAsTrUcTuRe , Cloudflare not only enable phishermen to phish and tunnelers to tunnel : Ironically, the very security measures they sell can be bypassed by bad actors using Cloudflare itself . It’s a similar irony that their systems, designed to shield clients from threats, sometimes struggle to defend their own infrastructure . Incidents like these highlight not only weaknesses in Cloudflare’s offerings but a broader issue: Cloudflare has become a highly attractive target for state-sponsored attacks , suffering from recurring breaches . Their sheer scale, considering that they are serving a substantial portion of the internet, means that an outage or compromise could have widespread, costly consequences. Another major concern is, that in many cases, Cloudflare acts as a man-in-the-middle SSL-terminating proxy between users and websites. They have visibility into everything users do on these sites, from browsing habits to submitting sensitive personal information. This makes Cloudflare a prime target for any actor seeking to harvest massive amounts of data. The Cloudbleed incident clearly demonstrated the risks: Tavis Ormandy posted the issue on his team’s issue tracker and said that he informed Cloudflare of the problem on February 17. In his own proof-of-concept attack he got a Cloudflare server to return “private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.” I stand with Hugo in considering Cloudflare harmful and recommend that websites avoid relying on it whenever possible. Cloudflare’s origins in Project Honeypot , and its early ties to the US Department of Homeland Security, are troubling to say the least: Five years later Mr Prince was doing a Master of Business Administration (MBA) at Harvard Business School, and the project was far from his mind, when he got an unexpected phone call from the US Department of Homeland Security asking him about the information he had gathered on attacks. Mr Prince recalls: “They said ‘do you have any idea how valuable the data you have is? Is there any way you would sell us that data?’. “I added up the cost of running it, multiplied it by ten, and said ‘how about $20,000 (£15,000)?’. “It felt like a lot of money. That cheque showed up so fast.” Mr Prince, who has a degree in computer science, adds: “I was telling the story to Michelle Zatlyn, one of my classmates, and she said, ‘if they’ll pay for it, other people will pay for it’.” Source: BBC Furthermore, Cloudflare has been criticized as an employer , reportedly fostering a hire-and-fire culture among its sales staff . Even its CEO has attracted controversy, such as suing neighbors over their dogs following objections to his plans to build an 11,300-square-foot estate. Plans that required lobbying to overcome local zoning laws . Given all this, it is time to reconsider Cloudflare’s dominant market position , controlling over 20% of the internet . Cloudflare has shown a pattern of equivocating on politically sensitive issues , perhaps to maintain its status as the world’s largest botnet operator , and they appear to defend “free speech” when it is profitable , but not when it isn’t . Cloudflare has also been accused of providing services to terrorists and drug traffickers while skirting international sanctions . Meanwhile, open-source developers have been harshly punished for less. Despite the brilliance of many engineers at Cloudflare, they are not infallible. They, too, experience recurring downtime and preventable mistakes . Cloudflare, like any other company, puts its pants on one leg at a time . There is no reason it should be treated as the default, or sole, solution for content delivery. If running your own Varnish instances isn’t feasible, and you need a global CDN, consider these alternatives to support competition and balance the scales: Info: Some hosting services might use Cloudflare without disclosing it openly/obviously, e.g. Render . Make sure to check whatever hosting service that you’re using whether it employs Cloudflare’s infrastructure in the background. If you currently have domains registered with Cloudflare, move them elsewhere immediately. As a general rule, never allow your CDN or hosting provider to also hold your domain registrations. Should the hosting provider cut you off, you’ll want the freedom to quickly redirect your domains to another provider without disruption. For more info, visit the cloud and domains sections of the infrastructure page. If, however, you’re running Cloudflare’s more advanced service offers, like Cloudflare Workers, you will likely have a harder time moving away. While some frameworks support different providers, like Vercel, Fastly, AWS, Azure, or Akamai, it is likely that most simple implementations will be heavily reliant on Cloudflare’s architecture. There’s unfortunately no easy path out of this, other than rewriting the specific components and infrastructure deployment configuration to support a different provider. If you wish to identify or avoid websites that make use of Cloudflare, you can use this browser extension for Firefox and Chrome (ironically created by Cloudflare). Beware that these extensions might transfer information about your browsing behavior to Cloudflare. Configure them to be active only when manually clicked on specific websites that you want investigate. There are third-party alternatives like this and this , as well as older/unmaintained extensions like this and this . PS: Decentraleyes is a solid option to enhance browsing privacy; check the browser section for other helpful extensions. All that said, you might think “Come on, Cloudflare isn’t that bad!” , and you’d be right: Every now and then, they do some good . *smirk* Still, we have to recognize that Cloudflare has grown into a cornerstone of modern digital infrastructure, which is a role that could eventually render it too big too fail , to borrow a term from the financial world. Footnote: The artwork was generated using AI and further botched by me using the greatest image manipulation program .
I’ll be starting from scratch and creating an IAM user with access to an S3 bucket that will be used to store the Pulumi state file. If you’re working in an enterprise setting, your authentication methods may vary. The TL;DR is that you can run this if you already have an S3 bucket and AWS credentials configured on your machine: This post assumes you have the Pulumi CLI installed. Check out the following guide if you don’t have it installed: Download & install Pulumi . First, we need to create the S3 bucket where the Pulumi state file will be stored. I created a bucket called and left all the default settings as-is. Then we need to create an IAM user in AWS that the Pulumi CLI can use. This IAM user needs permissions to access the S3 bucket we just created. I go to IAM and create a new user. I just called it : Then in the next step, I selected “Attach policies directly” and selected the AWS-managed “AdministratorAccess” policy just to keep things simple. You can provide more fine-grained access depending on your needs. Then click “Next” at the bottom. In the next screen, double check everything and then click on “Create user”. Now that we have a user with the appropriate permissions, we’ll need to get an AWS access key and secret to use with the Pulumi CLI. Go to your IAM user and click on “Create access key” on the right side. In the next screen, select “Command Line Interface (CLI)”. Check the box at the bottom, then click “Next”. The next screen will ask for setting a description tag. This is optional. I chose to skip it and clicked on “Create access key”. We finally have our Access key and Secret access key. Save these somewhere safe and click “Done”. (Don’t worry, the credentials in the screenshot are fake.) Now we can try using these credentials to tell the Pulumi CLI to use the S3 bucket as a backend. Note that you do NOT need the AWS CLI installed. Pulumi just needs the AWS credentials. Create the file if you don’t have it. Then add in your credentials there under the profile. (You can add more profiles, but this is beyond the scope of this post.) You’ll need the bucket’s region and your local AWS profile name to use S3 as a backend. The command formula looks like this: In my case, the command looks like this (make sure to edit for your needs): A successful login shows the following message: Alternatively, you can add your backend to your file. This is useful if you’re working on multiple Pulumi projects that each have different backends. You won’t need to run all the time. Just add a key and a nested key: More information here: Pulumi project file reference . Finally, it’s time to test this out. To demonstrate, I created a simple Pulumi program by running: You can choose whatever language you want though. This is the main Pulumi code that is generated. It’s code for creating an S3 bucket: Then I ran and it worked! And just to double check, I can see that my previously empty S3 bucket now has contents created by the Pulumi CLI: https://www.pulumi.com/docs/iac/download-install/ https://www.pulumi.com/registry/packages/aws/installation-configuration/ https://www.pulumi.com/docs/iac/concepts/state-and-backends/#aws-s3 https://www.pulumi.com/docs/iac/concepts/projects/project-file/ https://ashoksubburaj.medium.com/pulumi-with-aws-s3-as-backend-ac79533820f1
NVIDIA has become a giant, unhealthy rock on which the US markets — and to some extent the US economy — sits, representing 7-to-8% of the value of the market and a large percentage of the $400 billion in expected AI data center capex expected to be spent this year, which in turn made up for more GDP growth than all consumer spending combined. I originally started writing this piece about something else entirely — the ridiculous Oracle deal, what the consequences of me being right might be, and a lot of ideas that I'll get to later, but I couldn't stop looking at what NVIDIA is doing. To be clear, NVIDIA is insane, making 88% of its massive revenues from selling the distinct GPUs and associated server hardware to underpin the inference and training of Large Language Models, a market it effectively created by acquiring Mellanox for $6.9 billion in 2019 , and its underlying hardware that allowed for the high-speed networking to connect massive banks of servers and GPUs together, a deal now under investigation by China's antitrust authorities . Since 2023, NVIDIA has made an astonishing amount of money from its data center vertical , going from making $47 billion in the entirety of their Fiscal Year 2023 to making $41.1 billion in its last quarterly earnings alone. What's even more remarkable is how little money anyone is making as a result, with the combined revenues of the entire generative AI industry unlikely to cross $40 billion this year , even when you include companies like AI compute company CoreWeave, which expects to make a little over $5 billion or so this year , though most of that revenue comes from Microsoft, OpenAI (funded by Microsoft and Google, who are paying CoreWeave to provide compute to OpenAI, despite OpenAI already being a client of CoreWeave, both under Microsoft and in their own name)...and now NVIDIA itself, which has now agreed to buy $6.3 billion of any unsold cloud compute through, I believe, the next four years. Hearing about this deal made me curious. Why is NVIDIA acting as a backstop to CoreWeave? And why are they paying to rent back thousands of its GPUs for $1.5 billion over four years from Lambda , another AI compute company it invested in? The answer is simple: NVIDIA is effectively incubating its own customers, creating the contracts necessary for them to raise debt to buy GPUs — from NVIDIA, of course — which can, in turn, be used as collateral for further loans to buy even more GPUs . These compute contracts are used by AI compute companies as a form of collateral — proof of revenue to reassure creditors that they're good for the money so that they can continue to raise mountains of debt to build more data centers to fill with more GPUs from NVIDIA. This has also created demand for companies like Dell and Supermicro, companies that accounted for a combined 39% of NVIDIA's most recent quarterly revenues . Dell and Supermicro buy GPUs sold by NVIDIA and build the server architecture around them necessary to provide AI compute, reselling them to companies like CoreWeave and Lambda, who also buy GPUs of their own and have preferential access from NVIDIA. You'll be shocked to hear that NVIDIA also invested in both CoreWeave and Lambda, that Supermicro also invested in Lambda, and that Lambda also gets its server hardware from Supermicro. While this is the kind of merciless, unstoppable capitalism that has made Jensen Huang such a success, there's an underlying problem — that these companies become burdened with massive debt, used to send money to NVIDIA, Supermicro (an AI server/architecture reseller), and Dell (another reseller that works directly with CoreWeave ), and there doesn't actually appear to be mass market demand for AI compute, other than the voracious hunger to build more of it. In a thorough review of just about everything ever written about them, I found a worrying pattern within the three major neoclouds (CoreWeave, Lambda, and Nebius): a lack of any real revenue outside of Microsoft, OpenAI, Meta, Amazon, and of course NVIDIA itself, and a growing pile of debt raised in expectation of demand that I don't believe will ever arrive. To make matters worse, I've also found compelling evidence that all three of these companies lack the capacity to actually serve massive contracts like OpenAI's $11.9 billion deal with CoreWeave ( and an additional $4 billion added a few months later ), or Nebius' $17.4 billion deal with Microsoft , both of which were used to raise debt for each company. On some level, NVIDIA's Neocloud play was genius, creating massive demand for its own GPUs, both directly and through resellers, and creating competition with big tech firms like Microsoft's Azure Cloud and Amazon Web Services, suppressing prices in cloud compute and forcing them to buy more GPUs to compete with CoreWeave's imaginary scale. The problem is that there is no real demand outside of big tech's own alleged need for compute. Across the board, CoreWeave, Nebius and Lambda have similar clients, with the majority of CoreWeave's revenue coming from companies offering compute to OpenAI or NVIDIA's own "research" compute. Neoclouds exist as an outgrowth of NVIDIA, taking on debt using GPUs as collateral , which they use to buy more GPUs, which they then use as collateral along with the compute contracts they sign with either OpenAI, Microsoft, Amazon or Google. Beneath the surface of the AI "revolution" lies a dirty secret: that most of the money is one of four companies feeding money to a company incubated by NVIDIA specifically to buy GPUs and their associated hardware. These Neoclouds are entirely dependent on a continual flow of private credit from firms like Goldman Sachs ( Nebius , CoreWeave , Lambda for its IPO ), JPMorgan ( Lambda , Crusoe , CoreWeave ), and Blackstone ( Lambda , CoreWeave ), who have in a very real sense created an entire debt-based infrastructure to feed billions of dollars directly to NVIDIA, all in the name of an AI revolution that's yet to arrive. Those billions — an estimated $50 billion a quarter for the last three quarters at least — will eventually have the expectation of some sort of return, yet every Neocloud is a gigantic money loser, with CoreWeave burning $300 million in the last quarter with expectations to spend more than $20 billion in capital expenditures in 2025 alone . At some point the lack of real money in these companies will make them unable to pay their ruinous debt, and with NVIDIA's growth already slowing, I think we're watching a private credit bubble grow with no way for any of the money to escape. I'm not sure where it'll end, but it's not going to be pretty. Let's begin.
Time flies like an arrow. Fruit flies like a banana. Back at re:Invent 2018, we shared Firecracker with the world. Firecracker is open source software that makes it easy to create and manage small virtual machines. At the time, we talked about Firecracker as one of the key technologies behind AWS Lambda, including how it’d allowed us to make Lambda faster, more efficient, and more secure. A couple years later, we published Firecracker: Lightweight Virtualization for Serverless Applications (at NSDI’20). Here’s me talking through the paper back then: The paper went into more detail into how we’re using Firecracker in Lambda, how we think about the economics of multitenancy ( more about that here ), and how we chose virtualization over kernel-level isolation (containers) or language-level isolation for Lambda. Despite these challenges, virtualization provides many compelling benefits. From an isolation perspective, the most compelling benefit is that it moves the security-critical interface from the OS boundary to a boundary supported in hardware and comparatively simpler software. It removes the need to trade off between kernel features and security: the guest kernel can supply its full feature set with no change to the threat model. VMMs are much smaller than general-purpose OS kernels, exposing a small number of well-understood abstractions without compromising on software compatibility or requiring software to be modified. Firecracker has really taken off, in all three ways we hoped it would. First, we use it in many more places inside AWS, backing the infrastructure we offer to customers across multiple services. Second, folks use the open source version directly, building their own cool products and businesses on it. Third, it was the motivation for a wave of innovation in the VM space. In this post, I wanted to write a bit about two of the ways we’re using Firecracker at AWS that weren’t covered in the paper. Bedrock AgentCore Back in July, we announced the preview of Amazon Bedrock AgentCore . AgentCore is built to run AI agents. If you’re not steeped in the world of AI right now, you might be confused by the many definitions of the word agent . I like Simon Willison’s take : An LLM agent runs tools in a loop to achieve a goal. 1 Most production agents today are programs, mostly Python, which use a framework that makes it easy to interact with tools and the underlying AI model. My favorite one of those frameworks is Strands , which does a great job of combining traditional imperative code with prompt-driven model-based interactions. I build a lot of little agents with Strands, most being less than 30 lines of Python (check out the strands samples for some ideas ). So where does Firecracker come in? AgentCore Runtime is the compute component of AgentCore. It’s the place in the cloud that the agent code you’ve written runs. When we looked at the agent isolation problem, we realized that Lambda’s per-function model isn’t rich enough for agents. Specifically, because agents do lots of different kinds of work on behalf of different customers. So we built AgentCore runtime with session isolation . Each session with the agent is given its own MicroVM, and that MicroVM is terminated when the session is over. Over the course of a session (up to 8 hours), there can be multiple interactions with the user, and many tool and LLM calls. But, when it’s over the MicroVM is destroyed and all the session context is securely forgotten. This makes interactions between agent sessions explicit (e.g. via AgentCore Memory or stateful tools), with no interactions at the code level, making it easier to reason about security. Firecracker is great here, because agent sessions vary from milliseconds (single-turn, single-shot, agent interactions with small models), to hours (multi-turn interactions, with thousands of tool calls and LLM interactions). Context varies from zero to gigabytes. The flexibility of Firecracker, including the ability to grow and shrink the CPU and memory use of VMs in place, was a key part of being able to build this economically. Aurora DSQL We announced Aurora DSQL, our serverless relational database with PostgreSQL compatibility, in December 2014. I’ve written about DSQL’s architecture before , but here wanted to highlight the role of Firecracker. Each active SQL transaction in DSQL runs inside its own Query Processor (QPs), including its own copy of PostgreSQL. These QPs are used multiple times (for the same DSQL database), but only handle one transaction at a time. I’ve written before about why this is interesting from a database perspective. Instead of repeating that, lets dive down to the page level and take a look from the virtualization level. Let’s say I’m creating a new DSQL QP in a new Firecracker for a new connection in an incoming database. One way I could do that is to start Firecracker, boot Linux, start PostgreSQL, start the management and observability agents, load all the metadata, and get going. That’s not going to take too long. A couple hundred milliseconds, probably. But we can do much better. With clones . Firecracker supports snapshot and restore , where it writes down all the VM memory, registers, and device state into a file, and then can create a new VM from that file. Cloning is the simple idea that once you have a snapshot you can restore it as many time as you like. So we boot up, start the database, do some customization, and then take a snapshot. When we need a new QP for a given database, we restore the snapshot. That’s orders of magnitude faster. This significantly reduces creation time, saving the CPU used for all that booting and starting. Awesome. But it does something else too: it allows the cloned microVMs to share unchanged ( clean ) memory pages with each other, significantly reducing memory demand (with fine-grained control over what is shared). This is a big saving, because a lot of the memory used by Linux, PostgreSQL, and the other processes on the box aren’t modified again after start-up. VMs get their own copies of pages they write to (we’re not talking about sharing writable memory here), ensuring that memory is still strongly isolated between each MicroVM. Another knock-on effect is the shared pages can also appear only once in some levels of the CPU cache hierarchy, further improving performance. There’s a bit more plumbing that’s needed to make some things like random numbers work correctly in the cloned VMs 2 . Last year, I wrote about our paper Resource management in Aurora Serverless . To understand these systems more deeply, let’s compare their approaches to one common challenge: Linux’s approach to memory management. At a high level, in stock Linux’s mind, an empty memory page is a wasted memory page. So it takes basically every opportunity it can to fill all the available physical memory up with caches, buffers, page caches, and whatever else it may think it’ll want later. This is a great general idea. But in DSQL and Aurora Serverless, where the marginal cost of a guest VM holding onto a page is non-zero, it’s the wrong one for the overall system. As we say in the Aurora serverless paper , Aurora Serverless fixes this with careful tracking of page access frequency: − Cold page identification: A kernel process called DARC [8] continuously monitors pages and identifies cold pages. It marks cold file-based pages as free and swaps out cold anonymous pages. This works well, but is heavier than what we needed for DSQL. In DSQL, we take a much simpler approach: we terminate VMs after a fixed period of time. This naturally cleans up all that built-up cruft without the need for extra accounting. DSQL can do this because connection handling, caching, and concurrency control are handled outside the QP VM. In a lot of ways this is similar to the approach we took with MVCC garbage collection in DSQL. Instead of PostgreSQL’s , which needs to carefully keep track of references to old versions from the set of running transactions, we instead bound the set of running transactions with a simple rule (no transaction can run longer than 5 minutes). This allows DSQL to simply discard versions older than that deadline, safe in the knowledge that they are no longer referenced. Simplicity, as always, is a system property.
This week, something strange happened. Oracle, a company that had just missed on its earnings and revenue estimates, saw a more-than-39% single day bump in its stock , leading a massive market rally. Why? Because it said its remaining performance obligations — contracts signed that its customers yet to pay — had increased by $317 billion from the previous quarter, with CNBC reporting at the time that this was likely part of Oracle and OpenAI's planned additional 4.5 gigawatts of data center capacity being built in the US . Analysts fawned over Oracle — again, as it missed estimates — with TD Cowen's Derrick Wood saying it was a "momentous quarter" (again, it missed ) and that these numbers were "really amazing to see," and Guggenheim Securities' John DiFucci said he was "blown away." Deutsche Bank's Brad Zelnick added that "[analysts] were all kind of in shock, in a very good way." RPOs, while standard (and required) accounting practice and based on actual signed contracts, are being used by Oracle as a form of marketing. Plans change, contracts can be canceled (usually with a kill fee, but nevertheless), and, especially in this case, clients can either not have the money to pay or die for the very same reason they can't pay. In Oracle's case, it isn’t simply promising ridiculous growth, it is effectively saying it’ll become the dominant player in all cloud compute. A day after Oracle's earnings and a pornographic day of market swings, the Wall Street Journal reported that OpenAI and Oracle had signed a $300 billion deal , starting in "2027," though the Journal neglected to say whether that was the year or Oracle’s FY2027 (which starts June 1 2026). Oracle claims that it will make $18 billion in cloud infrastructure revenue in FY2026, $32 billion in FY2027, $73 billion in FY2028, $114 billion in FY2029, and $144 billion in FY2030. While all of this isn't necessarily OpenAI (as it adds up to $381 billion), it's fair to assume that the majority of it is. This means — as the $300 billion of the $317 billion of new contracts added by Oracle, and assuming OpenAI makes up 78% of its cloud infrastructure revenue ($300 billion out of $381 billion) — that OpenAI intends to spend over $88 billion fucking dollars in compute by FY2029, and $110 billion dollars in compute, AKA nearly as much as Amazon Web Services makes in a year , in FY2030. A sidenote on percentages, and how I'm going to talk about this going forward. If I'm honest, there's also a compelling argument that more of it is OpenAI. Who else is using this much compute? Who has agreed, and why? In any case, if you trust Oracle and OpenAI, this is what you are believing: I want to write something smart here, but I can't get away from saying that this is all phenomenally, astronomically, ridiculously stupid. OpenAI, at present, has made about $6.26 billion in revenue this year , and it leaked a few days ago that it will burn $115 billion " through 2029 ," a statement that is obviously, patently false. Let's take a look at this chart from The Information : A note on "free cash flow." Now, these numbers may look a little different because OpenAI is now leaking free cash flow instead of losses, likely because it lost $5 billion in 2024 , which included $1 billion in losses from "research compute amortization," likely referring to spreading the cost of R&D out across several years, which means it already paid it. OpenAI also lost $700 million from its revenue share with Microsoft. In any case, this is how OpenAI is likely getting its "negative $2 billion" number." Personally, I don't like this as a means of judging this company's financial health, because it's very clear it’s using it to make its losses seem smaller than they are. The Information also reports that OpenAI will, in totality, spend $350 billion in compute from here until 2030, but claims it’ll only spend $100 billion on compute in that year. If I'm honest, I believe it'll be more based on how much Oracle is projecting. OpenAI represents $300 billion of the $317 billion of new cloud infrastructure revenue it’ll from 2027 through 2030, which heavily suggests that OpenAI will be spending more like $140 billion in that year. As I'll reveal in this piece, I believe OpenAI's actual burn is over $290 billion through 2029, and these leaks were intentional to muddy the waters around how much their actual costs would be. There is no way a $116 billion burnrate from 2025 to 2029 includes these costs, and I am shocked that more people aren't doing the basic maths necessary to evaluate this company. The timing of the leak — which took place on September 5, 2025, five days before the Oracle deal was announced — always felt deeply suspicious, as it's unquestionably bad news... unless, of course, you are trying to undersell how bad your burnrate is. I believe that OpenAI's leaked free cash flow projections intentionally leave out the Oracle contract as a means of avoiding scrutiny. I refuse to let that happen. So, even if OpenAI somehow had the money to pay for its compute — it won't, but it projects, according to The Information, that it’ll make one hundred billion dollars in 2028 — I'm not confident that Oracle will actually be able to build the capacity to deliver it. Vantage Data Centers, the partner building the sites, will be taking on $38 billion of debt to build two sites in Texas and Wisconsin , only one of which has actually broken ground from what I can tell, and unless it has found a miracle formula that can grow data centers from nothing, I see no way that it can provide OpenAI with $70 billion or more of compute in FY2027. Oracle and OpenAI are working together to artificially boost Oracle's stock based on a contract that is, from everything I can see, impossible for either party to fulfill. The fact that this has led to such an egregious pump of Oracle's stock is an utter disgrace, and a sign that the markets and analysts are no longer representative of any rational understanding of a company's value. Let me be abundantly clear: Oracle and OpenAI's deal says nothing about demand for GPU compute. OpenAI is the largest user of compute in the entirety of the generative AI industry. Anthropic expects to burn $3 billion this year (so we can assume that its compute costs are $3 billion to $5 billion, Amazon is estimated to make $5 billion in AI revenue this year, so I think this is a fair assumption ), and xAI burns through a billion dollars a month . CoreWeave expects about $5.3 billion of revenue in 2025 , and per The Information Lambda, another AI compute company, made more than $250 million in the first half of 2025. If we assume that all of these companies were active revenue participants (we shouldn't, as xAI mostly handles its own infrastructure), I estimate the global compute market is about $40 billion in totality, at a time when AI adoption is trending downward in large companies according to Apollo's Torsten Sløk . And yes, Nebius signed a $17.4 billion, four-year-long deal with Microsoft , but Nebius now has to raise $3 billion to build the capacity to acquire "additional compute power and hardware, [secure] land plots with reliable providers, and [expand] its data center footprint," because Nebius, much like CoreWeave, and, much like Oracle, doesn't have the compute to service these contracts. All three have seen a 30% bump in their stock in the last week. In any case, today I'm going to sit down and walk you through the many ways in which the Oracle and OpenAI deal is impossible to fulfill for either party. OpenAI is projecting fantastical growth in an industry that's already begun to contract, and Oracle has yet to even start building the data centers necessary to provide the compute that OpenAI allegedly needs.
I recently started helping a less technical friend and had my first chance to see/use Contabo VPS. I’ve been really surprised at their default security practices so far. Contabo’s default VPS creation seems to be root user and password? If you go to “Advanced” the default is to create a user called “admin” (good!) and has the option for a public SSH key. Our new server, that barely any bot knows exists, already gets 350 failed password login attempts an hour. Worse, these bots can see that password login is enabled on our server, meaning they know they should keep trying. 350 password requests an hour on a strong password isn’t much, but eventually more bots will realize our IP is using passwords and try more. Eventually after copy pasting around a password enough some compromised browser plugin / discord plugin etc will capture the password and put it in a list. Contabo “knows” this, even if they don’t practice it: https://contabo.com/blog/how-to-use-ssh-keys-with-your-server Conatabo is bad that they encourage you via their defaults when setting up the VPS to use SSH password. They went out of their way to do that, likely to not put roadblocks up for new users, but it’s bad security. When we hit an issue we contacted Contabo support. They asked us to copy paste our password so they could help troubleshoot an issue. While I appreciate that level of support, assuming users have a SSH password and asking them to email it seems crazy to me. Now there is a record on both our email providers of that password and IP. 1) Use pub/private SSH keys. We can copy/paste public keys anywhere we want, super safe. 2) Make future servers with a user other than root and disable login with root. Root user has special login privileges (allowing SSH to get hammered). And in the future, I’ll complain a bit less about DigitalOcean/AWS/GCP.
I recently started using Contabo VPS’ while working with some friends and at some point we decided we wanted to use local networks. On all other major cloud providers this is free, faster and encouraged to send data over the private local network rather than across the public internet when communicating with two instances. Contabo? It costs $1.50 USD per month per VPS to add them to a private network. Ok, that’s a a surprisingly high amount compared to free / encouraged. So let’s just do it right? Wait! After having signed up two servers, both require “reinstallation” AKA they require being completely wiped and remade. Thankfully, these are not old servers, they’ve all been setup in the past couple weeks, but wow Contabo is prettty painful:
Hello and welcome to another premium newsletter. Thanks as ever for subscribing, and please email me at [email protected] to say hello. As I've written again and again , the costs of running generative AI do not make sense. Every single company offering any kind of generative AI service — outside of those offering training data and services like Turing and Surge — is, from every report I can find, losing money, and doing so in a way that heavily suggests that there's no way to improve their margins. In fact, let me explain an example of how ridiculous everything has got, using points I'll be repeating behind the premium break. Anysphere is a company that sells a subscription to their AI coding app Cursor, and said app predominantly uses compute from Anthropic via their models Claude Sonnet 4.1 and Opus 4.1. Per Tom Dotan at Newcomer , Cursor sends 100% of their revenue to Anthropic, who then takes that money and puts it into building out Claude Code, a competitor to Cursor. Cursor is Anthropic's largest customer. Cursor is deeply unprofitable, and was that way even before Anthropic chose to add "Service Tiers," jacking up the prices for enterprise apps like Cursor . My gut instinct is that this is an industry-wide problem. Perplexity spent 164% of its revenue in 2024 between AWS, Anthropic and OpenAI . And one abstraction higher (as I'll get into), OpenAI spent 50% of its revenue on inference compute costs alone , and 75% of its revenue on training compute too (and ended up spending $9 billion to lose $5 billion). Yes, those numbers add up to more than 100%, that's my god damn point. Large Language Models are too expensive, to the point that anybody funding an "AI startup" is effectively sending that money to Anthropic or OpenAI, who then immediately send that money to Amazon, Google or Microsoft, who are yet to show that they make any profit on selling it. Please don't waste your breath saying "costs will come down." They haven't been, and they're not going to. Despite categorically wrong boosters claiming otherwise , the cost of inference — everything that happens from when you put a prompt in to generate an output from a model — is increasing, in part thanks to the token-heavy generations necessary for "reasoning" models to generate their outputs, and with reasoning being the only way to get "better" outputs, they're here to stay (and continue burning shit tons of tokens). This has a very, very real consequence. Christopher Mims of the Wall Street Journal reported last week that software company Notion — which offers AI that boils down to "generate stuff, search, meeting notes and research" — had AI costs eat 10% of its profit margins to provide literally the same crap that everybody else does. As I discussed a month or two ago, the increasing cost of AI has begun a kind of subprime AI crisis , where Anthropic and OpenAI are having to charge more for their models and increasing the price on their enterprise customers to boot. As discussed previously, OpenAI lost $5 billion and Anthropic $5.3 billion in 2024, with OpenAI expecting to lose upwards of $8 billion and Anthropic — somehow — only losing $3 billion in 2025. I have severe doubts that these numbers are realistic, with OpenAI burning at least $3 billion in cash on salaries this year alone , and Anthropic somehow burning two billion dollars less on revenue that has, if you believe its leaks, increased 500% since the beginning of the year . Though I can't say for sure, I expect OpenAI to burn at least $15 billion in compute costs this year alone , and wouldn't be surprised if its burn was $20 billion or more. At this point, it's becoming obvious that it is not profitable to provide model inference, despite Sam Altman recently saying that OpenAI was. He no doubt is trying to play silly buggers with the concept of gross profit margins — suggesting that inference is "profitable" as long as you don't include training, staff, R&D, sales and marketing, and any other indirect costs. I will also add that OpenAI pays a discounted rate on its compute . In any case, we don't even have one — literally one — profitable model developer, one company that was providing these services that wasn't posting a multi-million or billion-dollar loss. In fact, even if you remove the cost of training models from OpenAI's 2024 revenues ( provided by The Information ), OpenAI would still have lost $2.2 billion fucking dollars. One of you will say "oh, actually, this is standard accounting." If that's the case, OpenAI had a 10% gross profit margin in 2024, and while OpenAI has leaked that it has a 48% gross profit margin in 2025 , Altman also claimed that GPT-5 scared him, comparing it to the Manhattan Project . I do not trust him. Generative AI has a massive problem that the majority of the tech and business media has been desperately avoiding discussing: that every single company is unprofitable, even those providing the models themselves. Reporters have spent years hand-waving around this issue, insisting that "these companies will just work it out," yet never really explaining how they'd do so other than "the cost of inference will come down" or "new silicon will bring down the cost of compute." Neither of these things have happened, and it's time to take a harsh look at the rotten economics of the Large Language Model era. Generative AI companies — OpenAI and Anthropic included — lose millions or billions of dollars, and so do the companies building on top of them, in part because the costs associated with delivering models continue to increase. Integrating Large Language Models into your product already loses you money, at a price where the Large Language Model provider (EG: OpenAI and Anthropic) is losing money. I believe that generative AI is, at its core, unprofitable, and that no company building their core services on top of models from Anthropic or OpenAI has a path to profitability outside of massive, unrealistic price increases. The only realistic path forward for generative AI firms is to start charging their users the direct costs for running their services, and I do not believe users will be enthusiastic to do so, because the amount of compute that the average user costs vastly exceeds the amount of money that the company generates from a user each month. As I'll discuss, I don't believe it's possible for these companies to make a profit even with usage-based pricing, because the outcomes that are required to make things like coding LLMs useful require a lot more compute than is feasible for an individual or business to pay for. I will also go into how ludicrous the economics behind generative AI have become, with companies sending 100% or more of their revenue directly to cloud compute or model providers. And I'll explain why, at its core, generative AI is antithetical to the way that software is sold, and why I believe this doomed it from the very beginning.
Cloud
Sql
Data
AI
DevOps
Web
Performance
JavaScript
Security
Open Source
Business
Go
TypeScript
Data Analysis
JSON
Web Development
Career
HTML
Tutorial
Python