Posts in Api (20 found)
Simon Willison 1 weeks ago

Stateless MCP has recaptured my interest (and inspired mcp-explorer and datasette-mcp)

Tuesday was Stateless MCP day - the rollout of MCP 2.0, or the 2026-07-28 Model Context Protocol specification to use the more formal but less memorable name. This is the most significant change to the MCP spec since it first launched, and has also served to reignite my personal interest in the protocol. For background: MCP is the Model Context Protocol, which describes a standard way to expose new tools to LLM-powered agent frameworks. It was introduced by Anthropic back in November 2024 , had a huge spike of interest through much of 2025, and then became somewhat eclipsed by Skills (another Anthropic invention) when it became apparent that an agent harness with access to a terminal and could do most of what MCP did in a more flexible way. I wrote about that in my review of 2025 . I'm coming back around to MCP now. Giving an agent a shell environment with the ability to access the internet is fraught with risk , and requires a strong model that is capable of effectively driving such an environment. MCP tools are easier to audit and control, and simple enough that smaller models that run on a laptop can still drive them reasonably well. The new stateless MCP specification also greatly decreases the complexity of implementing both clients and servers for the protocol. I built three of those this week! The best demonstration of the difference between stateful and stateless MCP is in this May 21st blog post that introduced the RC for the new specification. It included a clear before-and-after example. The older stateful MCP (I'm going to call it "legacy MCP") required two HTTP requests - the first to initialize a session and obtain a , and the second to actually call the tool: The new stateless way uses a single HTTP request which looks like this: This is so much cleaner from both a client- and server-side implementation perspective. It's also a better fit for building scalable web applications, since now you don't need to maintain server-side state to keep track of those session IDs, or worry about routing the same session to the same backend machine. I couldn't find a great CLI tool for interactively probing an MCP server, so I had Codex help build my own. mcp-explorer is the result. It's a stateless Python CLI tool, so you don't even need to install it to try it out - it works with uvx like this: This queries Ade Oshineye's agentic-mermaid.dev demo MCP. The above command returns the following list of tools: Then to inspect a tool: This outputs a whole bunch of information, including the JSON schema of the inputs and outputs. To call that tool and pass arguments to it: Which returns: To get just the raw SVG try adding to that command. I got back this image : There are a few more commands in the README, but you get the general idea. I find building CLI tools like this to be a really productive way to get familiar with a specification, even if an agent writes most of the actual code. The second project is datasette-mcp , a Datasette plugin which adds a endpoint to any Datasette instance. This is probably the fourth time I've tried building this plugin, but thanks to the new stateless MCP specification I finally have a version that feels good to release. It provides just three tools: , , and . They do exactly what you would expect them to do - though is read-only for the moment. Wire these into an agent, or a chat tool like ChatGPT or Claude, and they'll gain the ability to run SQL queries against your hosted Datasette instance. So far I'm running it on the Datasette mirror of my blog, at datasette.simonwillison.net/-/mcp . It took a bit of fiddling to figure out how to attach that to ChatGPT and Claude, but I got there in the end. Here's a new TIL showing exactly how to do that. Here's a shared Claude session where I asked it: It ran 7 separate SQL queries to figure out the answer. My LLM tool is long overdue for an official MCP integration. The new alpha llm-mcp-client plugin is my attempt at exactly that: Here's the output (including reasoning trace, I'm using LLM 0.32rc2 ): Considering note count I see the question "count the notes" is probably asking me to tally up blog notes. It could also mean published notes or drafts, so there's some ambiguity there. I'll need to figure out the total number of notes, likely by querying the count for both published notes and drafts to get a clear answer. Let's execute that count! There are 151 notes . And the output of llm logs for that prompt. Once this is fully baked, I'm considering bringing it directly into LLM core. I'm excited to experiment with MCP in Datasette Agent and llm-coding-agent as well. A few months after MCP was first released, I wrote Model Context Protocol has prompt injection security problems , where I noted that the pattern of having end users mix and match tools pushed responsibility for avoiding data exfiltration attacks out to the users themselves. I hadn't coined the Lethal Trifecta yet, but that was absolutely what I had in mind. Then general agents with arbitrary shell and access came along, and that's so much harder to keep secure! Something I've come to appreciate about MCP is that it's much easier to reason about agent capabilities and what might go wrong than with arbitrary command execution in an open network environment - the default for most of today's general and coding agent tools. I plan to lean into MCP a whole lot more when I'm building sensitive applications on top of LLMs. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options .

c# C# json JSON ai AI programming Programming api API
0 views
Filippo Valsorda 2 weeks ago

Opaque, Interoperable Passkey Records (and a Go API)

Passkeys are the most important thing happening in information security right now because they are the only principled solution to the overwhelming effectiveness of phishing attacks. Just like memory safety is the only principled solution to memory corruption attacks. Unfortunately, implementing them on the server side can appear more complex than using password hashes. Part of this is unavoidable because passkeys require interaction with the browser to get their phishing resistance properties. Part of it, however, could be abstracted away a little more effectively by defining interoperable passkey record encodings. The WebAuthn specification defines a credential record as an abstract concept with a number of components such as , , , , , and other flags. Google recommends a database table with a Credential ID primary key, and , , and columns. Adam Langley’s stellar Tour of WebAuthn similarly recommends a primary key, and separate and columns. All guidance recommends using libraries to handle WebAuthn authentication, but that still leaves applications with a potentially non-interoperable database schema. It might also be impractical to defer the whole authentication flow and database interaction to a library or framework. Interoperable, well-specified passkey records that the application can handle as opaque strings, like password hashes, can be a middle-ground abstraction layer. c2sp.org/passkey-record is a specification proposal which borrows the syntax of Password Hashing Competition (PHC) Strings and reuses the existing authenticator data encoding for the bulk of the work. A record looks like this: The payload is the authenticator data, a binary encoding of most of the credential record fields that is already specified by WebAuthn and included in the JSON encoding of an AuthenticatorAttestationResponse (which is the return type of even if attestation is not in use). Transports are the only missing field, and they are stored as PHC parameters. The application is then only in charge of keeping track of what passkey records are associated with a user account, which is a task that is familiar to web developers because it is not dissimilar to implementing password authentication (except there are multiple passkeys per account). These opaque strings can be passed to a library to verify the login assertion (or to generate a registration request with the appropriate ). With a well-specified interoperable storage format, it will hopefully be possible to switch passkey library (or even backend language) while preserving a credentials database. Along with the passkey record, applications might still wish to store metadata fields like a user-selected nickname and creation and last use timestamps, to provide pretty passkey management UIs. None of these require any special treatment by the WebAuthn library. The one exception is the backed up state flag. Passkeys report to the server whether they are backed up (e.g. to iCloud Keychain or a Google account), and servers can use that signal to suggest removing a password from an account. This flag can change across logins, while the passkey record is immutable, so it would have to be stored separately and updated on every login. I think this logic is overrated for the average website, which will keep supporting email password resets anyway. Building on top of these passkey records, I drafted a potential stateless Go package API . The registration flow is The login flow is The application is in charge of The library provides JSON values that can be passed straight to and , and accepts JSON values returned by calling on a PublicKeyCredential . This is optimized for the discoverable credential flow (a.k.a. passkeys, where the authenticator stores and provides to the server the user ID) but the method can also be used for second-factor flows or re-authentication prompts. The API works for both modal and conditional UI (autofill) flows. There are a few helpers to extract information from the passkey record ( , ) and from the JSON-encoded PublicKeyCredential ( ). Currently, there is no implementation; I would like to get feedback on the passkey record format and on the Go API , before potentially making a proposal for Go 1.28. One thing you can’t do with this storage model is ensure that different accounts don’t share passkeys with the same Credential ID, which the spec says you SHOULD do. The reason for that check is avoiding attacks where you look up a credential by its ID and land at the wrong public key or user ID because the attacker intentionally injected a colliding Credential ID through their own account. This attack simply can’t happen if you don’t have a Credential ID index in the first place! The index is only needed to mitigate an attack introduced by the existence of the index. Login attempts carry the user ID, and if you use that to look up the user’s passkey records to verify the login against, it doesn’t matter if some other user has a passkey with the same Credential ID, just like it doesn’t matter if two users share a password. Don’t let the attacker dictate your PRIMARY KEY and you won’t have PRIMARY KEY collision attacks. For more Go API previews, follow me on Bluesky at @filippo.abyssdomain.expert or on Mastodon at @[email protected] . More from this year’s CENTOPASSI (a GPS-tracked motorcycle competition involving careful planning, 100 coordinates, and 1700 km of secondary roads over three and a half days). Here’s a glimpse of Castel del Monte (AQ), after climbing down from a deserted and still snowy Campo Imperatore. My work is made possible by Geomys , an organization of professional Go maintainers, which is funded by Ava Labs , Teleport , Datadog , Tailscale , and Sentry . Through our retainer contracts they ensure the sustainability and reliability of our open source maintenance work and get a direct line to my expertise and that of the other Geomys maintainers. (Learn more in the Geomys announcement .) Here are a few words from some of them! Teleport — For the past five years, attacks and compromises have been shifting from traditional malware and security breaches to identifying and compromising valid user accounts and credentials with social engineering, credential theft, or phishing. Teleport Identity is designed to eliminate weak access patterns through access monitoring, minimize attack surface with access requests, and purge unused permissions via mandatory access reviews. Ava Labs — We at Ava Labs , maintainer of AvalancheGo (the most widely used client for interacting with the Avalanche Network ), believe the sustainable maintenance and development of open source cryptographic protocols is critical to the broad adoption of blockchain technology. We are proud to support this necessary and impactful work through our ongoing sponsorship of Filippo and his team. call with the logged-in (or otherwise identified) user details and any existing passkey records pass the returned JSON to and then to pass the returned JSON-encoded PublicKeyCredential to store the returned passkey record in the database call while generating the log-in page store the returned request in a key-value cache with a short TTL, under , and pass the returned JSON to and then to pass the returned JSON-encoded PublicKeyCredential to , and use the returned requestID to retrieve the request from the key-value cache and use the returned userID to retrieve the passkey records from the database pass the JSON PublicKeyCredential, the request, and the passkey records to associating an opaque, permanent, privacy-preserving user ID with each user; storing passkey records associated with a user; and caching request challenges produced by .

security Security api API go Go json JSON
0 views
Simon Willison 3 weeks ago

Kimi K3, and what we can still learn from the pelican benchmark

Chinese AI lab Moonshot AI announced Kimi K3 this morning, describing it as their "most capable model to date, with 2.8 trillion parameters". It's currently available via their website and API, but an open weight release is promised "by July 27, 2026". Moonshot are calling this the first "open 3T-class model" (I guess they're rounding 2.8 trillion up to 3 trillion), taking the crown from DeepSeek's 1.6T v4 Pro . Their self-reported benchmarks have K3 mostly beating Claude Opus 4.8 max and GPT-5.5 high, while losing out to Claude Fable 5 and GPT-5.6 Sol. A few highlights from the Artificial Analysis report on the model: The model is also now the leading model on Arena.ai's Frontend Code arena , surpassing even Claude Fable 5. The new model is notable for the pricing: $3/million input tokens and $15/million output tokens, putting it at the same level as Anthropic's Claude Sonnet series and making it the most expensive model released by a Chinese AI lab to date. This is a significant increase on their earlier models such as Kimi K2.6 at $0.95/$4. 2.8 trillion parameters is also more than twice the size of that 1T model. I used OpenRouter (to avoid signing up for a Moonshot API key) with the llm-openrouter plugin to generate an SVG of a pelican riding a bicycle: Here's the transcript . It looks like this: That pelican took 95 input tokens and 16,658 output tokens (13,241 were reasoning tokens), for a total cost of 25 cents ! Since K3 accepts image input I ran it against that rendered SVG above (with my alt text prompt ) and got back (for 0.6 cents ): Cartoon illustration of a white pelican wearing a red scarf, riding a red bicycle along a gray road with white dashed lines; the pelican has a large orange beak and webbed orange feet pedaling, with white motion lines behind it; the background shows a light blue sky with white clouds, a yellow sun, two small black birds in flight, and green grass with tiny white flowers in the foreground My Generate an SVG of a pelican riding a bicycle test is 21 months old now. It was never a particularly great benchmark. It started out as a joke on how absurdly difficult it is to compare these models, but then for the first year it turned out to have a surprising correlation to how good the models actually were. That connection has been mostly severed now. The GPT-5.6 and Claude Fable 5 pelicans are outclassed by GLM-5.2 , and much as I love GLM I don't think that's a Fable-class model. (I'm still not convinced that labs are training for the benchmark - if they were, I'd expect much better results. There's a chance that Gemini has optimized for any combination of an animal on a vehicle though!) The biggest limitation of the pelican is that it doesn't touch at all on the thing that matters most for today's model: agentic tool calling and the ability to operate tools reliably as conversations grow in length. So don't go using pelicans to compare models! All of that said, I still get a decent amount of value out of running the benchmark myself. Firstly, it's a forcing function for actually trying the model. If I show you a pelican, that means I've managed to run a prompt through it. If the model has an official API I'll use that, if it's open weight (and small enough to fit a 128GB M5 MacBook Pro) I'll try running it on my own machine, usually via llama.cpp or LM Studio or Ollama . I'll frequently use OpenRouter since that usually provides a proxy to an official API without me needing a new API key. Most of my pelicans are generated using my LLM CLI tool , which helps encourage me to ensure the latest models are supported by that (via one of its plugins). More importantly though, even the act of a single prompt to "Generate an SVG of a pelican riding a bicycle" can reveal interesting model characteristics. Consider the result for Kimi K3 today. Running those simple prompts helped emphasize several points about the model. K3 currently only has one thinking effort level, but I've been deriving quite a bit of value recently from running the same pelican prompt through different effort levels to get a quick idea for what impact those have. Here's my matrix for the GPT-5.6 model family , for example. Really though the main things I gain from the pelican test are: You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . "On our private long-horizon knowledge work evaluation, Kimi K3 reaches an overall Elo of 1547, +732 points from Kimi K2.6 and behind only Claude Fable 5." "Cost per task ($0.94) is similar to GPT-5.6 Sol ($1.04), ~1/2 the price of Opus 4.8 ($1.80) and higher than open weights peers" "Kimi K3’s token usage on the Artificial Analysis Intelligence Index decreased significantly, using 21% fewer output tokens than K2.6." It only has one reasoning effort right now, "max" - and it shows. The model consumed 13,241 reasoning tokens to output 3,417 tokens of response. This is expensive - the pelican cost 25 cents! How does the prompt "Generate an SVG of a pelican riding a bicycle" add up to 95 input tokens? OpenAI's tokenizer counts 10, Anthropic's counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting "hi" to Kimi K3 counted 86 tokens, suggesting there may be an 85 token hidden system prompt. It refused to leak it though. Vision works well: the alt text it generated is very good. It's a "hello world" exercise for prompting a model A rough cost and reasoning estimate for a simple task Confirmation that the model can output valid SVG and has a basic idea of geometry and spatial awareness. This is a much bigger deal for the smaller models that run on my laptop. It's still interesting to compare pelicans between releases in the same model family. K3's pelican is a notable improvement from Kimi 2.5 . It's something I can share that demonstrates I've tried it. Plus a comment with a pelican in it is kind of a tradition on Hacker News at this point, any time I'm late I get comments asking where it is!

machine-learning Machine Learning ai AI c++ C++ api API
0 views
daniel.haxx.se 3 weeks ago

Workshop Basel day three

See also: day one, day two . There is only one thing that is better than two days of HTTP workshop, and that is of course three days of HTTP workshop. The final day of this edition of the series started out with us again shuffling around where we parked ourselves around the big table. Except Mr captain of course who once again got to herd us forward through another day from the same seat. MOQ ( Media over QUIC transport ) is not HTTP, but it uses QUIC so it is at least tangentially interesting and it involves a lot of the same people so this status update still felt welcome and suitable. Compared to existing HTTP based solutions, MOQ is supposed to offer less complexity and lower latency. The moon landing was broadcasted with less latency than current live-streamed TV and maybe MOQ can make us come close to those numbers again. In MOQ clients subscribe to a track that then contains a lot of objects that are delivered. It’s not the request + response approach of HTTP. The fact that this is not HTTP of course brings a lot of questions and well, doubts, and we lingered on various aspects of this topic for quite a while. My prize for the best slides of the HTTP workshop 2026 goes to [redacted] for the excellent use of potato images in their presentation. PTTH is HTTP spelled backwards, commonly pronounced as PoTaToH. A client sets up the connection but the actual HTTP request is sent from the server to the client. One of the intended use cases for this, is to allow an origin server to connect to the CDN proxy and then be able to deliver traffic to the world, rather than to have the CDN connect to the origin the way they usually do. Apparently most CDNs already have custom and proprietary solutions for exactly this kind of feature, so maybe doing it in a standard way instead makes sense? The draft explains the new proposed way to continue a previously interrupted upload over HTTP. The upload request gets a Location: header back for the resource being uploaded, and if it gets stopped prematurely, a client can then HEAD that resource, figure out the size and then do a second upload (using the PATCH method) request that tells the server that this transfer should start at offset X. Exactly how this should be supported in browser’ upload forms seemed a little bit uncertain . For my own sake I can see a challenge to implement this nicely for curl in particular when the upload is using formpost upload (curl’s -F flag) which after all still is a very common way to do uploads on the current web. I’ll return to this topic at a later time when I written an implementation to test… io_uring is a Linux asynchronous I/O framework that avoids the overhead of traditional system calls. It uses two shared ring buffers between user space and the kernel, allowing applications to batch I/O operations with zero-copy efficiency. The feature is disabled by Google in ChromeOS, Android and in production Google servers which certainly holds back some use of it. io_uring can be helpful to speed up things, but might be complicated to use in existing software architectures and the presentation went into some details on why this is so. A walk-through of some of the recent developments and improvements in Firefox’s UDP networking stack . Going from single datagrams to the modern ways to ship large chunks of data offloaded to the kernel to speed things up. Upload throughput in Firefox is up 60-90% over the last 11 releases. Lots of fun graphs and metrics were shown. This work is based on the quinn-udp stack. Happy Eyeballs v3 is coming and Firefox is implementing it . It now takes into account many more data sources than before, including alt-svc and HTTPS-RR and races connections against each other to use the one that connects first. There are some recommended timers in the specification and parts of the discussion was around how maybe the timers could instead be tightened a bit, and maybe the delay between the subsequent attempts could then use an exponential backoff instead sticking to a fixed interval? (I know I’ll discuss some of these details with my curl hacker friends and see what we should adjust… curl already supports most of the Happy Eyeballs v3 specification.) As we approached the end of the day a few shorter topics were ventilated to give us a little more to consider before going home: With this, the seventh HTTP workshop had ended. Again a very fine event. This time graciously sponsored and arranged by Adobe. Thank you everyone! The general idea is to continue with these events roughly every second year and I support this. The HTTP workshops are definitely one of my favorite events. The top image on this post was used in the final presentation and the author told me he is aware of the AI errors in there, “of which there are at least two”. Why is there no UTF8 in URIs? “If we would do it again, we would have allowed UTF8 in there” was said by someone who was there in the mid 1990s… Optimistic DNS is a draft. Use stale DNS cache data while getting the new. Connection remains alive for 120 seconds while DNS data is often not cached for even 30 seconds. No one in the room seemed to hate it. Let’s do this! The journey to QUERY. One of the primary authors of the RFC took us through what it took to make it happen. It was sixteen years since the most previous registered HTTP method and maybe this was the last one ever?

backend Backend api API performance Performance
0 views
daniel.haxx.se 3 weeks ago

Workshop Basel day one

On this hot summer’s day in Basel, Switzerland, the seventh HTTP workshop started. These events tend to work roughly the same way and the people in the room are also to large extent familiar and known since previous editions. Forty people in a meeting room, where we take turns in doing short talks on HTTP and networking topics, with the following question and discussion session. The rules for the meetings are explicitly Chatham rules, which means that everything I write about the meeting will be sufficiently fuzzy and without many company or personal names. This is not the kind of meeting that can be easily summed up in a short blog post anyway. You really should be here. Present in the room were representatives from all the world’s most prominent and used HTTP deployments: clients, browsers, CDNs, proxies and servers. I’m happy to say that there were also several first-timers. We like fresh blood. (If you think I’m being overly brief or vague about specifics in this post; that is partially on purpose but primarily because I’m a lousy note-taker and mostly write this up after a busy day that also may have involved beer.) After a round of introductions, we started. REST is a set of constraints, and in this presentation it was argued that it can or maybe even should be extended to do more. A number of recent applications like Mastodon/ActivityPub, Bluesky/AT, Matrix, Nostr, IndieWeb, all currently use HTTP to do state synchronization but they all do it differently in their own unique ways. Can REST and maybe HTTP be adjusted to help this for improved interoperability? Looking at the Common Crawl data and comparing data over time, it was observed that responses use the Last-Modified header field more now than they did in the past, and there were great follow-up speculations on why this is so. Data also shows that a large share of these headers present dates that are almost identical to the time the requests were issued. With the cc-lint tool , data was gathered on how HTTP is actually used today, proving that there is work to be done: deprecated headers are used, some headers are done wrong, and many are overly big. This indicates that there are well used both servers and clients out there that would benefit from cleanup. It probably also shows that doing HTTP correctly and all the correct headers is far from an easy task. Another presentation showed data, this time from a well-known CDN, on the impact the existing AI scraper bots have on the Internet from their point of view. It showed that roughly half of the requests and half of the bandwidth are spent by scraper bots. A long discussion followed where the numbers were questioned as maybe the numbers look like this because a sufficiently large number of the “bad AI scrapers” appear as regular users to the classifiers. Speculations of different kinds were made.  As a follow-up from a presentation from a previous HTTP workshop we got to learn how the journey on developing their new HTTP stack has progressed and several fun adventures and lessons from that were shared with the audience. A look into new HTTP API development at Apple . Some discussions and lessons learned from creating new APIs for both servers and clients. We got an excellent walk-through of some details and internals of the Android networking stack. Emphasis was perhaps especially put on ECH and QUIC connection migration, and the final “don’t tell us when your connection closed” led to a long new discussion on how we really should fix the problem: when connection has been left idle for a long time and it is closed by the server, the client (mobile phones) don’t want to be told. This, because getting that RST and more, just wakes up the radio and more on the phone only to tell it to go back to sleep. It was theorized that if we could get rid of this unnecessary battery waste, the accumulated gain across billions of devices would make a serious dent. Several additional HTTP related problems were of course also subsequently solved as we then wandered into the city for dinner and maybe a beer. Of course yours truly returned back to his hotel room in good time to be able to write up this blog post. The best part of these workshops might be the (no pun intended) networking and discussions had completely outside of the agenda. End of day one. Two more to come,

web-development Web Development data-analysis Data Analysis api API
0 views
Steve Klabnik 3 weeks ago

Too many words about DIDs

Your “Bluesky account” is not just a Bluesky account: it is an account that can be used with a variety of other applications. This post is going to be an exploration of part of what that means from a technical perspective, so if you’re not a software developer, this post isn’t for you. But what I’m going to explain is the technical mechanism for how your account works separate from Bluesky, and in fact, separate from any particular app. Let’s talk about identity: who are you, anyway? Users of a system need some sort of way to describe who they are to use it. If you want to log in, you need to present who you are. If you want to make a post, well, we need to know who the author of that post is. For atproto, the protocol that underlies Bluesky and other apps in the ATmosphere, they use the “Decentralized Identity” standard, also known as DID. The W3C standardized DIDs in 2022 . As you might guess from the name, DIDs are, an identifier that you can use as the basis of identity for building applications. And the idea is that these identifiers are decentralized. However, a lot of people have a lot of feelings about that specific word, and often accuse atproto of not being properly decentralized. We’re going to go over the details so you can understand how this works, and you can decide for yourself if this approach suits you or not. Here is my DID, we’ll use this as an example: There are three parts, separated by colons: The scheme ( ), the method ( ), and the DID method-specific identifier ( ). To use a DID, such as , you resolve it into a DID Document A set of data describing the DID subject, including mechanisms, such as cryptographic public keys, that the DID subject or a DID delegate can use to authenticate itself and prove its association with the DID. That document contains various properties that describe the identity. Here’s my DID Document, at the time of writing: This document gives you everything you need to know to determine who I am, that is, given an arbitrary post that claims it’s written by me, this document describes how you’d verify that claim. We’ll get into how to do that that in a moment, but first, how do you resolve that DID into that DID document? Well, it’s pretty easy: each method is a standard that describes how you do that. So when you see , that means we use the PLC standard, which we’ll be going over in a moment. Another method supported by Bluesky is . In that case, you wouldn’t use the PLC standard, you’d use the Web one. This is the sense in which DIDs are decentralized: when you present your identity, you get to decide what method validates that that is a real identity. There’s no centralized authority that determines which DID types are valid. Now, of course, that doesn’t mean that every application supports every DID method, because while this specification is very generic, you’re still going to have to write some code to implement that particular method. I could say “Hey I’m ” and unless your app supports the method, it’s not gonna inherently just know what to do. So that is one important caveat. Let’s explain this resolution process for the method. While supported by Bluesky, a very small number of users actually use , but it’s a simpler method and so I think it’s illustrative to go over first. I’ll be using Liz Fong-Jones account as an example here. Her identity for that account is . So how do we resolve this DID into a DID Document? We take the method-specific identifier, which in this case is , and put it into this URL template: You can then go fetch this URL to resolve it into the DID Document, which at the time of writing, looks like this: This is very simple! So why might we not want to use ? Why bother with any other system? Well, this relies on the DNS system. One could make the argument that ultimately, this is still centralized in some form. If Liz’s domain registrar were to take away her domain, she would also lose control of this DID. In a more generic sense, if Liz decides she wants to not use that domain anymore, she will lose control of that identity to whoever does. That could be through non-malicious means, like letting it expire and someone else purchases it, or through malicious ones, like a hack which would compromise her registrar account and take the domain over. Also, you need to have a web server running on that domain with infinite uptime; if the server goes down, so does your ability to get the document. When this DID document changes, there’s no mechanism for clients to know that it’s changed, which means applications may use one that’s out of date, or that there is lag between updating the document and updating the application built on it, which may cause temporary problems until the latest document is fetched. All of these drawbacks led Bluesky to develop their own DID method, which attempts to fix these problems and others. This method is called . To resolve a , you take the entire DID, and put it in this template: You can then fetch that URL and get the DID document. So… what’s the difference? Well, in this case, both nothing and something. In a very literal sense, both are resolved in the same way: you fetch a URL. However, the details matter. There is already two ways in which this is different than DID:Web: I’ve presented the above as pros, but there are also cons. Before, I had to trust the DNS system and domain registrars, now I have to trust plc.directory. All of the same caveats apply in that sense, I have to trust that they don’t take my ID away from me, or that it doesn’t get stolen, etc. However, there are also some important details that mitigate this, which we’ll get to. But for some people, neither trusting DNS nor trusting plc.directory is acceptable, and there are other DID methods that use, for example, a blockchain to resolve the name. Bluesky does not support using any of those DID methods, so for this application, it’s not really relevant, but it’s important to know that they exist. Why do it this way? Well, the simplest way to put it is this: setting up a involves a lot of “nerd stuff.” You have to register a domain, and that’s also an ongoing monetary cost. You have to know how to set up a web server, and author some JSON to put on that server. You have to keep it running. You have to know how to store your private keys, and keep them safe. It’s a non-starter compared to “sign up for this web app.” And Bluesky’s goals involves making this platform accessible to non-nerds. By having plc.directory manage all of this, we eliminate all of those steps. While drafting this post, I have also been made aware of , which expands on and attempts to rectify some of its shortcomings. I have not read the spec yet, but it has reached 1.0, so it is probably worth checking out. I wanted to get this post shipped last week, and didn’t want to delay it further by adding another section, but if I were writing this post in the future, I’d probably want to talk about it as well, so just a little heads-up there. But it does also mean that, in some sense, Bluesky still owns your identity. They’ve generated a keypair for you, and the have access to the secret key. That’s unacceptable for some people. So how do you fix that? Well, has some additional features that does not. For example, will allow you to register additional keypairs with your ID and use them to rotate your signing keys. This allows you to remove the Bluesky generated keys and insert your own. While that is true, it’s also the case that your PDS needs to use your keys to sign your posts. As such, most people are likely to store their keys in their PDS, and so if you are using a Bluesky managed PDS, well, you’ve uploaded your keys to their infrastructure, and that’s probably not acceptable if you’re trying to keep your identity away from Bluesky. Of course, the solution there is to run your own PDS and then rotate your keys. At that point, your key is living on infrastructure you own, and Bluesky has no say over it any more. I think that this possibility is an important design property, and allows motivated users to meaningfully own their identity. A criticism of this boils down to “well, most users won’t do that,” and while that’s true, I also think that’s okay for most people, and that having the choice is more important than forcing every user to deal with their own key management. This is kind of an abrupt end to this post, but I just wanted to get some things down ‘on paper’ as it were. I hope you’ve learned a bit about identity and how it works with atproto. Here’s my post about this post on BlueSky: Too many words about DIDs: steveklabnik.com/writing/too-... Too many words about DIDs Blog post: Too many words about DIDs by Steve Klabnik Your DID is no longer tied to a specific domain name. I can let expire and move to and my stays the same. While a web server still needs to be running, that’s the job of plc.directory, not my own job. This is operationally much simpler.

api API Crypto programming Programming json JSON
0 views
Xe Iaso 1 months ago

Why are cached input tokens cheaper with AI services?

When you see AI model pricing pages, you usually see things broken down like this: Source: DeepSeek API Docs If you manage to have most of your input tokens be cached, you save a huge amount, in this case $0.20 per million tokens. What does this mean though? What does caching do that makes you save so much, in some cases upwards of tens of kilodollars? Someone explain the cached vs not thing to me for how this is $10,000 worth of savings lol [image or embed] — Chimney Sweepers Local 420 FKA yburyug ( @bobbby.online June 12, 2026 at 12:39 AM Warning I'm gonna be totally honest, I barely understand the basic outline of the math involved here. Where possible I am to not be completely wrong here, but I'm not going to emit something 1:1 accurate with the mathematical truth of large language models' inner workings. Bear with me. When you make an API call to large language model services, you make an API call like the following: That element is the key bit. Every time you accumulate messages from the initial system prompt, initial user request, AI responses and any tool use requests/responses, you add to that array and make it grow bigger and bigger. A good way to think about this is that sending a conversation to a large language model is like having a pair of people share a roll of paper on two different typewriters. Every time you finish your message, you send the roll of paper back to the AI model and it has to re-read through the entire conversation in order to start typing on the end with its response. As the conversation gets longer, this gets more and more expensive because the model has to recalculate its internal state all over again for every additional message. However, large language model inference is complicated but deterministic . Given the same inputs, you will always get the same output. This means that you can use a technique called key-value caching (KV caching) in order to save that intermediate state and use it for next time. Most of the time this cache is a prefix cache because that allows you to just add on more messages to the end of the request pretty easily and be fine. Imagine something like this: If the model has already processed the question about the sky being blue and generated the response about Rayleigh scattering, it doesn't need to process both of those messages again to answer the user's question about sunsets. In production AI model deployments you would put that generated intermediate state into the KV cache so that the model doesn't need to run twice for the same data. This saves time and effort on the side of the AI model provider, and currently model providers decide to pass that savings onto API users in the form of cheaper inference costs for cached lookups. As you develop an application with AI in it, try to avoid changing any inference settings or previous messages between prompts. This makes your application's queries much more likely to read from the cache, making it faster, reducing the environmental impact, and saving you(r users) money.

ai AI api API
0 views
Jack Vanlightly 2 months ago

Kafka Share Groups and Parallelizing Consumption - Part 3: Client-local parallelism

All tests were executed against Kafka 4.3.0 using Dimster.  In the last post Broker-Visible vs Client-Local Parallelism we looked at two ways of scaling Kafka consumption. The final unit of parallelism can be visible to the broker, as consumers, or it can be local to the client, as threads, virtual threads, async tasks, or some other execution mechanism hidden behind a smaller number of consumers.  Broker-visible parallelism is simple to reason about: if each consumer processes records serially, we add more consumers to increase parallelism. But each consumer adds overhead to the brokers: broker-side protocol state, TCP connections, group membership, fetch state, and participation in the consumer or share group protocol. With long processing times and/or high throughput, the required number of parallel workers can easily exceed what is practical to model as broker-visible consumers. That is where client-local parallelism becomes important. Instead of scaling by adding more consumers, each consumer application can poll records and process them concurrently inside the client. This allows a smaller number of Kafka consumers to drive a much larger amount of parallel work. In this post, we’ll compare client-local parallelism with consumer groups and share groups using the Apache Kafka clients, by way of Dimster, the benchmarking tool used throughout this series. Dimster uses the official Apache Kafka clients under the hood. The main comparison is between two styles of client-local parallelism: blocking and continuous styles. At the API level, applications obtain records by calling , then later record their progress by committing offsets or acknowledging records. Under the covers, the client sends fetch requests and commit/acknowledgment requests to the brokers. There is some indirection between API calls and network requests, but every parallel processing style has to fit into this general poll/commit cycle. Consumer-group consumers commit offsets (one offset per partition) whereas share-group consumers commit a set of per-record acknowledgments. Any parallel processing style must fit into this fetch/commit style. We can classify parallel processing within this fetch/commit style into two main methods: Blocking and Continuous Poll -> kick off parallel processing -> block on completion of all -> commit -> (repeat) There are many implementation options for decoupling polling, processing and commits, but the general pattern can be classified into two main mechanisms: Poll -> Dispatch loop : each record submitted for background processing. Keep polling independently of processing (though implement backpressure by limiting the number of inflight records, i.e. stop polling when your processing buffer is full). Accumulate -> Commit loop : Accumulate completed records to commit opportunistically. If you are rolling your own logic using the Apache Kafka clients (rather than choosing a parallel processing library), blocking is by far the simplest to code, but the most inferior in terms of performance profile. The Kafka clients support both blocking and continuous styles with consumer groups, but only blocking with share groups . Why don’t share groups support continuous mode with the Apache Kafka clients? It’s simple. You can only acknowledge a record from the current poll batch. If you try and acknowledge a record from the previous poll, it throws an exception. This may or may not change in the future, but it’s worth knowing this if you were planning on implementing a continuous parallel processing style with the AK clients and share groups. Dimster supports parallel processing simulation with both blocking and continuous mode, but share groups only support blocking mode. Dimster doesn’t actually process records (except for recording metrics), instead it simulates processing time by calculating how long each record would take, based on randomized processing times between the min/max. In Blocking mode , it figures out how long the processing would take to process a poll batch (based on the level of parallelism requested in the workload file) and performs one sleep per poll-loop-iteration for the aggregate processing time. In Continuous mode , it feeds each record, along with its randomized processing time, into an in-memory delay queue (accounting for how much parallel processing is requested). Separately it polls, drains completed records from the delay queue and commits continuously. Let’s run some benchmarks with Dimster in blocking vs continuous modes. We’ll use an example workload with: A long processing time of 1-5 seconds (3 second average) A moderate rate of 1,000 records a second.  Each consumer application is capable of processing around 300 records concurrently. The aggregate parallelism is 3,000 ( ) which puts us in the territory where serial consumers are not a great choice. Firstly, share groups only allow groups of up to 1000 members, and regardless, 3000 consumers would create more than 9000 TCP connections (in a three broker cluster), which is excessive for one use case of this size. We need to parallel process inside the clients. We’ll run 3 tests: Consumer group blocking style Consumer group continuous style Share group blocking style The workload file (single scenario with three test points): In the test analysis we’ll cover the configurations in this workload. All three tests resulted in the same 1000 records/s throughput. But end-to-end latency differed a lot, with consumer group continuous style easily winning. The latency distributions: The latency distribution of only cg-continous: Continuous is the clear winner here. You can download the Dimster result tarball here . Let’s dig into the results and why the workload was configured the way it was. With blocking mode, consumption is a factor of the poll rate and the number of records per poll: The poll rate is determined by how long the application blocks waiting for all records to complete. The number of records per poll is bounded by (though it is a soft cap). When estimating the poll rate in blocking mode, the average processing time is the wrong choice as we’ll really block for the longest processing time of a poll batch (not the average). With 100s of records per poll, we’ll likely hit or get close to the upper bound 5 seconds (assuming uniform distribution). More likely in the real world, we’d see a non-uniform distribution where, for example, p95 might be 500ms, but p99.9 be significantly higher at say, 5 seconds.  The workload we’re using has a rate of 1,000 records/s. Each consumer is capable of processing around 300 records concurrently so we set . With a poll rate of 0.2 (one every five seconds), the consumption throughput per consumer is 60 records/s. To reach 1000 records per second we need at least 17 consumers (and partitions), so I configured 18. The effective workload of test point 1 : The consumers managed the 1000 record/s but for some reason, the max end-to-end latency (processing-start-timestamp - publish-timestamp) was double the worst processing time. It turns out that this is a natural effect of Blocking mode with consumer groups. The highest e2e latency will be at least the blocking time of the previous poll iteration (as records kept arriving in the partition throughout the blocking time). However, you may note that the above e2e latency numbers are p50 is 7.5s and max is 10s. This can occur in blocking mode due to the way polls return buffered records and trigger an asynchronous fetch (pre-fetch) to fill the buffer before the next poll. Think of the consumer as having a two-step delivery path: first Kafka records are fetched asynchronously into the consumer’s internal buffer, and poll() returns buffered records to the application. In the above diagram, we see the application spending 5 seconds processing a batch it just received, but that batch had already spent 5s in the buffer as it was filled by a fetch triggered by the previous poll (5s before this one). This kind of e2e latency might not be a big deal, considering the long processing times. If we want to lower the e2e latency significantly, then we need the continuous style. With continuous style, we have decoupled polling from processing and we can use the average processing time of 3 seconds to calculate the consumption rate per consumer (we are not constrained by the max processing time). Parallelism is not defined by the number of records per poll but the total inflight capacity of parallel work (threads, virtual threads, async tasks). We can feed that capacity with a constant stream of small polls and stop polling once that capacity has been reached (polling again once there is free capacity again). Because the application can poll at a high frequency, buffered records remain in the buffer for only a few milliseconds before being submitted for parallel processing.  In continuous mode, worker throughput is approximately: In Dimster the inflight capacity is set by the workload field . With a capacity of 300 and average processing time of 3s, each consumer can process 100 records/s. To reach 1000 records per second, we need 10 consumers and partitions. I set the capacity to 400 to add some wiggle room. The effective workload of test point 2 : This time we see that e2e latency remains very low, as we don’t block on the longest processing time. Again with the Blocking style, so: The per-consumer poll rate is determined by the highest processing time per batch. The aggregate parallelism is 5000 ( ) Share groups introduce a new constraint, the per-partition inflight budget (aka ). The aggregate inflight budget must exceed the aggregate parallelism of 5000. The default is 2000 per partition and so just three partitions gives us an aggregate inflight budget of 6000. Another difference is that we can have multiple share consumers per partition. If we use then each partition needs parallelism of 1666 ( ). With , we need 6 consumers per partition to cover it. The effective workload of test point 3 : You might expect end-to-end latency to be lower than the blocking consumer group test and you’d be right! Each partition has six consumers so the time period between fetches is lower (records spend less time in the partition before being fetched). We are also using so there is no pre-fetching which inflated the e2e latency in the consumer group test. But it’s still higher than you might expect. Per-partition, we have 6 consumers with a fetch/poll rate of 1.2 per second and 333 records arrive per second. We might expect the worst e2e latency to be 277 ms (333 / 1.2). So what’s going on? The above calculation assumes each fetch arrives evenly spread over time. But fetches cluster to a greater or lesser degree, there is no coordination between the consumers. If a long period passes with no fetches, then the first fetch that arrives can drain the accumulated lag, and subsequent fetches just return the handful of records that arrived since the prior fetch a few milliseconds before. The only way for the consumers in this workload to reach the 1000 records/s is if each fetch returns around 277 records per fetch on average. With fetch request clustering, the only way fetches can be filled to this extent is if lag has built up. If 6 consumers attempt to fetch 300 records at exactly the same time, only if lag has reached 1800 on that partition will all those fetches return full. So the consumers settle into a stable amount of lag that is high enough such that fetches return with enough records to keep up. If consumers catch up and lag goes to 0, consumption throughput will naturally drop down until lag builds up to allow for the full consumption rate. Client-local parallelism is often the only practical way to handle long record processing times. But how that client-local parallelism is implemented in Kafka fetch/commit cycle has a big impact on latency. A blocking poll → process → commit loop is simple, but it couples consumption progress to the slowest record in each batch, which lowers poll frequency and can inflate e2e latency even when there is plenty of processing capacity. Continuous polling decouples polling, processing, and committing, allowing the client to keep records flowing into a processing pool while applying backpressure through an in-flight limit. For consumer groups, this provides much better latency and usually requires many fewer consumers and partitions for the same workload. Share groups improve the broker-visible side of the problem by allowing multiple consumers per partition, but the current Apache Kafka clients still constrain client-local parallelism to the blocking style. If your goal is highly parallel, low-latency processing, consumer groups remain the better fit. Removing the same-batch acknowledgment constraint from the Kafka share consumer would make that style possible with share groups as well. In the next post, I’ll look at some pathological share-group workloads with some gotchas to watch out for. Poll -> Dispatch loop : each record submitted for background processing. Keep polling independently of processing (though implement backpressure by limiting the number of inflight records, i.e. stop polling when your processing buffer is full). Accumulate -> Commit loop : Accumulate completed records to commit opportunistically. In Blocking mode , it figures out how long the processing would take to process a poll batch (based on the level of parallelism requested in the workload file) and performs one sleep per poll-loop-iteration for the aggregate processing time. In Continuous mode , it feeds each record, along with its randomized processing time, into an in-memory delay queue (accounting for how much parallel processing is requested). Separately it polls, drains completed records from the delay queue and commits continuously. A long processing time of 1-5 seconds (3 second average) A moderate rate of 1,000 records a second.  Each consumer application is capable of processing around 300 records concurrently. Consumer group blocking style Consumer group continuous style Share group blocking style The per-consumer poll rate is determined by the highest processing time per batch. The aggregate parallelism is 5000 ( )

performance Performance backend Backend api API
0 views
Simon Willison 2 months ago

Initial impressions of Claude Fable 5

I didn't have early access to today's Claude Fable 5 release, but I've spent the past ~5.5 hours putting it through its paces. My initial impressions are that this is something of a beast . It's slow, expensive and has been quite happily churning through everything I've thrown at it so far. As is frequently the case with current frontier models the challenge is finding tasks that it can't do. First, let's review the key characteristics. Anthropic claim that Claude Fable 5 offers the same performance as Claude Mythos 5, except with much more strict guardrails in place to prevent it being used for harmful things. Those guardrails trigger often enough that the Claude API has new mechanisms for letting you know when you hit them, and even has a new option to request it falls back to another model automatically if something gets rejected. Claude Mythos 5 is out today as well, Anthropic say it "Shares Claude Fable 5's capabilities without the safety classifiers". The models have a 1 million token context window, 128,000 maximum output tokens and a knowledge cut-off date of January 2026. They are priced at twice the price of Claude Opus 4.5/4.6/4.7/4.8: $10/million input tokens and $50/million output tokens. There's no increase in price for longer context usage. Other than that the upgrade guide is substantially thinner than the similar guide for Opus 4.8 . The best way to describe Fable is that it feels big . Not just in terms of speed and cost, but also in how much it knows. Here's an example prompt, which I used to compare Fable's knowledge with Opus 4.8 (typo included): Opus 4.8's reply started like this : I don't have a reliable, comprehensive, and date-verified list of all of Simon Willison's open source projects, and I want to be honest about that rather than risk giving you inaccurate dates or fabricated entries. Here's what I can tell you with reasonable confidence: Well-known projects by Simon Willison: LLM (~2023) – A command-line tool and Python library for interacting with large language models. Datasette (~2017) – A tool for exploring and publishing data, probably his most prominent recent project. sqlite-utils (~2018-2019) – A Python library and CLI tool for manipulating SQLite databases, often used alongside Datasette. Django (2005) – He is one of the co-creators of the Django web framework, along with Adrian Holovaty, originally developed at the Lawrence Journal-World newspaper. He also maintains a large ecosystem of Datasette plugins and many smaller utilities (often with names like , various tools for importing data, etc.). That's not bad. Here's the start of the response for the exact same prompt from Fable 5 - both models were running without access to search: I think you mean Simon Willison — the prolific open source developer, co-creator of Django, and creator of Datasette. An important caveat upfront: a truly complete list isn't feasible. Simon Willison is famously prolific — he has published hundreds of repositories (Datasette alone has 100+ plugins, many written by him), and my knowledge has a cutoff date, so I'll be missing his most recent work. Below are his notable projects with approximate initial release dates, most recent first: (Here's GPT-5.5 for good measure . It listed even more projects than Fable did!) In the past I've stated that I don't care about how much models know - I want them to be able to manipulate text and code in useful ways and actively look up the information they need via search tools, not bake it into their weights. But knowledge like this is a reasonably good proxy for model size - you can cram a whole lot more details about the world into a larger number of parameters. Does knowing more stuff mean the model is better at the tasks we pose to it? I can certainly imagine how a coding model with deeper knowledge of modern libraries and patterns could crunch through coding tasks more effectively. Is Fable really bigger than Opus? Anthropic haven't said anything about model size, so all we have are tea-leaves, but the speed, pricing and my own poking at its knowledge make me think that it's a large model. Maybe the largest yet from any vendor. Anthropic made Fable 5 available across all of their surfaces - the Claude.ai chat interface, Claude Code for web, Claude Code CLI and Claude Cowork as well. The model is available "until June 22nd" on the subscription plans (I'm on $100/month Max at the moment), after which it will be billed extra. Claude.ai is often under-estimated. Since September 2025 every chat has had access to a full container environment to run code, including the ability to install additional packages and even clone repositories directly from GitHub. Last week I released micropython-wasm , a Python library that uses wasmtime to run a custom build of MicroPython in WebAssembly to act as a sandbox for untrusted Python code. I decided to see if Fable could upgrade that to running full Python instead. I started with this prompt: Fable identified that it could use Brett Cannon's cpython-wasi-build builds for this, but was unable to download them itself due to environment restrictions. So I grabbed the two zip files from that page and uploaded them to Claude: ( , as attachments) And that was that. It churned away for a few minutes and got the entire thing working. Part of the response included: I tried the cleaner single-zip-stdlib approach to shrink the filesystem surface, but CPython's bootstrap fails to find from inside a zip without more prefix finessing — the directory-preopen approach works reliably, so that's what the PoC uses. The zip path is solvable but needs /frozen-getpath work. Then a little later: ... and it gave me this 13.9MB cpython_wasm-0.1.0-py3-none-any.whl file. You can try running Python code in a sandbox using that wheel URL and like this: Here's the full chat transcript . This was a very strong start. Before I'd realized it was Fable day, my stretch goal for today was to add a new feature to Datasette Agent : I wanted tool calls within that agent software to gain the ability to pause mid-execution and request approval directly from the user. This felt like a suitably meaty task to throw at the new model. Over the course of the day Fable not only solved that problem , it also identified and then implemented four issues in my underlying LLM library that would help support this kind of advanced pause-resume mechanism in tool calls. It got everything working first using somewhat gnarly hacks, but the moment I told it that changes to LLM itself were in scope it set to work unraveling the hacks and turning them into supported features of LLM instead. My stretch goal turned into LLM 0.32a3 , almost entirely written by Fable. Here are the release notes: Driven by the needs of Datasette Agent 's human-in-the-loop feature, made the following improvements to how tool calls work: I'm really impressed with the quality of API design, tests, code and documentation that Fable put together for this. I spent several hours on it today, but it feels like several days' worth of work. I recently started using AgentsView to help track my local LLM usage across all of the different coding agents. I published a TIL today about adding custom Fable pricing to that tool, which I expect will not be necessary in the very near future. After setting the price, I ran this command to start a localhost web server to explore my usage: Here's the treemap showing the breakdown of my Fable usage across various projects today: I used $110.42 worth of tokens today, all as part of my $100/month subscription. I ran "Generate an SVG of a pelican riding a bicycle" against all five thinking effort levels with Fable. Here are the results , including the token cost for each one: It's interesting that high ended up using fewer tokens than medium for this particular run. Here are the Opus 4.8 pelicans for comparison. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . LLM (~2023) – A command-line tool and Python library for interacting with large language models. Datasette (~2017) – A tool for exploring and publishing data, probably his most prominent recent project. sqlite-utils (~2018-2019) – A Python library and CLI tool for manipulating SQLite databases, often used alongside Datasette. Django (2005) – He is one of the co-creators of the Django web framework, along with Adrian Holovaty, originally developed at the Lawrence Journal-World newspaper. files-to-prompt (April 2024) – concatenates files into a single prompt for LLMs datasette-extract (2024) – plugin for extracting structured data using LLMs LLM (May–June 2023) – CLI tool and Python library for interacting with large language models, plus a large ecosystem of plugins (llm-gpt4all, llm-mistral, llm-claude, etc., 2023 onward) symbex (June 2023) – search Python code for symbols ttok and strip-tags (May 2023) – CLI tools for token counting and HTML cleanup for LLM pipelines datasette-lite (May 2022) – Datasette running entirely in the browser via WebAssembly/Pyodide shot-scraper (March 2022) – automated website screenshots via Playwright s3-credentials (November 2021) – CLI for creating scoped AWS S3 credentials django-sql-dashboard (2021) – SQL reporting dashboards for Django Dogsheep suite (2019) – personal analytics tools: twitter-to-sqlite, github-to-sqlite, healthkit-to-sqlite, dogsheep-beta, etc. sqlite-utils (2018) – CLI and Python library for manipulating SQLite databases Datasette (November 2017) – his flagship project; tool for exploring and publishing data csvs-to-sqlite (2017) – convert CSV files to SQLite Various early tools (~2007–2010) – soupselect, json-head, geocoders, and others Django (developed 2003–2005, open-sourced July 2005) – co-created with Adrian Holovaty at the Lawrence Journal-World Tool implementations can declare a parameter named in order to be passed the object for the current invocation. This allows them to access the current . See Accessing the tool call from inside a tool . #1480 Every tool call is now guaranteed a unique - providers that do not supply one get a synthesized -prefixed ULID. #1481 Tools can raise a exception to cleanly pause the tool chain, useful for things like waiting for human approval. The exception propagates to the caller with and (completed sibling results) attached, and no model call is made with a placeholder result. See Pausing a chain from inside a tool . #1482 Failure semantics for concurrent tool execution: async sibling tool calls always run to completion before a pause or hook exception propagates. #1482 Chains can now resume from a history ending in unresolved tool calls: the calls are executed through the normal / machinery before the first model call, skipping any that already have results. The method also accepts a new optional argument for executing an explicit list of objects in place of the calls requested by the response. See Resuming a chain with pending tool calls . #1482 Fixed a bug where the async tool executor silently dropped calls to tools not present in - these now return results, matching the sync executor. #1483

python Python json JSON api API ai AI html HTML
0 views
Xe Iaso 2 months ago

Giving your Go apps Tigris superpowers

Tigris is S3-compatible, which means you can point the AWS SDK at it and most things just work. The catch is that the Tigris-exclusive features—bucket forking, snapshots, object renaming, and the like—need verbose workarounds because the AWS SDK doesn't know they exist. So we wrote a Go SDK that does. It comes in two flavors: the package is a drop-in replacement for the standard S3 client with first-class methods for the Tigris-specific operations, and is a higher-level client for the common single-bucket case that infers its configuration from the environment so you stop passing the same parameters over and over. You can adopt the Tigris features incrementally without refactoring your existing S3 code, and the simpler API still works against other S3-compatible providers. I wrote up how it works and why we built it over on the Tigris blog.

api API programming Programming cloud Cloud
0 views
Aran Wilkinson 2 months ago

Introducing Headcode: A Unified API for UK Rail Data

Headcode is a unified, developer-friendly JSON API that takes the fragmented, legacy feeds of the UK rail network and turns them into clean, enriched real-time data.

data Data api API json JSON
0 views
Simon Willison 2 months ago

Gemini 3.5 Flash: more expensive, but Google plan to use it for everything

Today at Google I/O, Google released Gemini 3.5 Flash . This one skipped the modifier and went straight to general availability, and Google appear to be using it for a whole lot of their key products: 3.5 Flash is available today to billions of people globally: As usual with Gemini, the most interesting details are tucked away in the What's new in Gemini 3.5 Flash developer documentation. It mostly has the same set of platform features as the previous Gemini 3.x series, albeit with no computer use . The model ID is . The knowledge cut-off is January 2025, and it supports 1,048,576 input tokens and 65,536 maximum output tokens. Google are also pushing a new Interactions API , currently in beta, which looks to me like their version of the patterns introduced by OpenAI Responses - in particular server-side history management. Gemini 3.5 Flash is accompanied by a notable price bump. The previous models in the "Flash" family were Gemini 3 Flash Preview and Gemini 3.1 Flash-Lite . The new 3.5 Flash is 3x the price of 3 Flash Preview and 6x the price of 3.1 Flash-Lite (see price comparison here ). At $1.50/million input and $9/million output it's getting close in price to Google's Gemini 3.1 Pro, which is $2 and $12. The Gemini team promise that 3.5 Pro will roll out "next month" - presumably at an even higher price. This fits a trend: OpenAI's GPT-5.5 was 2x the price of GPT-5.4, and Claude Opus 4.7 is around 1.46x the price of 4.6 when you take the new tokenizer into account . Given the price increase it's interesting to see Google roll it out for so many of their own free-to-consumer products. It feels like all three of the major AI labs are starting to probe the price tolerance of their API customers. Artificial Analysis publish the cost to run their proprietary benchmark against models, which is a useful way to take things like tokenization and increased volume of reasoning tokens into account. Some numbers worth comparing: Running the benchmark for 3.5 Flash (high) cost significantly more than 3.1 Pro Preview! Here are some numbers from other vendors: I ran "Generate an SVG of a pelican riding a bicycle" against the Gemini API and got back this pelican, which is a lot : From the code comments: hedgehog on Hacker News : That pelican looks like it's in Miami for a crypto conference. That one cost me 11 input tokens and 14,403 output tokens, for a total cost of just under 13 cents . You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . For everyone via the Gemini app and AI Mode in Google Search For developers in our agent-first development platform Google Antigravity and Gemini API in Google AI Studio and Android Studio For enterprises in Gemini Enterprise Agent Platform and Gemini Enterprise. Gemini 3.5 Flash (high) : $1,551.60 Gemini 3.1 Pro Preview : $892.28 Gemini 3 Flash Preview (Reasoning) : $278.26 Gemini 3.1 Flash-Lite Preview : $93.60 Claude Opus 4.7 (Adaptive Reasoning, Max Effort) : $5,117.14 Claude Opus 4.7 (Non-reasoning, High Effort) : $1,217.23 GPT-5.5 (xhigh) : $3,357.00 GPT-5.5 (medium) : $1,199.14

ai AI api API
1 views
Unsung 3 months ago

“There seems to be a file that is just filled with undecipherable Morse.”

On April Fools in 2021, the popular xkcd comic ran Checkbox , which was a Morse code puzzle in disguise. (It’s interesting to see the community trying to figure out what it actually does .) Engineer Max Goodhart built the front-end and wrote a summary of the whole project : This year was a doozy. We specced and scrapped several different ideas in the months leading up to today. We finally settled on today’s concept just 3 days ago. The need to do something simple was a really useful constraint, and we leaned into the idea of making something primitive but deep. The team seems to have had a lot of fun with it, including even JavaScript being encoded in Morse Code (the link in the blog post no longer works, but you can still see it on the Internet Archive ). Goodhart also wrote about the immense challenge of adjusting the Morse tapping speed to the user, which counterintuitively ended up needing… adjusting the user to the speed. But the best part is that the server communications used the Morse code in URLs, as well: We took great pains to make the API for this project use morse code in the transport. If you take a look at the network inspector, you’ll notice that the URLs requested have morse code in them. This worked for every combination of letters imaginable, with two oddly specific exceptions: a solitary E, and a solitary I. I liked this description of what transpired next, which would have made me think I was going insane, too: Then, an even stranger thing happened . I copied and pasted the correct URL into my browser and pressed Enter, and right before my eyes, it deleted the ”.” from the end of the URL and returned a different result. I was delighted to discover an answer here, not only because in retrospect it’s such an obvious thing that was staring us all in the face for decades, but also because it has interesting URL construction consequences. #bugs #encoding #web

javascript JavaScript api API web-development Web Development
0 views
Josh Comeau 3 months ago

Scroll-Driven Animations

The new Animation Timeline API allows us to create dynamic scroll animations without any JavaScript! It’s honestly a very lovely API, and in this blog post, we’ll explore some of the super cool things we can do with it.

javascript JavaScript web-development Web Development api API
0 views
Simon Willison 3 months ago

A pelican for GPT-5.5 via the semi-official Codex backdoor API

GPT-5.5 is out . It's available in OpenAI Codex and is rolling out to paid ChatGPT subscribers. I've had some preview access and found it to be a fast, effective and highly capable model. As is usually the case these days, it's hard to put into words what's good about it - I ask it to build things and it builds exactly what I ask for! There's one notable omission from today's release - the API: API deployments require different safeguards and we are working closely with partners and customers on the safety and security requirements for serving it at scale. We'll bring GPT‑5.5 and GPT‑5.5 Pro to the API very soon. When I run my pelican benchmark I always prefer to use an API, to avoid hidden system prompts in ChatGPT or other agent harnesses from impacting the results. One of the ongoing tension points in the AI world over the past few months has concerned how agent harnesses like OpenClaw and Pi interact with the APIs provided by the big providers. Both OpenAI and Anthropic offer popular monthly subscriptions which provide access to their models at a significant discount to their raw API. OpenClaw integrated directly with this mechanism, and was then blocked from doing so by Anthropic. This kicked off a whole thing. OpenAI - who recently hired OpenClaw creator Peter Steinberger - saw an opportunity for an easy karma win and announced that OpenClaw was welcome to continue integrating with OpenAI's subscriptions via the same mechanism used by their (open source) Codex CLI tool. Does this mean anyone can write code that integrates with OpenAI's Codex-specific APIs to hook into those existing subscriptions? The other day Jeremy Howard asked : Anyone know whether OpenAI officially supports the use of the endpoint that Pi and Opencode (IIUC) uses? It turned out that on March 30th OpenAI's Romain Huet had tweeted : We want people to be able to use Codex, and their ChatGPT subscription, wherever they like! That means in the app, in the terminal, but also in JetBrains, Xcode, OpenCode, Pi, and now Claude Code. That’s why Codex CLI and Codex app server are open source too! 🙂 And Peter Steinberger replied to Jeremy that: OpenAI sub is officially supported. So... I had Claude Code reverse-engineer the openai/codex repo, figure out how authentication tokens were stored and build me llm-openai-via-codex , a new plugin for LLM which picks up your existing Codex subscription and uses it to run prompts! (With hindsight I wish I'd used GPT-5.4 or the GPT-5.5 preview, it would have been funnier. I genuinely considered rewriting the project from scratch using Codex and GPT-5.5 for the sake of the joke, but decided not to spend any more time on this!) Here's how to use it: All existing LLM features should also work - use to attach an image, to start an ongoing chat, to view logged conversations and to try it out with tool support . Let's generate a pelican! Here's what I got back : I've seen better from GPT-5.4 , so I tagged on and tried again : That one took almost four minutes to generate, but I think it's a much better effort. If you compare the SVG code ( default , xhigh ) the one took a very different approach, which is much more CSS-heavy - as demonstrated by those gradients. used 9,322 reasoning tokens where the default used just 39. One of the most notable things about GPT-5.5 is the pricing. Once it goes live in the API it's going to be priced at twice the cost of GPT-5.4 - $5 per 1M input tokens and $30 per 1M output tokens, where 5.4 is $2.5 and $15. GPT-5.5 Pro will be even more: $30 per 1M input tokens and $180 per 1M output tokens. GPT-5.4 will remain available. At half the price of 5.5 this feels like 5.4 is to 5.5 as Claude Sonnet is to Claude Opus. Ethan Mollick has a detailed review of GPT-5.5 where he put it (and GPT-5.5 Pro) through an array of interesting challenges. His verdict: the jagged frontier continues to hold, with GPT-5.5 excellent at some things and challenged by others in a way that remains difficult to predict. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . Install Codex CLI, buy an OpenAI plan, login to Codex Install LLM: Install the new plugin: Start prompting:

open-source Open Source css CSS ai AI api API
0 views
Zak Knill 3 months ago

SSE token streaming is easy, they said

I wrote about AI having ‘durable sessions’ to support async agentic applications, and in the comments everyone said: “Token streaming over SSE is easy” . …so I figured I’d dig into that claim. Agents used to be a thing you talked to synchronously. Now they’re a thing that runs in the background while you work. When you make that change, the transport breaks.

ai AI api API
0 views
Ed Zitron's Where's Your Ed At 3 months ago

News: Anthropic Removes Claude Code From $20-A-Month "Pro" Subscription Plan For New Users (Developing)

In developing news, Anthropic appears to have removed access to AI coding tool Claude Code from its $20-a-month "Pro" accounts. This is likely another cost-cutting move that follows a recent change ( per The Information ) that forced enterprise users to pay on a per-million-token based rate rather than having rate limits that were, based on researchers' findings, often much higher than the cost of the subscription. Previously, users were able to access Claude using their Pro subscriptions via a command-line interface and both the web and desktop Claude apps. Users were, instead of paying on a per-million-token basis, allowed to use their subscription to access Claude Code, but will likely now have to pay for API access. Anthropic's Claude Code support documents ( as recently as this April 10th archived page ) previously read "Using Claude Code with your Pro or Max plan." The page now reads "Using Claude Code with your Max plan." Pricing on Anthropic's website reflects the removal of Claude Code on both mobile and desktop. Some Pro users report that they are still able to access Claude Code via the web app and Command-Line Interface. It is unclear at this time whether this change is retroactive or for new Pro subscribers, or whether Anthropic intends to entirely remove access to Claude Code (without paying for API tokens) from every Pro customer. I have requested a comment from Anthropic, and will update this piece when I receive it, or if Anthropic confirms this move otherwise. If you liked this news hit and want to support my independent reporting and analysis, why not subscribe to my premium newsletter? It’s $70 a year, or $7 a month, and in return you get a weekly newsletter that’s usually anywhere from 5,000 to 18,000 words, including vast, detailed analyses of NVIDIA , Anthropic and OpenAI’s finances , and the AI bubble writ large . I recently put out the timely and important Hater’s Guide To The SaaSpocalypse , another on How AI Isn't Too Big To Fail , a deep (17,500 word) Hater’s Guide To OpenAI , and just last week put out the massive Hater’s Guide To Private Credit . Subscribing to premium is both great value and makes it possible to write these large, deeply-researched free pieces every week.  Anthropic appears to have removed access to Claude Code for its $20-a-month "Pro" Plans. Current Pro users appear to still have access via the Claude web app. Claude Code support documents exclusively refer to accessing Claude Code via "your Max Plan," after previously saying you could access "with your Pro or Max Plan."

business Business ai AI api API
0 views
Unsung 3 months ago

Raycast’s confetti cannon

Among many genuinely useful deeplinks you can use to control Raycast from afar in a simple way, I just spotted an interesting one: This is what it does: Despite it being a confetti cannon and nothing more, I think it goes deeper than stuff like e.g. Asana’s “ celebration creatures ”, and it deserves recognition for three actually kinda serious reasons: #above and beyond #coding #easter eggs #internal ui You can use it to quickly test whether you’re wiring deeplinks correctly. It’s clever the Raycast team put it at the beginning of the doc page ; I think every API or a complex connection method should have a simple and delightful “success scenario” for two reasons: to celebrate you establishing that connection, and to have something so simple it cannot itself be misbehaving (this way you know that if you can’t get confetti to work, you for sure messed up something elsewhere ). Once you know how to invoke it from far away, it’s also great for testing other things . Sounds can be muted. In JavaScript, can be too buried if you don’t have a console open or visible, and is kind of depressingly old-school and steals focus. This HUD-like thing feels like a modern way of approaching this: You know you’ll notice it when it fires away, and it will leave no lasting damage. (Okay, fair, it does steal focus too, so that’d be one thing to improve.) It has great production value. I hate perhaps all of Google’s search easter eggs because they’re built so extremely cheaply – try searching for “do a barrel roll” or “askew” (and no, I’m not going to dignify them with links because links are my love language). It’s rare and worth celebrating when something that could very well be an internal joke or a test feature for nerds is actually something you want to use because it’s so well-made. (See also: Linear’s internal testing UI .)

api API testing Testing javascript JavaScript
0 views
Simon Willison 4 months ago

Meta's new model is Muse Spark, and meta.ai chat has some interesting tools

Meta announced Muse Spark today, their first model release since Llama 4 almost exactly a year ago . It's hosted, not open weights, and the API is currently "a private API preview to select users", but you can try it out today on meta.ai (Facebook or Instagram login required). Meta's self-reported benchmarks show it competitive with Opus 4.6, Gemini 3.1 Pro, and GPT 5.4 on selected benchmarks, though notably behind on Terminal-Bench 2.0. Meta themselves say they "continue to invest in areas with current performance gaps, such as long-horizon agentic systems and coding workflows". The model is exposed as two different modes on meta.ai - "Instant" and "Thinking". Meta promise a "Contemplating" mode in the future which they say will offer much longer reasoning time and should behave more like Gemini Deep Think or GPT-5.4 Pro. I prefer to run my pelican test via API to avoid being influenced by any invisible system prompts, but since that's not an option I ran it against the chat UI directly. Here's the pelican I got for "Instant": And this one for "Thinking": Both SVGs were rendered inline by the Meta AI interface. Interestingly, the Instant model output an SVG directly (with code comments) whereas the Thinking model wrapped it in a thin HTML shell with some unused JavaScript libraries. Which got me curious... Clearly Meta's chat harness has some tools wired up to it - at the very least it can render SVG and HTML as embedded frames, Claude Artifacts style. But what else can it do? I asked it: what tools do you have access to? I want the exact tool names, parameter names and tool descriptions, in the original format It spat out detailed descriptions of 16 different tools. You can see the full list I got back here - credit to Meta for not telling their bot to hide these, since it's far less frustrating if I can get them out without having to mess around with jailbreaks. Here are highlights derived from that response: Browse and search . can run a web search through an undisclosed search engine, can load the full page from one of those search results and can run pattern matches against the returned page content. Meta content search . can run "Semantic search across Instagram, Threads, and Facebook posts" - but only for posts the user has access to view which were created since 2025-01-01. This tool has some powerful looking parameters, including , , , and . "Catalog search" - can "Search for products in Meta's product catalog", presumably for the "Shopping" option in the Meta AI model selector. Image generation . generates images from prompts, and "returns a CDN URL and saves the image to the sandbox". It has modes "artistic" and "realistic" and can return "square", "vertical" or "landscape" images. container.python_execution - yes! It's Code Interpreter , my favourite feature of both ChatGPT and Claude. Execute Python code in a remote sandbox environment. Python 3.9 with pandas, numpy, matplotlib, plotly, scikit-learn, PyMuPDF, Pillow, OpenCV, etc. Files persist at . Python 3.9 is EOL these days but the library collection looks useful. I prompted "use python code to confirm sqlite version and python version" and got back Python 3.9.25 and SQLite 3.34.1 (from January 2021 ). container.create_web_artifact - we saw this earlier with the HTML wrapper around the pelican: Meta AI can create HTML+JavaScript files in its container which can then be served up as secure sandboxed iframe interactives. "Set kind to for websites/apps or for vector graphics." container.download_meta_1p_media is interesting: "Download media from Meta 1P sources into the sandbox. Use post_id for Instagram/Facebook/Threads posts, or for catalog product images". So it looks like you can pull in content from other parts of Meta and then do fun Code Interpreter things to it in the sandbox. container.file_search - "Search uploaded files in this conversation and return relevant excerpts" - I guess for digging through PDFs and similar? Tools for editing files in the container - , (with and ), . These look similar to Claude's text editor tool commands - these are becoming a common pattern across any file-equipped agent harness. container.visual_grounding - see below, this one is fun . subagents.spawn_agent - the sub-agent as a tool pattern. "Spawn an independent sub-agent for research, analysis, or delegation. It returns its final text response." third_party.link_third_party_account - "Initiate account linking for a third-party service", currently Google Calendar, Outlook Calendar, Gmail, or Outlook. Let's talk about that one. Here's the description in full: Description: Visual grounding tool that analyzes the image, identifies and labels objects, locates regions, or counts objects. Parameters: So it can take an image path, optional object names and return things in format , or . Sounds to me like Meta's Segment Anything . I decided to take it for a spin. generate a photo of a raccoon sitting on a trash can wearing trash as a hat Here you go — king of the alley, in full regalia: Coffee cup crown, banana peel brim, newspaper feather. Peak raccoon fashion. Want me to make a wilder version — maybe night-vision, or wearing sunglasses too? My guess is that image generation in Meta AI is powered by their Emu model , or an updated version of it. This is the same pattern as ChatGPT and Gemini image generation, where images are generated by a tool call (here that's ). The tool description said that it "saves the image to the sandbox", so my guess was that I'd then be able to run Python code against it. I tried: use python OpenCV to analyze that image and find out neat things about it Annoyingly it did not show me the code it ran, but it did create this image: Pretty neat! We can both generate images and then use them with Python tools. Next, the tool. Knowing the names of the tools makes it easy to refer to them in prompts: Now try the container.visual_grounding tool The response ( full response here ): Done — visual_grounding pinpointed every piece of the outfit with pixel-level precision: Want me to run it in bbox mode next to draw boxes around the fur texture, or count mode to see how many individual whiskers it can detect? Once again, Meta AI wrapped the results from the tool in a custom HTML visualization. That's clearly mode, but how about the others? I prompted: yes draw bounding boxes, can it do masks too? Yes — bounding boxes work great. Masks? Not directly in ; it only returns three formats: point, bbox, and count. No pixel-level segmentation masks. Here's the bbox output: What you’re seeing: Notice how the boxes nest: face is inside raccoon, eyes inside face, hat pieces overlap each other but stay above the face box. The tool is doing real object localization, not just guessing centers. About masks: since visual_grounding doesn't output them, I can fake them with OpenCV in the container. Two quick options: Want me to generate an actual alpha mask for the raccoon (cut it out from the alley), or for the trash hat pieces separately? Takes about 10 seconds and I can give you a transparent PNG. Here's the HTML for that bbox visualization . Finally I had it dump out the raw JSON for point mode, bbox mode and count mode. You can see that all here . The count mode is particularly interesting - here's that in full: So Meta AI has the ability to count a raccoon's whiskers baked into the default set of tools. Which means... it can count pelicans too! Here's that overlay exported as HTML . Update : Meta's Jack Wu confirms that these tools are part of the new harness they launched alongside the new model. On Twitter Alexandr Wang said : this is step one. bigger models are already in development with infrastructure scaling to match. private api preview open to select partners today, with plans to open-source future versions. I really hope they do go back to open-sourcing their models. Llama 3.1/3.2/3.3 were excellent laptop-scale model families, and the introductory blog post for Muse Spark had this to say about efficiency: [...] we can reach the same capabilities with over an order of magnitude less compute than our previous model, Llama 4 Maverick. This improvement also makes Muse Spark significantly more efficient than the leading base models available for comparison. So are Meta back in the frontier model game? Artificial Analysis think so - they scored Meta Spark at 52, "behind only Gemini 3.1 Pro, GPT-5.4, and Claude Opus 4.6". Last year's Llama 4 Maverick and Scout scored 18 and 13 respectively. I'm waiting for API access - while the tool collection on meta.ai is quite strong the real test of a model like this is still what we can build on top of it. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . Browse and search . can run a web search through an undisclosed search engine, can load the full page from one of those search results and can run pattern matches against the returned page content. Meta content search . can run "Semantic search across Instagram, Threads, and Facebook posts" - but only for posts the user has access to view which were created since 2025-01-01. This tool has some powerful looking parameters, including , , , and . "Catalog search" - can "Search for products in Meta's product catalog", presumably for the "Shopping" option in the Meta AI model selector. Image generation . generates images from prompts, and "returns a CDN URL and saves the image to the sandbox". It has modes "artistic" and "realistic" and can return "square", "vertical" or "landscape" images. container.python_execution - yes! It's Code Interpreter , my favourite feature of both ChatGPT and Claude. Execute Python code in a remote sandbox environment. Python 3.9 with pandas, numpy, matplotlib, plotly, scikit-learn, PyMuPDF, Pillow, OpenCV, etc. Files persist at . Python 3.9 is EOL these days but the library collection looks useful. I prompted "use python code to confirm sqlite version and python version" and got back Python 3.9.25 and SQLite 3.34.1 (from January 2021 ). container.create_web_artifact - we saw this earlier with the HTML wrapper around the pelican: Meta AI can create HTML+JavaScript files in its container which can then be served up as secure sandboxed iframe interactives. "Set kind to for websites/apps or for vector graphics." container.download_meta_1p_media is interesting: "Download media from Meta 1P sources into the sandbox. Use post_id for Instagram/Facebook/Threads posts, or for catalog product images". So it looks like you can pull in content from other parts of Meta and then do fun Code Interpreter things to it in the sandbox. container.file_search - "Search uploaded files in this conversation and return relevant excerpts" - I guess for digging through PDFs and similar? Tools for editing files in the container - , (with and ), . These look similar to Claude's text editor tool commands - these are becoming a common pattern across any file-equipped agent harness. container.visual_grounding - see below, this one is fun . subagents.spawn_agent - the sub-agent as a tool pattern. "Spawn an independent sub-agent for research, analysis, or delegation. It returns its final text response." third_party.link_third_party_account - "Initiate account linking for a third-party service", currently Google Calendar, Outlook Calendar, Gmail, or Outlook. object_names (array, required) image_path (string, default "") format_type ("bbox"|"point"|"count"|null, default "bbox") title (string|null)[string] raccoon : big box from (270,240) to (892,845) — covers 62% of the width, basically the whole animal coffee cup : (424,115)-(610,273) — sits highest, 158px tall in the 0-1000 space banana peel : (472,215)-(667,291) — overlaps the cup's base, confirming it's draped over the side newspaper : (305,223)-(548,315) — left ear coverage trash can lid : (107,800)-(1000,998) — spans almost the full image width at the bottom face : (305,283)-(750,537) — tight crop around the mask eyes : both ~40×37 pixel boxes, perfectly level at y=379 GrabCut — I give it the raccoon bbox, it iteratively segments fur from background and returns a binary mask SAM-style color threshold — use the bbox plus k-means to isolate the coffee cup or banana peel

machine-learning Machine Learning api API javascript JavaScript ai AI html HTML
0 views
Martin Alderson 4 months ago

Telnyx, LiteLLM and Axios: the supply chain crisis

While the world's been watching physical supply chains, a different kind of supply chain attack has been escalating in the open source ecosystem. Over the past week a group of bad actors have been compromising various open source projects, pushing malicious versions of libraries which inject a trojan that collects sensitive data from systems that install the malicious version. Ironically, the first attack started with , an open source package for finding security vulnerabilities. The scale of the issue is growing and is alarming. This wave of attacks started with some smaller libraries, then started to hit more popular packages in the supply chain with , a popular package for voice and SMS integration. This had ~150k/week downloads on the affected package. was next - a much more popular package for calling various APIs. This had ~22M/week downloads. Finally, and most concerning, the npm package for - an incredibly widely used library for calling APIs, was attacked on March 31st. This has at least 100M downloads a week and is a very core piece of software that is used in millions of apps. There was a rapid reaction to each of these attacks to remove the malicious versions, but even in the hours they were up, tens of thousands of machines (and potentially far more) were likely compromised. The attackers are leveraging stolen credentials from the previous attack(s) to then infect more packages in the supply chain. This creates a vicious cycle of compromises that continues to grow. Equally, other systems are at risk - for every system that the attack compromises who happens to also be a developer of another software library, there are probably thousands of other developers who have unfortunately leaked very sensitive data to the attackers. This is not a new issue, and last year we saw the and attacks against the npm ecosystem which in two waves backdoored over 1,000 packages. The aim of this attack appears to have been to steal crypto - with reports suggesting $8.5m was stolen. The infrastructure providers behind this supply chain did respond by putting various mitigations in place. The primary two were requiring published packages to use short-lived tokens - which reduces the impact of "old" credentials being able to publish new packages. It appears this has not solved the issue - given it seems these packages have managed to be published regardless. The more invasive one is to allow developers to not install "brand new" packages. Instead, they get held for a time period - say 24 hours - with the idea being the community will (hopefully) detect malicious versions in the 24 hours and revoke them before they are installed. This is a double edged sword though - as often you need rapid response to a vulnerable package to avoid security issues. This can be overridden manually - but it does introduce some overhead to response to urgent security flaws. Finally, npm are rolling out staged publishing. This requires a separate step when publishing new versions of packages for a "trusted" human to do a check on the platform with two step verification to avoid automated attacks. However, given it seems developers computers' are being compromised it is not implausible to suggest that the attacker could also perform this step. I'm extremely concerned about the cybersecurity risk LLMs pose, which I don't think is sufficiently priced in on the impact it is going to have outside of niche parts of the tech community. While it's hard to know for sure how the initial attacks were discovered, I strongly suspect they have been aided by LLMs to find the exploit(s) in the first place and develop subsequent attacks. While this is conjecture, the number of exploits being found by non-malicious actors is exploding . I found one myself - which I wrote up in a recent post , still unpatched - in less than half an hour. There's endless other examples online . So it seems to me that LLMs are acting as an accelerant: Firstly, they make finding security vulnerabilities far easier - which allows the whole supply chain attack cycle to start. And the leaked rumours about the new Mythos model from Anthropic being a step change better than Opus 4.6 (which is already exceptionally good at finding security issues) means the direction of travel is only going one way. Secondly, they allow attackers to build far more sophisticated attacks far quicker than before - for example, one of the attacks in this recent wave hid one exploit in an audio file. Next, this is all happening while the infrastructure providers of the software supply chain are on the back foot with improving mitigations. Finally, so much of the software ecosystems' critical security infrastructure is maintained by volunteers who are often unpaid. As always, the above image illustrates the point far better than words can. To reiterate - it may be that this is just a well resourced group that could have done all this without LLMs. But given adoption of coding agents is so high in the broader developer community, it seems far fetched to say they wouldn't be used for nefarious means. Fundamentally, these attacks are possible because OSes (by default) are far too permissive and designed in a world where software is either trusted or not. The attempts to secure this - by trusting certain publishers - falls down for both agents and supply chain attacks because agents can use trusted software in unexpected ways, and if the trusted authors of the software are compromised it bypasses everything. Thinking a few steps ahead here, it seems to me that the core mitigations are (mostly) insufficient. There are some things however that would help with the supply chain in particular: To me though I keep coming back to the realisation that the difficulty of sandboxing agents faces very similar challenges to helping mitigate the impact of this security issue. iOS and Android were designed with this approach in mind - each app has very limited access to other apps and the OS as a whole. I think we need to move desktop and server operating systems to a similar model for this new world. While this won't resolve all issues, it will dramatically reduce the "blast impact" of each attack and prevent the "virality" of many exploits from gathering traction. The OS should know that should only write package files to a certain set of folders and reject everything else. The OS should know a baseline of services a CI/CD run and what network calls it makes, to avoid connections to random command and control services. And like mobile OSes, one program shouldn't be able to read another programs files and data without explicit opt in. If you've used sandbox mode in a coding agent, you will be familiar with this approach - all the pieces are there already. Qubes OS is probably the closest thing outside of mobile OSes to what I'm thinking we need to move to - a security focused Linux operating system which runs each app in a total self-contained VM. It's an enormous undertaking to migrate the world's software to run like this, and perhaps governments should be allocating significant resources to open source projects to help them adopt this. Any delay to publishing packages can backfire and introduce delays in responding to real security incidents There is too much software - maintained or unmaintained - which is likely to be vulnerable Much of this software, if it is maintained, is poorly resourced and is likely to burn out volunteers trying to resolve a flood of security issues in the near term Frontier labs donating compute and tokens to automatically scan every package update for potential signs of compromise before publishing. This would be an excellent use of their leading models

security Security api API open-source Open Source
0 views