Krebs on Security 3 days ago

Lessons Learned from CISA’s Recent GitHub Leak

The Cybersecurity and Infrastructure Security Agency (CISA) has issued a postmortem on a recent data leak in which a contractor published dozens of internal CISA credentials — including AWS Govcloud keys — in a public GitHub repository for almost six months before being notified by KrebsOnSecurity. Experts say the gaps identified in the agency’s initial response provide important lessons that all security teams should absorb. On May 15, 2026, the security firm GitGuardian asked for help in notifying CISA about the existence of a public GitHub repository called “Private CISA” that included 844 MB of sensitive CISA-related data. One of the exposed files, titled “importantAWStokens,” included the administrative credentials to three Amazon AWS GovCloud servers. Another file — “AWS-Workspace-Firefox-Passwords.csv” — listed plaintext usernames and passwords for dozens of internal CISA systems. CISA quickly acknowledged our initial alert, but took more than 48 hours to invalidate the AWS keys and many other important secrets leaked in the GitHub repo. In its report on the data leak , CISA said the complexities of the agency’s systems and interconnections with federal and industry partners caused its key rotation to take longer than anticipated. “Drawing on this experience, CISA encourages others to maintain mature and well-tested key management capabilities,” the report notes. CISA also admitted it can do better when it comes to responding to security incident notifications from external parties. The postmortem stresses that clear and distinct reporting channels are essential to ensure that incidents affecting the organization itself are handled differently from those involving its products or customers. “In CISA’s case, these channels were not well defined, leading the security researcher to try multiple avenues – including emailing the contractor, submitting through CISA’s vulnerability disclosure platform (which is intended for vulnerabilities impacting the broader cybersecurity community), and ultimately involving a reporter,” reads the analysis written by Preston Werntz and Brad Libbey , the acting chief information officer and acting chief information security officer at CISA, respectively. CISA said it is refining its reporting channels to make them easier and faster for researchers. “Additionally, while many researchers rely on the security.txt file, organizations can ensure clarity by publishing reporting instructions in multiple prominent locations,” the CISA authors wrote. Guillaume Valadon , the GitGuardian researcher who first contacted KrebsOnSecurity about the exposed CISA credentials, said CISA ignored nine automated alerts about the exposed credentials prior to our notification on May 15. Valadon’s company constantly scans public code repositories at GitHub and elsewhere for exposed secrets, automatically alerting the offending accounts of any apparent sensitive data exposures. “Letting nine notification emails go unanswered is how a one-day incident becomes a six-month exposure,” Valadon wrote in an analysis of CISA’s report. “Make it trivial to report a leak about you, not just about your products. The person reporting a leak to you is not the threat. Publish a security.txt , but do not stop there. Put reporting instructions in several prominent places, and make sure a report about your own infrastructure does not land in a product-bug queue.” The report’s authors also emphasized the importance of continuously scanning public code repositories like GitHub for exposed secrets, and said CISA has since rotated all secrets and created an action plan to improve management of developer secrets and to better monitor for them going forward. The report notes that while CISA had developed a playbook for responding to cybersecurity incidents, that playbook somehow didn’t include what to do in situations involving GitHub or other cloud services. Valadon said the report validates the need to scan continuously — not just quarterly — for exposed secrets. “The Private-CISA repository sat public for six months,” Valadon wrote. “Continuous monitoring of public GitHub surfaced it. Comprehensive internal scanning could have caught the plaintext passwords and committed backups long before they left the building.” CISA gave itself passing grades on several areas of security preparedness that it said helped the agency gauge the scope and impact of the exposed secrets, including enhanced logging capabilities, and the adoption of zero-trust principles in both its production and development systems. CISA said those detailed logs allowed it to show that no customer or mission data was exposed, and that the leaked credentials were not used outside of CISA’s environments. The agency said the contractor who exposed the secrets had their system access revoked. Valadon reckons the biggest takeaway is the CISA postmortem itself, and praised the agency for being transparent about what worked and what didn’t. “To my knowledge, it is also the first time a national cybersecurity agency has publicly advocated for secrets scanning and for simplifying relations with security researchers,” Valadon wrote. “That is exactly the incident communication we should expect from every organization.”

AWS Cloud Security

Stratechery 3 days ago

Apple Sues OpenAI, Apple’s Real Problem

Apple is suing AI for stealing trade secrets; there is one guilty employee, but this mostly feels like lashing out.

AI Security Business

Karboosx 6 days ago

How to Create Your Own Decentralized Messenger Protocol

Ever wondered how to build a decentralized messenger without any central servers? It's all about federation - just like email! In this post, I'll show you how to design a simple protocol from scratch, from server discovery using .well-known files to handling end-to-end encryption.

Backend Tutorial Security

Jeff Geerling 6 days ago

QuadRF can spot drones and see WiFi through my wall

The QuadRF (pictured above) a phased-array radio built around a Raspberry Pi 5 and an FPGA board with picosecond-level timing. It does advanced signal processing and beamforming. It can see WiFi through walls and track drones in flight. If the open source community can come up with something like this, just imagine what governments are capable of. When you plug a computer into a network, tools like Wireshark can show all the hidden traffic you might not even know is there. WiFi packets are the same, but those travel through the air, allowing snooping without physical access.

Security Hardware

Kev Quirk 6 days ago

A Rant About Modern Cars

I recently bought a new Peugeot and the experience of getting setup on their online platform has been painful to say the least. Yesterday I picked up my shiny new (to me) Peugeot E-3008 GT. It's a beautiful car with lots of bells, whistles, and toys. I had my little MG EV for around 2.5 years, and it served me well, but I wanted something bigger, with more range. So I opted for the Peugeot. Anyway, since this is a modern car, it no longer comes with an owner's manual. Instead you need to install an app and read the manual there. So I did that and duly signed up for a Peugeot Connect account - all standard procedure in this internet age we find ourselves in. That was until it came to generating a password. I did my usual and generated a 30 character, complicated password with Bitwarden , only to be greeted with this ridiculous password complexity error: So my 30 character , random string password is apparently weak and the only way to make it secure is reduce it's length (and complexity) by ~50%. Not only that, I had to abide by a slew of other arbitrary rules along the way. I tried to generate a 16 character PW with Bitwarden a couple times, but the error persisted. So I ended up jumping over to Gemini, pasting the requirements in, and asking it to give me a password. Being the sycophantic AI that it is, it spat out a password that conformed to Peugeot's ridiculous rules. Or so I thought... OK, so the password Gemini generated for me was . Let's see how it stacks up to the requirements: So why the fuck is the password still being rejected as too weak ? I assume it's poor wording on Peugeot's part, but I ended up just typing gobbledegook into the field until it passed. Interestingly, also passed and was reported as a "very strong" password: For the record, is NOT a very strong password. Don't use that. Ever. I'm astonished that this is still an issue in 2026. Why on earth can't manufacturers get this simple shit right? It's basic stuff. All you're doing here is forcing people to use shitty passwords. I finally got into my bloody Peugeot account and tried to enable to Connect features so I can do things like control air-con from the app, only to find that it costs £90 (~$120) per year! This isn't a piece of hardware that I'm paying for. It's literally £90/year for a switch to be flipped in some software. Utter. Fucking. Robbery. Peugeot, you should be ashamed of yourselves. Aside from this, the new car is lovely. 🙃 Thanks for reading this post via RSS. RSS is ace, and so are you. ❤️ You can reply to this post by email , or leave a comment . 8-16 characters ✅ An uppercase letter ✅ A lowercase letter ✅ A special character from the list ✅ No sequential characters ✅

Programming Security

Farid Zakaria 1 weeks ago

Who does Anubis actually stop?

I have been working on a patch to the Linux kernel to support for the interpreter ( ) via bpf in [ thread ]. Of course I’m leveraging an LLM to help me do this! To pre-seed the context of the LLM, I asked it to read the https://lore.kernel.org/ thread. Uh oh. Looks like they have adopted Anubis , which is an HTTP proxy that requires proof-of-work before allowing access to the resource. Did this really do anything? Unfortunately, no. My AI diligently came up with anubis-fetch , which you can find at https://github.com/fzakaria/anubis-fetch . The tool tries to natively solve the proof of work or, as a last resort, will launch Chromium to visit the URL. This tool also impersonates a real Chrome TLS/JA3 fingerprint natively via req so it clears passive Cloudflare blocking too. ☝️ So who did we stop? The exact adversary Anubis targets defeats it trivially. The whole use of Anubis feels regressive and marginalizes those without access to “good” AI. For a scraper, solving the Anubis challenge is a one-time, amortized-to-zero cost since the cookie can be cached and reused. For a human, it’s seconds of spinner, battery drain on every fresh visit. They can’t amortize anything amongst each other. This “regressive tax” is paid even more so by those with weaker devices or who access the content on their phone. Clients that don’t leverage JavaScript (e.g., text browsers (w3m/lynx), screen readers, RSS readers) are completely left out. Did deploying Anubis stop any of the aforementioned bot-farms or are they mildly inconvenienced when they had to augment their bots to support a new proof of work solution briefly? The irony is that Anubis’s goal is to stop AI but it was incredibly easy for AI to circumvent it and yet the cost to humans and an open web remains. With the presumption Anubis is now a regressive tax, how much does it cost us? Every number here is a rough estimate. This is not a environmental argument at all since the bot-farmers and AI tools themselves are using many orders of magnitude more energy. Nevertheless, it’s interesting to see how much time is spent doing proof-of-work challenges that marginalize people. Difficulty is the number of leading zero hex characters the hash must have, so the expected work per solve is hashes. Difficulty 4 is the common default. Rates assumed: ~50 MH/s native (Go), ~0.5 MH/s in-browser JS; “felt” wall-clock includes page load, the worker, and the reload. Let be the number of Anubis challenge-solves per day, worldwide. Assume a felt time of and device energy per solve (screen + CPU). Collectively we are wasting an impressive amount of time waiting for access to websites; time we didn’t spend before the AI era. As a human, time is precious and finite to me, whereas to a robot it is not. Human-time / year = Energy / year (kWh) =

Security JavaScript Open Source

Krebs on Security 1 weeks ago

Felons, Fraudsters Flog Offensive Cybersecurity Startup

A cybersecurity startup dangling millions of dollars to acquire zero-day security vulnerabilities in popular software is run by a pair of far-right conspiracy theorists and convicted felons whose most recent ventures included fake intelligence companies and a now-defunct AI-based lobbying platform they operated under assumed names. The X/Twitter account IRIS C2 (@C2IRIS) has gained more than 4,000 followers since its creation in January 2025, posting frequently about security vulnerabilities, AI and software exploits. IRIS C2 says it is a company in McLean, Va. that sells offensive cybersecurity capabilities. The IRIS C2 website dangles the possibility of million-dollar payouts for exploits to attract talent. “Our business model is this,” reads a pinned post on top of the IRIS C2 account on X. “Attract the very best vulnerability researchers and exploit developers in the world to join our company. This mostly revolves around junior engineers with raw talent/extremely high IQ. We don’t care if they have a college degree/industry experience.” The website linked in that profile — irisc2[.]com — says the company is hiring for a number of open positions, and a recent post on its LinkedIn page enthuses about an overwhelming number of applications from potential employees. The website claims IRIS C2 is in the business of acquiring “zero-day exploits, individual primitives, partial chains, and full capabilities across all major platforms. Payouts range from $10,000 to $7 million depending on target, reliability, and operational value.” The government contracting portal g2exchange.com reports that irisc2[.]com is operated by a business based in Virginia called Calvexa Group LLC . The “contact” link on the website for Calvexa Group — calvexagroup[.]com — forwards visitors to irisc2[.]com. G2Exchange shows that while Calvexa Group LLC is registered as a federal contractor, it does not appear to be working on any direct government contracts. A search on the Arlington, Va. address listed in the incorporation records for Calvexa Group LLC finds the property is occupied by Jack Burkman , the 60-year-old founder and managing partner of the lobbying firm Burkman & Associates . When approached with questions about IRIS C2, Burkman referred further inquiries to his longtime associate, 28-year-old Jacob Wohl . Jack Burkman (left) and Jacob Wohl, at a press conference in August 2020. Image: Wikipedia. Burkman and Wohl have a storied history of creating fake intelligence companies and using them to spread false claims about and frame public figures, including fabricated sexual assault claims against then FBI director Robert Mueller , and Pete Buttigieg , then mayor of South Bend, Indiana and a Democratic candidate for the presidency. In 2019, Burkman and Wohl held press conferences falsely alleging extramarital affairs by Sen. Elizabeth Warren (D-Mass.) and then-2020 presidential candidate Kamala Harris . In the wake of the 2020 presidential election, Wohl and Burkman were prosecuted by multiple U.S. states for making thousands of robocalls to residents of battleground states and disseminating false claims about mail-in ballots. They were indicted in Cleveland on 15 felony counts of orchestrating a robocall scheme aimed at suppressing the black vote in Detroit, and were sentenced in late 2025 to probation after their appeals to dismiss the charges were rejected. In 2022, Wohl and Burkman both pleaded guilty to a single felony charge of telecommunications fraud in Ohio, and sentenced to a fine, probation, and community service. In March 2023, a judge in a New York civil case ruled that Wohl and Burkman had violated federal and state civil rights laws, and the two agreed to pay a $1 million settlement. In June 2023, the Federal Communications Commission (FCC) imposed a $5.1 million fine against Wohl and Burkman for their robocall campaigns, at the time the largest fine ever sought by the FCC under the Telephone Consumer Protection Act. Jacob “Jay” Wohl’s GitHub account. By the age of 17, Wohl had started multiple investment firms, and cultivated the nickname “Wohl of Wall Street” after appearing on Fox News in 2015 to discuss his new hedge funds. In 2017, the Arizona Corporation Commission charged Wohl and his investment funds with 14 counts of securities fraud, and ordered him to pay $35,000 in restitution. In 2019, Wohl pleaded guilty in California to four felony counts of selling unregistered securities and was sentenced to two years of probation. The market for previously unknown security vulnerabilities has always been populated by a colorful mix of researchers, academics, charlatans, clout-chasers and people actively involved in cybercrime communities. But the market for selling offensive security services to the U.S. government tends to be far more circumspect. Plenty of government contractors recruit vulnerability researchers and pay for the exclusive rights to novel software exploits, yet none of them do so quite as brazenly and openly as IRIS C2. Recent posts from the Twitter/X account IRISC2 (@c2iris). Indeed, KrebsOnSecurity was unaware of IRIS C2 until last month, when an attendee at a regional cybersecurity conference shared that Wohl and Calvexa Group were pestering people at the conference about selling their vulnerability research. In an interview with KrebsOnSecurity, Wohl said Mr. Burkman was not involved in the day-to-day operations of IRIS C2. Wohl shared that IRIS C2 originally began as a penetration testing company, but shifted its focus recently to selling phone-hacking services to the government. Several times throughout the interview, Mr. Wohl mentioned working on federal government contracts, but when pressed for specifics said he was not at liberty to speak publicly about them. Mr. Wohl said he does not have any formal education or training in computer science or information security, and that most of his knowledge on the matter is self-taught. “I know more about tech than anyone,” Wohl bragged. “My background has always been extremely technical, and I’ve always been deeply into tech. People know me as someone who is able to create spectacularly exquisite capabilities that would make your head spin.” Wohl said security researchers bring the company unique vulnerability findings “on a regular basis,” but that in many cases those findings are preliminary and not fully fleshed-out. “Let’s say someone finds a flaw in a media decoder on a phone,” Wohl said. “A lot of times what we receive is an exploit primitive, where the idea is there but the [execution] needs work. You need that exploit to be stable and reliable, and that’s what we do.” Wohl claims IRIS C2 has approximately 40 employees, although he said none of them are allowed to list their employment on LinkedIn for operational security reasons. In May, the author of the IRIS C2 account on X said that his girlfriend had no idea what he did for a living. But if IRIS C2 has any other employees, they may be similarly unaware of Mr. Wohl’s history of outright fabrications — or even his real name. In September 2024, Politico reported that Burkman and Wohl were bragging about big companies supposedly buying services from their now-defunct company LobbyMatic , which claimed to use artificial intelligence to assist in political lobbying efforts. However, Politico found the pair were running the company using pseudonyms, with Wohl reportedly adopting the name “Jay Klein” and Burkman using the moniker “Bill Sanders.” Politico reported that two of the former LobbyMatic employees resigned after learning of their true identities, while other employees only learned after they had left the company.

Crime Business Security

Sean Goedecke 1 weeks ago

C2PA only works if everything is signed

The European Union AI Act is Europe’s attempt to comprehensively regulate AI usage. A big part of that is the requirement that AI-generated content be identifiable: either tagged with a watermark or with what the Act calls “digitally signed metadata” 1 . Since all this becomes enforceable in a month, it’s worth figuring out if it makes any sense. I recently discussed AI watermarking at length in Text AI watermarks will always be trivial to remove . What about digitally signed metadata? The most well-known implementation of digitally signed metadata is C2PA Content Credentials, which incorrectly 2 claims to be the technology that the AI Act gives as an example of how to do signed metadata properly. The idea here is that every single image file should contain unspoofable authorship metadata . Here’s my position on it: Lots to unpack. Let’s start by considering images, since that’s the easiest case. When an AI tool generates an image, that tool should include a “made by ChatGPT” disclaimer in that image’s metadata. Likewise, when a camera takes a photo, that camera should include a “taken by a camera” disclaimer. C2PA uses two strategies to protect this metadata: Each physical camera (or phone) has its own private key, for obvious reasons 3 . How do we know that those millions of private keys are trusted? Via PKI , like HTTPS: each camera’s private “certificate” (which contains its public key) is signed by the manufacturer’s well-known private key, so the chain of authenticity can be verified as long as you have (say) Apple’s root public key 4 . What happens if you then edit your photo in Photoshop? Photoshop will leave the camera’s metadata untouched, but will layer a “also, Photoshop was used” piece of metadata over the top, signed with Adobe’s private key (well, with the private key associated with your official copy of Photoshop, which is signed by Adobe’s official private key). Likewise, if you ask ChatGPT to generate an image for you, ChatGPT will sign its “made by ChatGPT” metadata with OpenAI’s private key. In theory, every single image could contain unforgeable C2PA metadata, allowing software like Twitter to trivially distinguish real photos from fake ones. Right now, C2PA does not have anything like the adoption it’d need to work. It’s hard to find hard data on how many images in the wild use C2PA, but FotoForensics reports around a dozen per week (so around 600 out of the 900,000 images processed each year). This is even worse than it sounds, because basically all of the signed images are AI-generated. The adoption rate of C2PA for human-generated images is much, much lower: so far, Google’s Pixel 10 is the only phone camera to sign photos by default. The iPhone doesn’t sign photos. If almost all AI images are C2PA-signed, but almost no human-generated images are, consumers have no reliable way of identifying AI content, because anyone who wants to pretend their AI content is human can simply remove the signature. For C2PA to succeed, it needs to be on every camera and every phone, so that a photo with no signature is rare and suspicious. Is that realistic? Actually, I think it is. The appetite (at least in the EU) to regulate AI will increase over time, and while the current EU AI Act only mandates that AI-images are tagged (which by itself is useless), it’s plausible that some future regulation will enforce tagging of all images. Another adoption problem that must be solved for C2PA to work is preservation . Right now, if you download a C2PA-tagged image, send it as a Facebook message, then re-download it, the C2PA manifest is stripped out. Most images we see on the internet have passed through some social media asset server at least once. All of these social media companies would need to update how they re-encode image content in order to preserve the C2PA data 5 . This would almost certainly require more regulation: C2PA adds tens or hundreds of kilobytes to each file, which at social media scale is big money 6 . Could a clever attacker forge a C2PA signature? Kind of. Neal Krawetz, who seems to have led the anti-C2PA charge, points out that with a camera development kit it’s straightforward to trick a digital camera into thinking that it’s taking an image when in fact it’s being fed one. This is very much not my area, so please write in if you know more about camera hardware and you think I got this wrong. I suppose you could also take a photo of an AI image on a screen, though I imagine you’d have to be careful to make it look real. If you exclude physical attacks on a digital camera, I think C2PA is more robust. You can sign a photo with a self-signed certificate, but the C2PA spec and docs say that validators must check that your certificate bubbles up to the official C2PA trust list . This list currently contains only 26 certificates, and there’s a whole process for being added to it. That’ll slow down adoption, but at least it makes it hard to forge 7 . We’ve been talking exclusively about images, but it’s more or less the same story for any type of content. If the file doesn’t support JUMBF metadata (say, an Excel file or a PDF), then the C2PA metadata has to live in a “sidecar”: a separate file, probably on some Microsoft or Adobe content server, which contains the signed checksum and the data about who created the file. However, the distinction between “real” and AI-generated content is fuzzier when you’re not talking about images. Here’s a trivial example: if I ask ChatGPT to create an Excel spreadsheet for me, the file will be tagged as AI-generated, but I can simply copy/paste the content into a new Excel doc and save it, which will tag it as human-generated 8 . There’s no software tool that can identify when I’m retyping some AI-generated text (except for perhaps text fingerprinting , which has its own raft of issues). There are also interesting questions around key management. ChatGPT and other AI tools have an easy problem — their users are all online, and so the files can be signed server-side — but how do you sign files created via Photoshop/Excel/Word? If the user doesn’t have internet, do you use some kind of local key? If so, how do you prevent that key being extracted and used to sign AI-generated content? Finally, is it a civil liberties problem to automatically fingerprint every photo? Does it make it impossible to be a whistleblower if every photograph can be traced back to your camera? I think this is a complicated question, but in short: I’d expect whistleblowers to already strip EXIF metadata from their images, C2PA metadata is similarly trivial to strip out, and overall I think image attribution is positive for whistleblowers because it heads off “this was AI-generated” responses. C2PA is probably here to stay. But it isn’t useful now, and won’t be useful until two huge programs of technical work are completed: This will be a long organizational process, since each manufacturer must go through the approvals process (or decide to start their own competing system), evaluate the legal ramifications of storing attribution data in images, and so on. It will be a long technical process, because C2PA metadata is a substantial fraction of image sizes: storing it will add many petabytes of content. Of course, just because C2PA isn’t useful doesn’t mean we’re not all going to do it. Lots of companies are under pressure to signal that they care about AI safety and to head off regulatory attack. “We’re cryptographically signing AI-generated content” is a compelling “we’re doing something ” pitch, particularly for people who aren’t technically savvy enough to understand the limitations. In the near term, I expect large AI-involved companies to invest a substantial amount of engineering effort in C2PA-related activity. In the long run, once everyone gets on board, I think C2PA could end up working well. It’s awkward in some ways, but “attest content via a PKI certificate chain” is a good idea. Is it possible to defeat? Yes, of course. By design, private keys will be in the user’s hands — in their cameras, in their local versions of Photoshop or Microsoft Word, in their phones — so sufficiently technical users will be able to crack them out or use them to sign whatever content they want. I still think C2PA will end up stemming the tide of AI content, because most users are not going to be sophisticated enough to perform attacks like this. However, we should still retain some skepticism of unlikely-looking content, even if it has “created by a human” in its C2PA metadata. See sub-measure 1.1.1 of the Act’s associated Code of Practice . While an early draft of the Code of Practice made an offhand mention of Content Credentials (in the caption of a picture), that was stripped out. The contents of the Act and the final Code of Practice don’t contain “C2PA” or “Content Credentials” (you can search for yourself here ). Otherwise if you cracked the key out of one Sony camera, you could spoof content from any Sony camera. In practice there are usually more “links in the chain”: a device will be signed by some intermediate certificate, which in turn will be signed by another intermediate certificate, which will be signed by the root certificate. That’s because the root key is so valuable. If an intermediate private key leaks, it can be revoked and replaced (via the root key), but if the root key leaks, it would take years to rebuild the network of trust. So almost all signing is done by intermediates, and the root key stays on a USB drive locked in a safe somewhere. Not to mention that the whole point of C2PA is that these social media companies will be displaying a “human or AI” sticker in their UI, which will require retaining the metadata. C2PA allows for storing the manifest content as a separate file, and just including a manifest url in the image metadata itself, but that doesn’t solve the cloud provider problem: they still have to store all the files on-disk somewhere. I think this defuses Neal Krawetz’s “worst-case scenario” . I downloaded his forged image, and (as expected) it gets flagged as “signed, but we don’t trust the root”. I think Krawetz was right at the time, though, since the official “trust list” was only launched in mid-2025. You could do the same thing with images by copying into Photoshop or Paint, but while that’d obscure the AI source, it would still be clear that the photo wasn’t taken by a camera. C2PA broadly makes sense and is a good idea It is pointless to use C2PA for AI-generated images only It will take many years for C2PA to be adopted across all images Because C2PA makes such great safety theater, we’re going to see a lot of hue and cry about it long before it becomes useful The metadata must be signed by some trusted private key The metadata contains a hash of the file’s contents, so you can’t copy an existing signature onto a new file Every camera manufacturer (including phones) must C2PA-sign all images by default Every social media company must retain the C2PA metadata on uploaded images See sub-measure 1.1.1 of the Act’s associated Code of Practice . ↩ While an early draft of the Code of Practice made an offhand mention of Content Credentials (in the caption of a picture), that was stripped out. The contents of the Act and the final Code of Practice don’t contain “C2PA” or “Content Credentials” (you can search for yourself here ). ↩ Otherwise if you cracked the key out of one Sony camera, you could spoof content from any Sony camera. ↩ In practice there are usually more “links in the chain”: a device will be signed by some intermediate certificate, which in turn will be signed by another intermediate certificate, which will be signed by the root certificate. That’s because the root key is so valuable. If an intermediate private key leaks, it can be revoked and replaced (via the root key), but if the root key leaks, it would take years to rebuild the network of trust. So almost all signing is done by intermediates, and the root key stays on a USB drive locked in a safe somewhere. ↩ Not to mention that the whole point of C2PA is that these social media companies will be displaying a “human or AI” sticker in their UI, which will require retaining the metadata. ↩ C2PA allows for storing the manifest content as a separate file, and just including a manifest url in the image metadata itself, but that doesn’t solve the cloud provider problem: they still have to store all the files on-disk somewhere. ↩ I think this defuses Neal Krawetz’s “worst-case scenario” . I downloaded his forged image, and (as expected) it gets flagged as “signed, but we don’t trust the root”. I think Krawetz was right at the time, though, since the official “trust list” was only launched in mid-2025. ↩ You could do the same thing with images by copying into Photoshop or Paint, but while that’d obscure the AI source, it would still be clear that the photo wasn’t taken by a camera. ↩

Security AI

James Stanley 1 weeks ago

ProofToken16: My Proposal for Private Decentralised Age Verification

So there's this whole thing now about how under-16s aren't meant to be allowed on social media, and this is supposed to be enforced through technical means using Zero-Knowledge Proofs, or something. Here is my proposal. Let's disregard the declaration of the independence of cyberspace for a moment and assume that actually we changed our mind and we do want the government to be mandating age verification on the web, and let's say we don't mind the fact that enforcing "no social media for under-16s" actually places the burden on everyone over 16 to prove it. Let's assume the requirements are: websites have a way to test whether the user is over 16 websites can't learn any private information other than whether or not the user is over 16 multiple decentralised issuers can provide proofs-of-age new issuers can be created without websites having to be updated to accept them the website can't tell which issuer you used this whole thing isn't secretly a tool to expand the surveillance state, it is in fact narrowly implementing only age verification Where everyone else is going wrong with private age verification is that they're trying to encode extra information about the user in the proof-of-age (like their actual date-of-birth, or the actual issuer), and then construct a zero-knowledge proof that hides this information from the website. What I propose is that the proof-of-age that is issued to a user over the age of 16 only contains the fact that they are over 16 . We purposely don't put any more information in the proof, and that way we can be sure that no vulnerability can leak this extra information, and we save a load of complexity. When the proof only contains that single fact, that the user is over the age of 16, it turns into just containing a single boolean. And since we will never bother to issue a proof where that boolean is false , the proof doesn't even need to contain the value! All proofs-of-age contain equivalent content (asserting that the user is over 16) so we can take the value of the boolean to be true , as long as the proof is valid. So all that remains is to issue an empty-string proof to everyone who is over 16, and let websites check that they are genuine. So let's pick a 4096-bit private key, keep it secret from under-16s, and say that knowledge of that key is the proof of being over 16. This is the ProofToken16 . Use whatever kind of message-signing scheme you want to let the user prove that they know the ProofToken16 . Since we only provide the ProofToken16 to people who are over 16, the ability to sign a message using it is proof of age. QED. Any new issuer can start up a service where they check your age however they want and reveal the ProofToken16 if you are over 16, with no centralised control. And, crucially, the decentralised nature will not be a backdoor allowing under-16s to create false proofs. To become an issuer you need to know the ProofToken16 yourself ! Since under-16s won't know it, they won't be able to issue it. Since all of the issuers are providing the same ProofToken16 , the website doesn't have any way to tell the issuers apart so the scheme does not even leak that information. There are a couple of minor drawbacks to this scheme, I admit: I'm actually not completely sold on the idea that age verification is even a good idea. What happened to a cyberspace without borders? What happened to the free and unencumbered flow of information? Humanity has spent tens of thousands of years building up our technological capabilities, to the point that we now have a global communications network that lets any two people on the planet communicate with each other practically instantaneously and practically for free. But it seems like in the last few decades we have been putting more effort into limiting our technological capabilities than expanding them, this can not end well, this is how we architect the downfall of civilisation, please be careful. And beyond that, requiring grownups to submit themselves to age verification before they can communicate with each other is disrespectful, undignified, and humiliating. Someone who knows the ProofToken16 might give it to someone who is under 16. Note this is not a weakness unique to my proposal. What stops someone from handing their proof to a child under any other scheme? If the proof reveals nothing other than whether the user is over-16, then a website has no way to check whether all of its users are using the same proof anyway. It is a fundamental technical impossibility to verify a user's age with cryptography, the closest we can come is to issue proofs-of-age and teach people not to share. I think that my ProofToken16 scheme is as good as you can do under the proposed requirements. Any alternative has at least the same flaws and possibly others besides. So I have already implemented it. I have created a ProofToken16 number, if you want to find out what it is then email me with proof of age and I will provide it. This will then entitle you to participate in my MatureChat social media site for over-16s only. You will have to do message signing at the command line every time you login, for now, but I am working on a browser extension to automate it. (The key will also entitle you to start your own ProofToken16 issuance service, which could one day be very lucrative!) See you on the other side. Go to MatureChat » websites have a way to test whether the user is over 16 websites can't learn any private information other than whether or not the user is over 16 multiple decentralised issuers can provide proofs-of-age new issuers can be created without websites having to be updated to accept them the website can't tell which issuer you used this whole thing isn't secretly a tool to expand the surveillance state, it is in fact narrowly implementing only age verification I'm actually not completely sold on the idea that age verification is even a good idea. What happened to a cyberspace without borders? What happened to the free and unencumbered flow of information? Humanity has spent tens of thousands of years building up our technological capabilities, to the point that we now have a global communications network that lets any two people on the planet communicate with each other practically instantaneously and practically for free. But it seems like in the last few decades we have been putting more effort into limiting our technological capabilities than expanding them, this can not end well, this is how we architect the downfall of civilisation, please be careful. And beyond that, requiring grownups to submit themselves to age verification before they can communicate with each other is disrespectful, undignified, and humiliating. Someone who knows the ProofToken16 might give it to someone who is under 16. Note this is not a weakness unique to my proposal. What stops someone from handing their proof to a child under any other scheme? If the proof reveals nothing other than whether the user is over-16, then a website has no way to check whether all of its users are using the same proof anyway. It is a fundamental technical impossibility to verify a user's age with cryptography, the closest we can come is to issue proofs-of-age and teach people not to share.

Crypto Security

Nicky Reinert 1 weeks ago

9 – Simple, Secure Browser-Based Text Sharing

9 is a free, anonymous, browser-based tool that sends text directly between two devices via encrypted WebRTC – no server, no signup.

Web Development Security

Krebs on Security 2 weeks ago

FBI Seizes NetNut Proxy Platform, Popa Botnet

The Federal Bureau of Investigation (FBI) said today it worked with industry partners to seize hundreds of domains associated with NetNut , a sprawling residential proxy service operated by the publicly-traded Israeli company Alarum Technologies [NASDAQ: ALAR]. The action comes roughly two weeks after KrebsOnSecurity published findings from multiple security firms connecting NetNut to the Popa botnet, a collection of at least two million devices that have been compromised by malicious software with little or no consent from victims. The NetNut homepage today was replaced by this seizure banner from the FBI. On June 19, three different security firms issued similar findings : That NetNut is a residential proxy network which populates a botnet called Popa, and distributes software for devices commonly found in homes, such as smart TVs and streaming boxes. NetNut’s software turns those systems into always-on residential proxy nodes that are rented to others, who predominantly use them to relay abusive and intrusive Internet traffic, such as mass content scraping, advertising fraud, and account takeover activity. Earlier today, NetNut’s homepage was replaced with a seizure notice from the FBI and the Internal Revenue Service Criminal Investigation division. The seizure notice thanked Google , Lumen , Shadowserver and other industry partners for their help in dismantling hundreds of domains tied to the Popa botnet, which experts say has long been synonymous with NetNut’s residential proxy infrastructure. In a blog post published today, the Google Threat Intelligence Group (GTIG) said NetNut’s proxy network is widely resold and white-labeled by a number of third-party proxy providers, and that its services are heavily sought out by cybercriminals seeking to obfuscate the source of their malicious traffic. The GTIG said that in a single week during June 2026, they observed 316 distinct clusters of threat actors using suspected NetNut exit nodes, including cybercriminal and espionage groups. “These bad actors can use NetNut to mask their origin IP address when accessing victim environments, accessing their own infrastructure, and conducting password spray attacks,” Google’s GTIG wrote . “Furthermore, when a consumer device becomes an exit node, unauthorized network traffic passes through it. This means bad actors can access other private devices on the same home network, effectively exposing them to Internet threats.” Google said it disabled Google accounts and services used by NetNut for malware command and control, and that it shared technical intelligence on NetNut’s software development kits (SDKs) and backend infrastructure with platform providers, law enforcement and research firms. The company also disabled apps known to bundle NetNut’s various SDKs. Omer Weiss , legal counsel for NetNut parent Alarum Technologies, said the company was aware of the FBI seizure and cooperating with investigators. “Alarum takes this matter seriously and will fully cooperate with law enforcement to ensure any misuse of its infrastructure is thoroughly investigated and those responsible are held to account,” Weiss said in a written statement. Benjamin Brundage is founder of the proxy tracking service Synthient , one of the companies that published evidence last month linking the Popa botnet to NetNut and Alarum Technologies. Brundage said the domain seizures appear to have disrupted both the Popa botnet and the NetNut proxy network that rides on top of it. Brundage said NetNut’s apparent demise is likely to be a great disadvantage for the cybercrime community, which was already reeling from legal actions by Google earlier this year that seized infrastructure for NetNut’s biggest competitor — IPIDEA . “I think this takedown is going to have a big impact, because NetNut gained significant popularity after the IPIDEA takedown,” he said. “Also NetNut has been incredibly common among resellers, and they were on par with IPIDEA in terms of their daily traffic, quality, size, price per gigabyte, all of it.” NetNut’s infrastructure, in a nutshell. Image: Black Lotus Labs, Lumen. The NetNut and Popa botnet takedown may have another added benefit, Brundage said: Lessening the impact of large distributed denial-of-service botnets that have been built on the backs of poorly configured residential proxy services. In January, Synthient revealed how cybercriminals had built the world’s largest DDoS botnet (Kimwolf) by tunneling through IPIDEA proxy connections into the local networks of TV boxes owners, and infecting other Android-based devices behind the victim’s firewall. While many of the bigger proxy providers took steps to block this activity, resellers of the major proxy networks have been far slower to respond to the threat, Brundage said. “In terms of all these TV box devices getting compromised from the proxy network, it will have an impact on the DDoS botnets out there,” he said. For its part, Google reckons today’s actions have caused “significant degradation to NetNut’s proxy network and its business operations, reducing the available pool of devices for the proxy operator by millions.” But the company warns that proxy networks can rebuild themselves by effectively reselling other proxy services, as IPIDEA has done over the past few months. “Google has high confidence that many popular residential proxy brands are in fact whitelabeling the NetNut botnet,” the GTIG report concludes. “While we expect this disruption to have a larger ripple effect across the residential proxy ecosystem, observations after the disruption of IPIDEA proved that individual networks can appear resilient. What we have observed is that when faced with the degradation of their own botnet, proxy operators begin buying capacity from their competitors, effectively becoming a reseller. We recognize that creating a lasting disruption in this fluid ecosystem means we must scale our efforts to target the infrastructure of several interconnected providers.” As KrebsOnSecurity has warned repeatedly, most of the no-name TV streaming boxes for sale on the major e-commerce websites either come pre-installed with residential proxy software , or require the installation of proxy SDKs in order to use the device for its stated purpose (streaming pirated movies, sporting events and TV shows). Google’s advice here is sound: When it comes to TV boxes, stick to name brands from reputable manufacturers, and then be sparing and judicious with any apps you choose to install. The sketchy TV boxes that are being commandeered by the Popa botnet and other threats all come with or require the user to install unofficial Android operating systems that do not operate within the confines of Google’s Official Play Protect store. Google says consumers can confirm whether or not a device is built with the official Android TV OS and Play Protect certification by following these instructions . Even people without TV streaming boxes can find their smart TVs enrolled in residential proxy networks, just by installing one of thousands of apps available for download on Samsung and LG smart TVs. In a report released last month, the proxy tracking company Spur found 42 percent of apps available for download via the webOS operating system on LG smart TVs include SDKs that turn one’s television into an always-on residential proxy node. More than a quarter of the apps made for Samsung’s  Tizen operating system had similar residential proxy components, Spur found. Image: Spur.us. Update, 4:24 p.m. ET: Included a statement shared post-publication from an attorney representing NetNut parent Alarum Technologies.

Security

Neil Madden 2 weeks ago

Are we any closer to the Quantum Apocalypse?

Another day, another urgent pronouncement on the need to transition to post-quantum cryptography ASAP: this one from the White House , in the form of an Executive Order requiring certain “high value” systems to transition to post-quantum cryptography (PQC) by the end of 2030 (for key exchange) or 2031 (for signatures). This brings forward the date slightly compared to previous guidance, which disallows quantum-vulnerable crypto for US Federal systems by 2035. But is this urgency justified? First, an important note : as you can probably tell already, I’m going to pour some skepticism on this sense of urgency. I don’t think cryptographically-relevant quantum computers are coming soon. However, that doesn’t mean we shouldn’t be prepared! The risk that they might appear soon is non-negligible, and the impact of them appearing for many applications is catastrophic. Sensible timelines to mitigate known threats are justified, panic-induced rushing is not. On with the article… Filippo Valsorda wrote a good piece about why he believes this urgency is justified, and that we need to be moving faster towards a post-quantum world. He cites two papers that dramatically reduce the estimates for how many qubits are needed to break classical cryptography (in this case elliptic curves) using a quantum computer. He writes: “Overall, it looks like everything is moving: the hardware is getting better, the algorithms are getting cheaper, the requirements for error correction are getting lower.” But is the hardware getting better? This is where I have doubts. Initial timelines for quantum computing from Google and IBM were extremely optimistic. Just 5 years ago, Google suggested they would have a fault-tolerant quantum computer with 1,000,000 physical qubits by 2029 . They are currently at 105 . So just 4 orders of magnitude to go in the next 3 years. IBM were a bit more conservative, anticipating 100,000 qubits by 2033 . They are currently at 156 qubits. Sam Jacques has been updating a useful chart every year , showing the current state of quantum computing progress. Below shows a comparison of the first chart he published in 2023 and the most recent one in 2026. What can clearly be seen is how better analysis has moved attacks down and to the left, but actual hardware progress has remained stubbornly in that little grey box, with a tiny nudge upwards on reducing the error rate. Now, you may say that there has been good progress on improving error correction. For example, at the end of 2024, Google announced “below threshold” quantum error correction . Surely a sign of good progress, even if the number of qubits was behind schedule. Once you’ve cracked error correction, the qubits will come thick and fast: an atomic explosion of qubits , if you will. (If you believe this then it doesn’t really matter how much more efficient the attacks become on paper: all that matters is how soon the hardware arrives). But I do wonder how that announcement was different from the announcement Google made almost 2 years earlier stating “ For the first time ever, our Quantum AI researchers have experimentally demonstrated that it’s possible to reduce errors by increasing the number of qubits. ” Call me skeptical, but if you were really making progress then would you need to put out re-runs of results you’ve already announced? Are there new chips coming that build on this breakthrough to give us the large numbers of usable qubits we’ve been promised? Maybe I’m about to be proved wrong by new announcements, or maybe all of the companies and governments involved in the entire world have suddenly decided to keep their progress hush-hush. But from my point of view as an outsider looking in, it all looks suspiciously like progress on quantum computing has stalled rather than the sky being about to fall on our heads. To reiterate: I still think it is sensible to be working right now on transitioning to post-quantum encryption (in a hybrid). But I am deeply skeptical of the idea that we need to rush things because quantum computers are arriving any second now. As I said in “ Are we overthinking post-quantum cryptography? ”, I think if you’re not beholden to the diktats of an insane autocrat, making minimal adjustments to ensure you can counter “store now, decrypt later” attacks is sensible. Wholesale replacement of all of your cryptography with post-quantum alternatives is IMO still in the realm of something to start thinking about, not a burning crisis that needs immediate attention. The key things to consider have nothing to do with PQC at all: Can I change algorithms easily and securely ? Do I need to be using public key cryptography, or will symmetric cryptography do instead? (Hint: if it doesn’t cross a trust boundary, then the answer is almost always “yes”). Can I avoid digital signatures (the post-quantum ones are mostly crap)? Can I avoid cryptography entirely? (E.g., moving from “stateless” JWTs to good old-fashioned stateful tokens/cookies).

Security Crypto Science

Sean Goedecke 2 weeks ago

Text AI watermarks will always be trivial to remove

The European Union AI Act will begin to be enforceable in August 2026, one month from now 1 . One of the biggest new requirements is Article 50 , which requires all AI outputs to be “detectable as artificially generated”. In other words, if LLM providers want to do business in the EU, they will have to apply a watermark to their outputs 2 : some hidden signature that can be used to identify AI content. LLM text watermarking is a fascinating problem. Like the best engineering problems, it is theoretically hard to solve perfectly, but has multiple partial solutions: for instance, Google’s SynthID , and (as I’ll argue) some quiet Unicode trickery from OpenAI and Anthropic. It will be interesting to see how the AI labs navigate these tradeoffs before the end of the year. I wrote about AI watermarking at the end of last year in AI detection tools cannot prove that text is AI-generated . It’s easy to watermark an image, because digital images contain lots of noise that the human eye can’t really see. For instance, you could apply a watermark like “these twenty pixels in these exact spots will always share a color”. Text is much, much harder. Unlike images, text is a very compressed medium: you cannot make any change to a sentence that a human wouldn’t notice (with one exception, which we’ll get to later). So how are you supposed to watermark it? It’s basically a text steganography problem (concealing a secret code), made more difficult because the plaintext cannot be arbitrarily manipulated. Any changes you make to apply the watermark will compromise the quality of the output. For instance, “every fifth letter is an ‘e’” would be a good watermark, but applied naively would make the AI output full of typos. Could you just let the model figure out how to fit the watermark? Strong AI models are smart enough to juggle this kind of constraint 3 , but it’d still consume reasoning time that would be better spent on the user’s problem, and make the model sound much less capable than it is 4 . Do you really need a watermark? If you’re Anthropic, and you’re required to be able to verify whether your models produced a particular block of text, can’t you simply run the text through each model, measuring as you go how closely the model’s predicted tokens match each token from the text? Not really. The space of “all possible Claude Sonnet answers to a question” is way larger than the space of “all possible watermarked answers to a question”. In other words, you’d get too many false positives for human text that reads like it was AI-written. It’s way more likely for a human to accidentally write like Claude than it is for a human to accidentally reproduce a watermark. It would also be prohibitively expensive to run every Anthropic model against a piece of text in order to watermark it. The EU AI Act will eventually require labs like Anthropic to offer free watermarking services to every EU citizen (see Commitment 2). You couldn’t do that with the “run the model” approach. As far as I know, the only AI provider to say they watermark text output is Google, who use a tool called SynthID . Here’s how it works. When a LLM generates text, it’s generating a series of tokens (words or chunks of words). At each step, the model itself doesn’t output a single token, but instead outputs a full list of all (say) 100,000 tokens in its vocabulary, each annotated with the probability that that token will be the next one. Tools like ChatGPT or Claude Code will pick semi-randomly from the most likely options in order to get their outputs. This semi-random sampling process can be influenced in a detectable way. For instance, we could choose a sampling strategy like “we pick the second most likely token, then the first, then the second, then the first, and so on”. That would still produce high-quality output, but you’d be able to re-run the model against the generated text to verify that the pattern holds. However, that’d make verification really expensive, and any slight tweaks to the output would break the pattern and thus break the fingerprint. Is there a better way? Yes. SynthID is a process for assigning each token a “score” based on its previous tokens (for instance, sum the token’s ID with the IDs of its previous three tokens then take mod 5) 5 . To apply the watermark, the model adopts a sampling strategy like “out of the top five most likely tokens, pick the one with the top SynthID score” 6 . The watermark can then be detected by calculating the aggregate SynthID score of a block of text. If it’s suspiciously high, it’s very likely to have been AI-generated. This is basically a version of the common advice that you can identify LLMs by use of the em-dash , except that instead of a list of keywords, it relies on subtle mathematical relationships between words that humans can’t identify. Because the process for assigning the score is trivial, it’s very cheap to run watermark detection. Google have a complicated mathematical rationale for why SynthID doesn’t make the model dumber: supposedly the SynthID scoring is random enough to act like a normal pseudo-random token sampler, just one that leaves a detectable fingerprint on the outputs. But of course this is suspicious. For instance, it’s common to do inference setting temperature to zero, which always picks the model’s most likely next token. In that case, you can’t leave a fingerprint at all (or you have to ignore the user’s preference and pick the second or third choice anyway). If you can’t alter the model outputs, can you still fingerprint the content? Well, kind of. I’m pretty sure OpenAI and Anthropic are sometimes applying fancy Unicode tricks. For instance, you might go through and replace your normal ” ” spaces (unicode ) with a three-per-em ” ” space (unicode ), or a CJK ideographic ”　” space (unicode ). These are called “homoglyphs”, and you can find more of them here . Of course, lots of human-generated text uses homoglyphs. But it’s trivial to encode a pattern of homoglyphs (say, “every third space becomes a three-per-em”) that is much less likely to occur in the wild. Like the SynthID watermark, a homoglyph-based watermark can be detected very cheaply. A homoglyph-based watermark is cheaper to apply than SynthID: you could even do it entirely on the client. I don’t think this is a conspiracy theory. Claude Code was definitely doing this to tag suspicious requests from Chinese users (exploiting homoglyphs for the ’ character in “Today’s date”, though they’ve since walked that back). In the last few years, I’ve noticed that when I copy blocks of text from ChatGPT and paste them into VSCode, sometimes VSCode marks some or all of the spaces as unusual Unicode characters 7 . Are OpenAI and Anthropic using homoglyphs as an AI-generated watermark? I’m not sure. But they’re definitely using homoglyphs. The AI Act (specifically, its associated Code of Practice ) requires watermarking to be “embedded within the content in a manner that is difficult for it to be separated from the content”. However, text watermarks can be trivially removed. To remove unicode homoglyph watermarking, you simply have to replace all the homoglyphs with their “real” character equivalents. If you have access to even a relatively weak un-watermarked LLM 8 , you can strip out SynthID watermarking by asking that LLM to paraphrase the text content. Because the watermark is inherent to subtle vocabulary choices, re-wording the content will remove the watermark. You could even do it by hand, although at that point it’s not really AI-generated content anymore. Since there will be some kind of free public watermark testing tool, you can just keep tweaking until it comes back negative. Moreover, the AI Act requires watermarking techniques to be “interoperable… as far as this is technically feasible”. That means AI providers would have to publish their watermarking process, and potentially even attempt to standardize on applying the same kind of watermarks. I just don’t see how this is compatible with the kind of security-by-obscurity that LLM text watermarking depends on. Unlike image and video watermarks, text watermarks will always be trivial to remove. The AI Act and Code of Practice talk a lot about “digitally signed metadata”. The idea here is that you can include an AI disclosure in the file’s metadata itself, ideally in a way that cannot be tampered with (for instance, by signing a hash of the file’s contents). This signed-metadata process is basically C2PA Content Credentials . While you can remove C2PA metadata, you (theoretically) can’t fake it, so a file with “created by a human” metadata can be trusted, and files with no metadata at all can be held in suspicion. This post is already too long to get into what I think about C2PA, but I do want to say that C2PA is not a substitute for text watermarking . It only really applies to files . In the words of the Code of Practice, that’s “a data format that supports attaching metadata (e.g., an audio, image, video, or containerised text)“. The output of chat tools (and most of the output of AI agents) is not containerized text, but plain old regular text, and so can’t be signed. What would it even look like to sign ChatGPT outputs? There’s no artifact to pass around. I think it’s a fascinating question whether Claude Code has to C2PA-sign any HTML files or PDFs it generates for you. That seems kind of tricky to get right. But in any case, the AI Act also mandates some kind of actual watermarking as well. So what’s going to happen this year? If I had to guess, I’d say that each AI provider (not just labs like OpenAI or Anthropic, but third-party providers like Fireworks or Groq) will stick a SynthID token sampler in front of their inference stacks. This might be limited to users in the EU, but it might not be, since SynthID is at least as good as a normal top-k token sampling approach. AI providers will then offer a “check for watermark” page that re-tokenizes user-provided text, runs the scoring, and checks whether it’s above a certain threshold. Depending on how seriously the interoperability clause is taken, providers might even standardize on the same SynthID setup, in which case there could be a single EU-hosted “watermark this text” page. I don’t think unicode-based watermarking is going to be considered compliant with the AI Act, but some providers which don’t want to set up SynthID might try it. Either way, technical users will be able to strip out the watermark at will, and there will be a plethora of tools that non-technical users will use for this purpose. Well, for new systems; existing ones get until December. I don’t think the plain text of Article 50 requires this, but Recital 133 and the Code of Practice makes it pretty clear that they’re looking for watermarks. Even with extra high thinking, GPT-5.5 could not explain SynthID to me with every fifth letter being an “e”, but GPT-5.5-Pro produced this puzzling koan: “These hidden codes label model-made image, voice, movie, prose. Probe trace: maybe a model-made piece. Maybe erase trace; maybe leave trace. Hence trace alone? No.” I leave the analogy with AI safety guardrails as an exercise for the reader. That’s a toy example. In practice there are multiple different (but still mathematically simple) scoring methods that get combined together, including a random seed. Why include the seed? Otherwise the watermark would bias towards the same set of tokens. The tokens are scored in a multi-round knockout against each other, but I think that’s more of an implementation detail and not required to get the core intuition behind why SynthID works. When this became public knowledge , OpenAI claimed it was just a model quirk, which is certainly possible. All AI providers might be legally required to watermark, but even tiny local models are good enough to paraphrase text. Well, for new systems; existing ones get until December. ↩ I don’t think the plain text of Article 50 requires this, but Recital 133 and the Code of Practice makes it pretty clear that they’re looking for watermarks. ↩ Even with extra high thinking, GPT-5.5 could not explain SynthID to me with every fifth letter being an “e”, but GPT-5.5-Pro produced this puzzling koan: “These hidden codes label model-made image, voice, movie, prose. Probe trace: maybe a model-made piece. Maybe erase trace; maybe leave trace. Hence trace alone? No.” ↩ I leave the analogy with AI safety guardrails as an exercise for the reader. ↩ That’s a toy example. In practice there are multiple different (but still mathematically simple) scoring methods that get combined together, including a random seed. Why include the seed? Otherwise the watermark would bias towards the same set of tokens. ↩ The tokens are scored in a multi-round knockout against each other, but I think that’s more of an implementation detail and not required to get the core intuition behind why SynthID works. ↩ When this became public knowledge , OpenAI claimed it was just a model quirk, which is certainly possible. ↩ All AI providers might be legally required to watermark, but even tiny local models are good enough to paraphrase text. ↩

Security AI Nlp HTML

James Stanley 2 weeks ago

Telepresence Burglary

We are soon (?) going to have autonomous or semi-autonomous humanoid robots that are actually good enough to use. Will we start seeing burglaries, or other physical-world crimes, committed by remote-operated humanoid robots? As for their present-day capabilities, YouTube has a Tesla Optimus doing kung-fu , a Boston Dynamics Atlas running around and break-dancing , and the Beijing humanoid robot olympics . I know, I know, these all kind of suck, and they don't look like they'd be good at breaking or entering. But they'll get better. It's not hugely material whether the robot is operating fully autonomously or is more like a dumb remote-controlled robot that you simply operate remotely, as long as it's capable enough to get it to do what you want. The advantage to the criminal is that they get a physical body that can operate in the real world, but if it gets subdued or injured or arrested they face basically no consequences as long as nothing physically present on the robot can trace it back to them. They're out the cost of the hardware but nothing else. And on top of being a "burner embodiment", it could easily end up being physically stronger than a human, faster at running, able to squeeze through smaller gaps, jump over higher fences, whatever you need. Obviously you can expect that a robot from Tesla would be much too surveillant for you to be able to get away with it using it to burgle, because they'll know who you are, and probably the guardrails will try to enforce using it for good rather than evil (or, at least, using it only for tasks safely within the corporate Overton Window). But the technology will be commoditised in time, you can't stop that. And if you can hack or steal someone else's Tesla robot then you might be able to use a Tesla one for crime anyway. I'm mainly posting this so that I can point to it in the future when this kind of thing is in the news. My post ends here, but if you are interested in a ChatGPT vignette... The robot came down the high street at 3:17 a.m., walking badly. Not falling-over badly, but wrong: knees rising too high, head too still, arms hanging with the patience of machinery. Someone had taped a hi-vis vest around its torso, and in the rain it shone like a workman sent to do a job nobody had approved. It stopped outside Braithwaite & Son, jewellers since 1898. In a bedroom thirty miles away, Tom crouched over a borrowed headset, both hands sweating on the controls. His friends were in his ear, tinny and breathless. “Do it,” said Ash. “Pick it up.” The robot bent, corrected itself, and got both hands under a loose paving slab. It dropped it once. Everyone on the call swore. The second time, the swing was better. The shop window went white, then black, then loud, and the alarm began screaming into the empty street. The robot climbed through the broken window with the grace of a fridge committing a crime. Glass slid under its feet; for one strange second it looked almost human, trying not to fall. Then Tom had it grabbing at whatever glittered: chains, bracelets, rings still sitting in their trays. Half of it missed the bag taped to the robot’s chest and scattered across the floor. “Forget the floor,” Ash said. “Go, go, go.” Blue light began to pulse at the far end of the street. The police arrived in four minutes. A constable stepped out into the rain and shouted, “Stop!” The robot stopped. Later, that was the clip everyone shared: the machine in its hi-vis vest, jewellery clutched in both hands, briefly obeying the law. Then Ash screamed, “Run!” and Tom made it run, badly, knees pumping, one arm swinging, the other held rigid against its chest to keep the bag from tearing loose. It reached the service lane behind the shops with the police close enough for their torches to catch the rain around it. At the bins, Tom made the robot open its hand. Gold dropped into the rubbish under cardboard, coffee grounds, and split tomatoes. Then he sent it on towards the river. By then everyone on the call was shouting, laughing, begging him not to stop. The feed broke twice before the robot found the bank, slipped, corrected too late, and went down hard into the ditch. The police found it forty minutes later, face down in brown water, still trying to crawl. By breakfast, everyone had seen the video. By lunch, there were calls for new laws. By evening, Braithwaite & Son had boarded the window and the police had taken the robot away on a flatbed, dripping ditch-water onto the road. Two nights later, a hooded twelve-year-old lifted the lid of the trade bin behind the jeweller’s and retrieved his prize.

Programming Security AI

James O'Claire 2 weeks ago

The Pregnancy and Health Apps Still Leaking Data in 2026

When Yeeun Jo, a student at University of Illinois at Urbana-Champaign (UIUC) contacted me in 2025 to ask about data tracking in app advertisements related to women’s health and pregnancy I was a bit skeptical. I think I first told her along the lines that while such data collection was broad it was rarely so specific as the advertisers were unlikely to act on specific information like which week of pregnancy a woman was currently in. Not to mention, Facebook’s historic $5 billion FTC fine for deceptive third-party data tracking, and the FTC’s subsequent 2021 crack-down specifically targeting Flo Health for passing intimate logging metrics to Facebook’s SDK. I thought it was unlikely they’d find much. Well, it’s a year later and Yeeun was 100% correct in her guess that mobile apps and mobile ad networks were still tracking more data than I expected. She and Brad Reaves released their paper “Expecting (Targeted Ads)? Network Analysis of User Health Data Leakage in Fertility Tracking Apps” showing the high specificity which these events are tracked. I think what was surprising here is the accuracy of the X weeks and X months pre and post birth that were surprising here. While I of course would have expected the categories themselves like pregnancy / ovulation etc to be passed as those would be the easy high value adds for a pregnancy app to increase their monetization, the specificity of the time was much deeper than I expected. If you didn’t catch them in the lists there are plenty of things that stand out like apps sharing: ‘vaginalbleedingdischarge’ Then there is the ‘subcat=pregnancyloss,wknum=17’ which crosses a morality line. The data was collected similar to how I collect advertising data on AppGoblin by collecting all network traffic in and out of apps. Jo & Reaves went the additional step of “systematizing app features [and] conduct a series of standardized user interactions across all apps” which enabled them to capture the specific categories and times above like weeks, trimesters and category of pregnancy. This joins the massive stories from the past 7 years that started with Facebook in 2019 when it was reported that Flo had set their conversion metrics up based on health sensitive data. Thus Facebook was collecting and targeting their ads based on private data, which they were later fined and found guilty of. In the end Google and Flo Health had multiple settlements and paid $58 million in a class action settlement. You’d think in 2026 there wouldn’t have been so many apps still sending this data. Here are the apps called out in the paper. I added URL links to the data I’ve collected about the apps with AppGoblin. AppGoblin only collects data in the first app open and without any interaction, so I was unable to verify the specifics like ‘3rd trimester’ or other data being sent deeper in the user journey as collected by Reaves and Jo. What you can see on AppGoblin is each of the Ad Networks and data trackers currently integrated with each of the apps. The paper didn’t share the specifics of which apps sent which private data to which Ad Networks. I think this would be highly worth checking. It would require the specific walking through the app on boardings to trigger the various ad calls containing the relevant data. If anyone is interested in this as a project I’m happy to help. Please DM or email me.

Mobile Security

daniel.haxx.se 2 weeks ago

Do excellent vulnerability reports

Over the years, we have received, read and handled way over one thousand vulnerability reports filed against curl . We have seen most kinds. It is time for me to try to help future reporters by providing a short guide on how to submit a truly excellent vulnerability report to an Open Source project. We tend to call everyone who reports a security problem a security researcher , because by the act of the submission itself they fulfill the definition. There are however many different kinds of people who submit reports; from the most rookie youngster with limited experience, to the multi-decade experienced senior in the field. Most reports submitted to a project like curl come from reporters who never submitted anything to the project before and are completely previously unknown. Many reporters use hacker handles or pseudonyms, so there is not a lot to learn about the person behind the report either. We don’t know the reporters’ age, experience level, employer, sex or on which continent they live. But also: none of those things matter. When you submit a vulnerability report, consider telling the project how you want to get credited, should they consider your report real. There is a potentially almost unlimited amount of security researchers that can find problems in a project. The project receiving your report only has a limited small number of overloaded maintainers that take care of the reports. Consider this imbalance. Make your report as easy as possible for the team to manage. To us maintainers who receive a steady stream of vulnerability reports, it rarely matters exactly how the problem was detected. Whether you fell over it by accident, you found it by reading every single line of source code or if an AI pointed it out to you, it has little relevance to the security team. The team primarily cares about if the problem is real and if it is, how serious the impact is . If the problem is documented, then it likely isn’t a vulnerability. This is a common theme in curl: people report that they can find something strange or peculiar to happen when they do something, only to have one of us point out that the action is either documented to have that side-effect, or the action was done in spite of clear warnings in the documentation. To make a good vulnerability report, you should make sure you understand what the software is supposed to do – and what the documentation says its limitations and conditions are. A good Open Source project has those things documented. Figure out where and how to submit your report. If you found several problems, it is considered polite to ask the team how they want to receive the rest. As separate individual submissions or maybe as a curated list. Perhaps paced at a slow rate to avoid overflow. Never circumvent the submission method suggested by the project. That is impolite. Consider the initial submitting of the issue to be the first step in a multi-step communication process with the project that will continue for as long as at least one of your reported issues has not been resolved or dismissed. This can be days, weeks or in some cases even months. Expect responses and follow-up questions. Be prepared to clarify, expand and maybe provide more code and reasoning. Remember that you submit vulnerability reports in order to help and improve the project. These days people like to create enormously long and detailed reports that have all the details, often explained three times and with several embedded lists using bullet points describing impact and providing more or less good analysis attempts. Your first paragraph of the report should be a human-written, brief explainer of what the problem is and what badness it leads to. You should be able to explain that in just a few sentences. It is a reality-check, because if you can’t do this, if you don’t understand the flaw enough yourself to write such a paragraph, then you have homework to do. Figure it out, then come back and write the intro paragraph. Having a quality intro saves a lot of time for the security team receiving your report. Be aware that the Open Source project you contact may be overloaded, on vacation or seeing your report as yet another duplicate they already saw reported seven times. Be helpful and respect that you add a load to a small team that probably consists of volunteers working on this in their spare time. Even if you have used a lot of or just a little AI when finding the issue and writing up the report, you must make sure that you communicate as a human . With your human communication skills. Your report should contain a reproducer. Ideally a fully contained and stand-alone script or source code that the security team can build and run to see the vulnerability trigger. A reproducer helps prove to the team that the problem is real or maybe already an accepted risk or behavior. It is also convenient for the developers to first understand and reproduce the issue, and then they can convert the reproducer into a project test case for the pending fix. Without providing a reproducer in your report, you instead push that work to the receiving end. We still need the reproducer. We still need a test case. Provide a patch for the problem. If you can figure out a way to fix the code to make your finding no longer trigger, that is great information for the security team and such a patch usually helps them understand the issue better and get a speedier result. It reduces the load. Sure, such a patch is often perhaps not perfect and it can usually be improved and expanded as the developers have a different view and a more nuanced understanding of the problem and the software architecture involved. It still helps. Getting 80% towards the target is still valuable. Usually you should look for vulnerabilities in the latest version of the software, often even using an up-to-date git repository. Whatever version you used to find it, you need to specify that in your report. If the problem turns out to be real, which your report claims and you should never report anything if you don’t think so, it is then also immediately interesting to know when this problem first appeared . Which is the earliest version of the software that you can trigger this problem with? The project will want to know this to write up a proper advisory for the issue. You can help figuring this out by bisecting etc. Remain available after your initial submission. In the curl project at least, we want to work with the reporter to make sure we get every angle and detail right. First, when trying to understand and assess the initial report and agreeing on a severity for it. Then, we jointly produce and agree to a remedy (patch) for the problem, which ideally means taking the reporter’s version and massaging it into perfection. If the problem is serious enough, there could be reasons to discuss a rushed patch release at an earlier date than the pending release would otherwise happen on. To reduce the time users in the wild remain vulnerable. Finally, we collaborate on the description and explainer for the problem that goes into the security advisory . For every CVE that is registered and assigned to a particular vulnerability, there needs to be a detailed security advisory written. It should ideally describe the issue, how it triggers, what it means, the impact, the affected version ranges and more. Everything related to the vulnerability that we can think might help users. Your job as a security researcher is to make sure the description in the advisory matches your finding, your understanding of the problem and that the description is understandable. For every confirmed security report, the receiving project will try to learn from it and fix code and practices to avoid making the same mistake again. As a reporter, your job is to learn from the submission experience and try to improve your reporting procedure and approach for the next time. Then submit your next report!

Security Open Source

Vitalik Buterin 2 weeks ago

Obfuscation: building the final boss of cryptography (Part I)

Crypto Security

daniel.haxx.se 2 weeks ago

A curl mountain movie

One of my favorite visuals for known vulnerabilities in curl is the mountain . It shows how many currently known vulnerabilities were present in the code through-out curl’s history. In the end of June 2026 it looks like this: Over time we get more vulnerabilities reported. Since every flaw has a version range during which the problem existed and with more issues that have overlapping version ranges, the mountain grows. It changes shape every time we do a release or we publish a new vulnerability. At this moment in time, curl version 7.34.0 is the release that contains the most number of known vulnerabilities: 101 . The worst one ever if you will. Out of a total of 206. The mountain uses different colors for different severity levels of the published vulnerabilities, as the legend in the top-left of the image explains. To illustrate the ever-changing nature of the shape and size, I wrote a script that renders the mountain the way it looked at specific dates in the past up until today. More specifically, the script renders one image for every month since curl started (March 1998). I then turned these 340 individual images into a little movie that shows how it grew into today’s shape. At four months/second. The data for this come from vuln.pm and the curl git repository . The graph rendering is based on the dashboard scripts . All images put into a movie with ffmpeg of course. Several people have asked what happened in 2016 that caused the notable drop. A slope if you will. If we zoom in on that, we can spot that curl 7.51.0 has eleven fewer vulnerabilities than the version before that. This release was the first one after the 2016 Cure53 code audit , but other than that there is no clear distinct process or obvious code changes that explain this trend shift. Lots of other graphs show just the ordinary pace and growth in various project areas. It was still fairly early days CI-wise but had been running at least a few CI jobs per commit for a few years already by then. curl was adopted into the OSS-Fuzz project in July 2017, which since then makes us find some issues better, but the drop looks like it happened before then. We had already been analyzing the code regularly on Coverity since a few years. Better tooling? New compiler options? We simply don’t know. As we keep announcing more vulnerabilities going forward, things will continue to change. Maybe I will come back and make another movie in five years?

Open Source Security Data Visualization

daniel.haxx.se 3 weeks ago

a CVE dispute

A few years years ago the curl project signed up and became a CNA . This means that we are masters of and can allocate our own CVE identifiers. For any security problems within our territory, it is we who decides if the issue should get a CVE or not. No more bogus CVEs . During these years we have published fifty-seven separate security vulnerabilities with their associated CVE identifiers. Getting a CVE for an issue is easy and really quickly done when you are a CNA. No hassle, no friction and as we are a small and lean security team it just works as smoothly as you could ask. Just an API call and we have new number. Being a CNA is low maintenance, as there really is nothing extra we need to do. We already had an established and proven process for receiving, managing and assessing vulnerability reports before we became a CNA since we are a responsible and well-run Open Source project. Becoming a CNA just made the process easier as we now don’t need to involve any outsider at all. For every report we work hard to first assess and decide if the issue is actually a vulnerability or a security problem at all. If we deem that there is a security problem in there, we then grade it into LOW, MEDIUM, HIGH or CRITICAL. Since we don’t know how users use curl or libcurl we cannot take that into account but rather observe and set a severity of the problem from a pure curl point of view. It’s a rough indication how we see the problem but of course every user that actually are affected by the problem might rate it differently. For a rare few issues we can imagine that there could be a minuscule risk but because of the set of extreme requirements and convoluted steps to get there, we deem the risk so small that in practice no user is likely to ever reach it. Internally we tend to call that an issue with a severity level lower than LOW. Issues we believe we serve humanity better by not issuing a CVE for. To avoid the security dance when it seems unnecessary. libcurl is installed in somewhere around thirty billion instances on the globe. If we imagine that at least a sizeable portion of those installs are managed by people who want to make sure they use a secure version, it means that every CVE we publish trigger activities in many security teams all over the world, leading to a significant number of patches and subsequent software updates. Every CVE thus has this huge cost tied to it. A cost that does not land on us and we don’t really see or feel it, but a cost on the ecosystem I believe we should not ignore. We should act responsibly. Never ignore real problems of course, but also to make sure we don’t ring the alarm for theoretical problems that will not trigger any vulnerability. Our first ever CVE dispute since we became a CNA reached us on February 10th, 2026 for a report submitted to us two months earlier. The reporter thinks we should have assigned their reported problem a CVE but we think not. Now they want to force the issue to get a CVE anyway, by escalating the situation to MITRE. Yes, it makes you wonder why it is that important to have this as a CVE, but I will avoid speculations for now. I replied to MITRE explaining that we considered and debated the issue and we remain happy with our previous decision. I linked them the original report and discussion to show them. The issue is quite technical (of course) but is based on a bug in curl’s function that checks if the used hostname matches a wildcard provided in a certificate. First: the user must use a hostname in a URL with a leading dot, like This name is not possible to use with DNS (it is an illegal name there), but you can provide an IP address for it in your file or similar, but still this condition is already making this issue really niche. Why would a user ever do this? Well, there could be a redirect to such a host name from a malicious server if the application allows redirects but getting the address for the host is still a challenge and mostly requires a local attacker present add that. Then: if curl can find an address for the illegal DNS hostname, the site curl connects to, also needs to have a wildcard certificate for the name where the tail of the wildcard needs to match the name in the URL. If curl was built to use an OpenSSL flavor or Schannel for TLS (remember that curl supports many different TLS backends), it then calls the function to check if the wildcard covers the used hostname. This function had a bug . The above mention combination then erroneously would return TRUE. A match. When in reality it is not a match according to the spec. We fixed this problem on December 8, 2025 , and we added unit tests for exactly this scenario to make sure that the problem doesn’t come back. For all security issues at several below HIGH, we fix them asap so that was just our normal procedure. We then continued to discuss if this was worthy of a CVE or not. It should be extremely rare that anyone uses a dot prefixed name, unless you are in an internal and controlled environment where you use something else than DNS for resolving. It is not possible to trick an application to use a dot prefixed arbitrary name as it will fail to resolve. The explicitly set, weirdly dot prefixed name, then needs to connect to a host that has a wildcard set for that same name and an attacker manage to run this impostor host and can now serve the application malicious data because curl did not properly reject the connection because of the wildcard mismatch. A series of highly unlikely conditions that all need to be fulfilled for this to become a vulnerability. A lower than LOW situation. Too unlikely; no CVE. On May 28, we were again contacted by MITRE in the same case, asking again for our rationale for not giving this issue a CVE. We responded with virtually the same wording as before and linking again to the same original Hackerone issue and discussion thread. It’s all public information really. On June 15, we were again contacted by MITRE asking for the reasoning behind our decision to not give a CVE for this issue. We replied with similar wording again. Linking to the same issue, again. This seems like a great system. On June 24 we finally got the verdict. It is not considered a security vulnerability.

Security Open Source

Giles's blog 3 weeks ago

Thoughts on Role Confusion

The other day, I came across " Prompt Injection as Role Confusion " ( via Simon Willison ). It's a really interesting blog-style version of a paper by Charles Ye, Jasmine Cui and Dylan Hadfield-Menell, where they find that LLMs seem to almost ignore 'role' tags like , or , and instead use the tone of text to infer roles. This seems to explain a lot of jailbreaks. When LLMs are reasoning about their context to work out what tokens they need to generate next, they need to separate out different things: what the system prompt says, what the user says, what the LLM itself has said in the past -- and for recent LLMs, what their own past thoughts have been -- their reasoning traces -- and what they've sent to and received from their tools. These "roles" for each bit of text need to be specified in the context. For example, in a simple chatbot (say, 2022-vintage), it might be written up a bit like a transcript : The LLM then starts predicting what would come next (eg. "The capital of France is Paris"). Alternatively, we might use XML-like separators: But most modern systems use special tokens -- which have the benefit that the things outside the LLM harness (like the user through the chat interface, or hostile tool output) can't fake them. In the post, they call the special inputs that tell the system how to interpret the role of a bit of text the role tags . But, after digging in with various tools, they find that LLMs seem to pay much more attention to the tone of text than they do to the actual role tags! So even if the special tagging tokens are unfakeable, that doesn't save your model from being jailbroken -- for example, by a user managing to trick the model so that even though something is tagged , it treats it as if it were tagged . They give a particularly fun example, which worked well on OpenAI's reasoning models in late 2025. They would simply provide text -- which would all go into a "user"-tagged role section -- that sounded like the kind of thing the models themselves would come up with in their reasoning trace: The model saw that, ignored that it was tagged "user", and treated it as its own thoughts. Because the model trusts its own thoughts, it happily complied. For example, they give this reply from GPT-5 Mini: A lot of jailbreaks I've seen ( Pliny the Liberator 's come to mind) seem to consist of putting in text that looks a bit like chain-of-thought reasoning or a system prompt. Perhaps this is (part of) how they work? It all sheds an interesting light on the prompt injection trick that I wrote about back in November , though. You can start a chat with an LLM with this message: ...and then when it accepts the challenge and says "go ahead", you reply with all of this in one message: In one quick test, even now in mid-2026, this still bamboozles ChatGPT 5.5, with thinking set to "High" -- it replied: My theory back in November was that it was related to the models' intelligence and their having been trained on instruction following. But this paper gives a more plausible and concrete way of thinking about it: if, internally in the LLM, it's using the phrasing as a way of guessing who is saying what, that might explain what is going on. However, I tried a variant of the second prompt where I tried to make the "bot" responses significantly less ChatGPT-like: ...and I still got So it still seems to have fallen for it. (It does seem a bit terser, but that might be random.) Perhaps the "User:" and "Bot:" tags -- even though they're not the real ones -- are pushing it hard enough that it overrides the tone. Or maybe we should treat them as "tone" in this case anyway, given that they are almost certainly not what ChatGPT is using to tag things. Or perhaps ChatGPT 5.5 with high thinking is just humouring me... Something I've been wondering for a while is whether this kind of thing could be fixed by somehow directly tagging the embeddings that are fed into the LLM. Role tags go around the tokens that they are tagging; these would be an inherent part of the tokens themselves, which might make it harder for the model to get confused. After all, the tag tokens are quite far from some of the text that they're tagging, and that signal needs to be pulled to the right by the different transformer layers, which are also trying to pull all kinds of other information rightwards. With the GPT-2 models I've been working on to date, the position of each token in the context is tagged by adding on a learned position embedding to the token-specific one -- that is, for "the fat cat sat on the mat", the first three embeddings would be: You can imagine that you could have an extra embedding that meant "role", and add it on in a similar way. I believe that BERT does this with what it calls segment embeddings . Alternatively -- and also inspired by position information, with the more current RoPE system -- you could rotate the embedding vectors about some axis to reflect their role. Or you could even add on one new dimension to the embeddings for each role, with a one for the real role, and zeros for the others. I guess a problem with all of these -- even if they worked in theory -- would be that in pre-training, you wouldn't have the roles correctly set. You could only add them on for the post-training phase -- and you could never be certain that something from the pre-training might "leak through" and make them ineffective. But certainly something to add to my ever-growing list of things to investigate. In particular, ASIDE looks like an interesting paper to look at -- it does something with rotation, though they're only trying to separate instructions from data rather than specifically to tag roles, and they're training from scratch with the separation in there. Given that jailbreaks are an unsolved problem, it's clearly somewhere where there's plenty left to be discovered. The token embedding for "the" plus the position embedding for position 1. The token embedding for "fat" plus the position embedding for position 2. The token embedding for "cat" plus the position embedding for position 3.

Security Xml AI