ENOSUCHBLOG 3 weeks ago

Dear GitHub: no YAML anchors, please

TL;DR : for a very long time, GitHub Actions lacked support for YAML anchors. This was a good thing . YAML anchors in GitHub Actions are (1) redundant with existing functionality, (2) introduce a complication to the data model that makes CI/CD human and machine comprehension harder, and (3) are not even uniquely useful because GitHub has chosen not to support the one feature (merge keys) that lacks a semantic equivalent in GitHub Actions. For these reasons, YAML anchors are a step backwards that reinforces GitHub Actions’ status as an insecure by default CI/CD platform. GitHub should immediately remove support for YAML anchors, before adoption becomes widespread. GitHub recently announced that YAML anchors are now supported in GitHub Actions. That means that users can write things like this: On face value, this seems like a reasonable feature: the job and step abstractions in GitHub Actions lend themselves to duplication, and YAML anchors are one way to reduce that duplication. Unfortunately, YAML anchors are a terrible tool for this job. Furthermore (as we’ll see) GitHub’s implementation of YAML anchors is incomplete , precluding the actual small subset of use cases where YAML anchors are uniquely useful (but still not a good idea). We’ll see why below. Pictured: the author’s understanding of the GitHub Actions product roadmap. The simplest reason why YAML anchors are a bad idea is because they’re redundant with other more explicit mechanisms for reducing duplication in GitHub Actions. GitHub’s own example above could be rewritten without YAML anchors as: This version is significantly clearer, but has slightly different semantics: all jobs inherit the workflow-level . But this, in my opinion, is a good thing : the need to template environment variables across a subset of jobs suggests an architectural error in the workflow design. In other words: if you find yourself wanting to use YAML anchors to share “global” configuration between jobs or steps, you probably actually want separate workflows, or at least separate jobs with job-level blocks. In summary: YAML anchors further muddy the abstractions of workflows, jobs, and steps, by introducing a cross-cutting form of global state that doesn’t play by the rules of the rest of the system. This, to me, suggests that the current Actions team lacks a strong set of opinions about how GitHub Actions should be used, leading to a “kitchen sink” approach that serves all users equally poorly. As noted above: YAML anchors introduce a new form of non-locality into GitHub Actions. Furthermore, this form of non-locality is fully general : any YAML node can be anchored and referenced. This is a bad idea for humans and machines alike: For humans: a new form of non-locality makes it harder to preserve local understanding of what a workflow, job, or step does: a unit of work may now depend on any other unit of work in the same file, including one hundreds or thousands of lines away. This makes it harder to reason about the behavior of one’s GitHub Actions without context switching. It would only be fair to note that GitHub Actions already has some forms of non-locality: global contexts, scoping rules for blocks, dependencies, step and job outputs, and so on. These can be difficult to debug! But what sets them apart is their lack of generality : each has precise semantics and scoping rules, meaning that a user who understands those rules can comprehend what a unit of work does without referencing the source of an environment variable, output, &c. For machines: non-locality makes it significantly harder to write tools that analyze (or transform) GitHub Actions workflows. The pain here boils down to the fact that YAML anchors diverge from the one-to-one object model 1 that GitHub Actions otherwise maps onto. With anchors, that mapping becomes one-to-many: the same element may appear once in the source, but multiple times in the loaded object representation. In effect, this breaks a critical assumption that many tools make about YAML in GitHub Actions: that an entity in the deserialized object can be mapped back to a single concrete location in the source YAML. This is needed to present reasonable source locations in error messages, but it doesn’t hold if the object model doesn’t represent anchors and references explicitly. Furthermore, this is the reality for every YAML parser in wide use: all widespread YAML parsers choose (reasonably) to copy anchored values into each location where they’re referenced, meaning that the analyzing tool cannot “see” the original element for source location purposes. I feel these pains directly: I maintain zizmor as a static analysis tool for GitHub Actions, and makes both of these assumptions. Moreover, ’s dependencies make these assumptions: (like most other YAML parsers) chooses to deserialize YAML anchors by copying the anchored value into each location where it’s referenced 2 . One of the few things that make YAML anchors uniquely useful is merge keys : a merge key allows a user to compose multiple referenced mappings together into a single mapping. An example from the YAML spec, which I think tidily demonstrates both their use case and how incredibly confusing merge keys are: I personally find this syntax incredibly hard to read, but at least it has a unique use case that could be useful in GitHub Actions: composing multiple sets of environment variables together with clear precedence rules is manifestly useful. Except: GitHub Actions doesn’t support merge keys ! They appear to be using their own internal YAML parser that already had some degree of support for anchors and references, but not for merge keys. To me, this takes the situation from a set of bad technical decisions (and lack of strong opinions around how GitHub Actions should be used) to farce : the one thing that makes YAML anchors uniquely useful in the context of GitHub Actions is the one thing that GitHub Actions doesn’t support. To summarize, I think YAML anchors in GitHub Actions are (1) redundant with existing functionality, (2) introduce a complication to the data model that makes CI/CD human and machine comprehension harder, and (3) are not even uniquely useful because GitHub has chosen not to support the one feature (merge keys) that lacks a semantic equivalent in GitHub Actions. Of these reasons, I think (2) is the most important: GitHub Actions security has been in the news   a great deal recently , with the overwhelming consensus being that it’s too easy to introduce vulnerabilities in (or expose otherwise latent vulnerabilities through ) GitHub Actions workflow. For this reason, we need GitHub Actions to be easy to analyze for humans and machine alike. In effect, this means that GitHub should be decreasing the complexity of GitHub Actions, not increasing it. YAML anchors are a step in the wrong direction for all of the reasons aforementioned. Of course, I’m not without self-interest here: I maintain a static analysis tool for GitHub Actions, and supporting YAML anchors is going to be an absolute royal pain in my ass 3 . But it’s not just me: tools like actionlint , claws , and poutine are all likely to struggle with supporting YAML anchors, as they fundamentally alter each tool’s relationship to GitHub Actions’ assumed data model. As-is, this change blows a massive hole in the larger open source ecosystem’s ability to analyze GitHub Actions for correctness and security. All told: I strongly believe that GitHub should immediately remove support for YAML anchors in GitHub Actions. The “good” news is that they can probably do so with a bare minimum of user disruption, since support has only been public for a few days and adoption is (probably) still primarily at the single-use workflow layer and not the reusable action (or workflow) layer. That object model is essentially the JSON object model, where all elements appear as literal components of their source representation and take a small subset of possible types (string, number, boolean, array, object, null).  ↩ In other words: even though YAML itself is a superset of JSON, users don’t want YAML-isms to leak through to the object model. Everybody wants the JSON object model, and that means no “anchor” or “reference” elements anywhere in a deserialized structure.  ↩ To the point where I’m not clear it’s actually worth supporting anchors to any meaningful extent, and instead immediately flagging them as an attempt at obfuscation.  ↩ For humans: a new form of non-locality makes it harder to preserve local understanding of what a workflow, job, or step does: a unit of work may now depend on any other unit of work in the same file, including one hundreds or thousands of lines away. This makes it harder to reason about the behavior of one’s GitHub Actions without context switching. It would only be fair to note that GitHub Actions already has some forms of non-locality: global contexts, scoping rules for blocks, dependencies, step and job outputs, and so on. These can be difficult to debug! But what sets them apart is their lack of generality : each has precise semantics and scoping rules, meaning that a user who understands those rules can comprehend what a unit of work does without referencing the source of an environment variable, output, &c. For machines: non-locality makes it significantly harder to write tools that analyze (or transform) GitHub Actions workflows. The pain here boils down to the fact that YAML anchors diverge from the one-to-one object model 1 that GitHub Actions otherwise maps onto. With anchors, that mapping becomes one-to-many: the same element may appear once in the source, but multiple times in the loaded object representation. In effect, this breaks a critical assumption that many tools make about YAML in GitHub Actions: that an entity in the deserialized object can be mapped back to a single concrete location in the source YAML. This is needed to present reasonable source locations in error messages, but it doesn’t hold if the object model doesn’t represent anchors and references explicitly. Furthermore, this is the reality for every YAML parser in wide use: all widespread YAML parsers choose (reasonably) to copy anchored values into each location where they’re referenced, meaning that the analyzing tool cannot “see” the original element for source location purposes. I feel these pains directly: I maintain zizmor as a static analysis tool for GitHub Actions, and makes both of these assumptions. Moreover, ’s dependencies make these assumptions: (like most other YAML parsers) chooses to deserialize YAML anchors by copying the anchored value into each location where it’s referenced 2 . That object model is essentially the JSON object model, where all elements appear as literal components of their source representation and take a small subset of possible types (string, number, boolean, array, object, null).  ↩ In other words: even though YAML itself is a superset of JSON, users don’t want YAML-isms to leak through to the object model. Everybody wants the JSON object model, and that means no “anchor” or “reference” elements anywhere in a deserialized structure.  ↩ To the point where I’m not clear it’s actually worth supporting anchors to any meaningful extent, and instead immediately flagging them as an attempt at obfuscation.  ↩

JSON Ci Cd Yaml DevOps

▲ 0

0 views

ENOSUCHBLOG 1 months ago

One year of zizmor

This is a dual purpose post: I’ve released zizmor v1.13.0 , and zizmor has turned one year old 1 ! This release isn’t the biggest one ever, but it does include some nice changes (many of which were contributed or suggested by zizmor’s increasingly large community). Highlights include: A new (pedantic-level) undocumented-permissions audit rule, which flags explicit permission grants (like ) that lack an explanatory comment. Many thanks to @johnbillion for contributing this! (Long requested) support for disabling individual audit rules entirely, via in the configuration. Support for auto-fixing many of the obfuscation audit’s findings, including a brand new constant expression evaluator for github-actions-expressions that can unroll many common cases of obfuscated GitHub Actions expressions. Many thanks to @mostafa for contributing this, along with so much else of zizmor’s auto-fix functionality over the last few releases! Support for “grouped” configurations, which resolve a long-standing architectural limitation in how zizmor loads and applies a user’s configuration. The TL;DR of this change is that versions of zizmor prior to 1.13.0 could only ever load a single configuration file per invocation, which meant that invocations like wouldn’t honor any configuration files in or . This has been changed so that each input group (i.e. each input argument to ) loads its own isolated configuration. In practice, this should have no effect on typical users, since the average user likely only runs or . But it’s something to be aware of (and will likely benefit you) if you’re a bulk user! These changes build on a lot of other recent (v1.10.0 and later) improvements, which could easily be the subject of a much longer post. But I’d like to focus the rest of this post on some numbers for and reflections on the past year of zizmor’s growth. As of today (September 12, 2025), zizmor has: 26 unique audit rules , encompassing a wide range of reactive and proactive security checks for GitHub Actions workflows and action definitions. This is up from 10 rules in the first tagged release (v0.1.0), which predates the changelog. It also undercounts the growth of zizmor’s complexity and feature set, since many of the rules have been significantly enhanced over time. Just over 3000 stars on GitHub, growing from close to zero at the start of the year. Most of that growth was in the shape of a “hockey stick” during the first few months, but growth has been steady overall: About 3.2 million downloads from PyPI alone , just barely squeaking us into the top 10,000 most-downloaded packages on PyPI 2 . Potentially more interesting is the growth in PyPI downloads: around 1 in 5 of those downloads (633K precisely) were in the last month alone . Notably, PyPI is just one of several distribution channels for zizmor: the official Docker image has another ~140K downloads, the Homebrew formula has another ~50K, conda-forge has another ~90K, and so forth. And these are only the ones I can easily track! So, so many cool downstream users: cURL, CPython, PyPI itself, Rust, Rustls, Sigstore, Anubis, and many others are all running zizmor in their CI/CD to proactively catch potential security issues. In addition to these projects (who humble me with their use), I’ve also been thrilled to see entire communities and companies adopt and/or recommend zizmor: pyOpenSci , Grafana Labs , and Wiz have all publicly recommended zizmor based on their own experience or expertise. Roughly 50 contributors , excluding myself and bots, along with several dozen “regulars” who file bug reports and feature requests. This number is easily the most satisfying of the above: it’s a far cry from the project’s start, when I wasn’t sure if anyone would ever use it, let alone contribute to it. Some thoughts and observations from the past year. People like to use zizmor, even though it’s a security tool! Crazy! Most people, including myself , hate security tools : they are, as a class, frustrating to install and configure, obtuse to run, and are arrogantly incorrect in their assumption of user tolerance for signal over noise. My own negative experiences with security tooling made me hesitant to build zizmor in the open: I was worried that (1) no one would use it, or (2) lots of people would use it in anger , and hate it (and me) for it. To my surprise, this didn’t happen! The overwhelming response to zizmor has been positive: I’ve had a lot of people thank me for building it, and specifically for making it pleasant to use. I’ve tried to reflect on what exactly I did to succeed in not eliciting a “hate” reaction to zizmor, and the following things come to mind: It’s very easy to install and run: distributing it via PyPI (even though it’s a pure Rust binary) means that it’s a single or away for most users. It’s very fast by default: offline runs take tens of milliseconds for representative inputs; online runs are slower, but still fast enough to outpace typical CI/CD setups. In other words: people very rarely wait for zizmor to complete, or in the worst case wait no longer than they’re already waiting for the rest of their CI/CD. It strikes the right balance with its persona design: the default (“ regular ”) persona prioritizes signal over noise, while letting users opt into more noisy personae (like pedantic and auditor ) as they please. One outcome from this choice which has been (pleasantly) surprising is seeing users opt into a more sensitive persona than the default, including even enforcing zizmor at the pedantic or auditor level in their CI/CD. I didn’t expect this (I expected most users to ignore non-default personae), and it suggests that zizmor’s other usability-first design decisions afford us a “pain budget” that users are willing to spend on more sensitive checks. On a very basic level, I don’t really want zizmor to exist: what I really want is a CI/CD system that’s secure by construction , and that doesn’t require static analysis to be bolted onto it for a degree of safety. By analogy: I want the Rust of CI/CD, not C or C++ with a layer of ASan slapped on top. GitHub Actions could have been that system (and arguably still could be), but there appears to be relatively little internal political 3 appetite within GitHub for making that happen 4 . In this framing, I’ve come to see zizmor as an external forcing function on GitHub: a low-friction, low-complexity tool that reveals GitHub Actions’ security flaws is also a tool that gives the hardworking engineers inside of GitHub the kind of political ammunition they need to convince their leadership to prioritize the product itself. I’m not conceited enough to think that zizmor (or I) alone am the only such external forcing function: there are many others, including entire companies seeking to capitalize on GitHub Actions’ flaws. However, I do think the last year of zizmor’s development has seen GitHub place more public emphasis on security improvements to GitHub Actions 5 , and I would like to think that zizmor has played at least a small part in that. Six months ago, I would have probably said that zizmor is mostly done : I was happy with its core design and set of audits, and I was having trouble personally imagining what else would be reasonable to add to it. But then, stuff kept happening! zizmor’s user base has continued to identify new things that zizmor should be doing, and have continued to make contributions towards those things. They’ve also identified new ways in which zizmor should operate: the auto-fix mode (v1.10.0) and LSP support (v1.11.0) are just two examples of this. This has made me less certain about what “done” will look like for zizmor: it’s clear to me that a lot of other people have (very good!) ideas about what zizmor can and should do to make the GitHub Actions ecosystem safer, and I’m looking forward to helping to realize those ideas. Some specific things I see on the horizon: More interprocedural analysis: zizmor is largely “intraprocedural” in the moment, in the sense the it analyzes individual inputs (workflows or action definitions) in isolation. This approach makes zizmor simple, but it also leaves us with an partial picture of e.g. a repository’s overall CI/CD posture. As a simple example: the unpinned-uses audit will correctly flag any unpinned action usages, but it won’t detect transitively unpinned usages that cross input boundaries. Better support for large-scale users: it’s increasingly clear to me that a significant portion (and increasing) portion of zizmor’s userbase is security teams within bigger open source projects (and companies), who want to use zizmor as part of “estate management” for dozens, hundreds, or even thousands of individual repositories. zizmor itself doesn’t struggle to operate in these settings, but large scales make it harder to triage and incrementally address zizmor’s findings. I’m not sure exactly what an improved UX for bulk triage will look like, but some kind of GitHub App integration seems worth exploring 6 . A bigger architectural reevaluation: zizmor’s current architecture is naïve in the sense that individual audits don’t share or re-use computation between each other. For example, two audits that both evaluate GitHub Actions expressions will independently re-parse those expressions rather than caching that reusing that work. This is not a performance issue at zizmor’s current audit count, but will likely eventually become one. When this happens, I’ll likely need to think about a larger architectural change that allows audits to either share computed analysis state or push more analysis state into zizmor’s input collection phase. Another (unrelated) architectural change that’ll likely eventually need to happen involves , which zizmor currently uses extensively: its deprecated status is a long-term maintenance risk. My hope is that alternatives (like saphyr , which I’ve been following) will become sufficiently mature and feature-rich in the medium-long term to enable fully replacing (and potentially even some of our use of in e.g. and .). It depends on how you count: zizmor’s first commit was roughly 13 months ago, while its first tagged release (v0.1.0) was roughly 11 months ago. So I’m splitting the two and saying it’s been one year.  ↩ Number #9,047 as of time of writing.  ↩ Read: product leadership. I have a great deal of faith in and respect for GitHub’s engineers, who seem to be uniformly upset about the slow-but-accelerating degradation of GitHub’s product quality.  ↩ A more cynical framing would be that GitHub has entered the “value strip-mining” phase of its product lifecycle, where offerings like GitHub Actions are kept alive just enough to meet contractual obligations and serve as a staging bed for the next round of “innovation” (read: AI in more places). Fixing longstanding flaws in GitHub Actions’ design and security model would not benefit this phase, so it is not prioritized.  ↩ Off the top of my head: immutable actions/releases , fixing action policies , acknowledging how dangerous is, &c.  ↩ This avenue of exploration is not without risks: my experience has been that many security tools that choose to go the “app” integration route end up as Yet Another Infernal Dashboard that security teams dread actually using.  ↩ A new (pedantic-level) undocumented-permissions audit rule, which flags explicit permission grants (like ) that lack an explanatory comment. Many thanks to @johnbillion for contributing this! (Long requested) support for disabling individual audit rules entirely, via in the configuration. Support for auto-fixing many of the obfuscation audit’s findings, including a brand new constant expression evaluator for github-actions-expressions that can unroll many common cases of obfuscated GitHub Actions expressions. Many thanks to @mostafa for contributing this, along with so much else of zizmor’s auto-fix functionality over the last few releases! Support for “grouped” configurations, which resolve a long-standing architectural limitation in how zizmor loads and applies a user’s configuration. The TL;DR of this change is that versions of zizmor prior to 1.13.0 could only ever load a single configuration file per invocation, which meant that invocations like wouldn’t honor any configuration files in or . This has been changed so that each input group (i.e. each input argument to ) loads its own isolated configuration. In practice, this should have no effect on typical users, since the average user likely only runs or . But it’s something to be aware of (and will likely benefit you) if you’re a bulk user! 26 unique audit rules , encompassing a wide range of reactive and proactive security checks for GitHub Actions workflows and action definitions. This is up from 10 rules in the first tagged release (v0.1.0), which predates the changelog. It also undercounts the growth of zizmor’s complexity and feature set, since many of the rules have been significantly enhanced over time. Just over 3000 stars on GitHub, growing from close to zero at the start of the year. Most of that growth was in the shape of a “hockey stick” during the first few months, but growth has been steady overall: About 3.2 million downloads from PyPI alone , just barely squeaking us into the top 10,000 most-downloaded packages on PyPI 2 . Potentially more interesting is the growth in PyPI downloads: around 1 in 5 of those downloads (633K precisely) were in the last month alone . Notably, PyPI is just one of several distribution channels for zizmor: the official Docker image has another ~140K downloads, the Homebrew formula has another ~50K, conda-forge has another ~90K, and so forth. And these are only the ones I can easily track! So, so many cool downstream users: cURL, CPython, PyPI itself, Rust, Rustls, Sigstore, Anubis, and many others are all running zizmor in their CI/CD to proactively catch potential security issues. In addition to these projects (who humble me with their use), I’ve also been thrilled to see entire communities and companies adopt and/or recommend zizmor: pyOpenSci , Grafana Labs , and Wiz have all publicly recommended zizmor based on their own experience or expertise. Roughly 50 contributors , excluding myself and bots, along with several dozen “regulars” who file bug reports and feature requests. This number is easily the most satisfying of the above: it’s a far cry from the project’s start, when I wasn’t sure if anyone would ever use it, let alone contribute to it. It’s very easy to install and run: distributing it via PyPI (even though it’s a pure Rust binary) means that it’s a single or away for most users. It’s very fast by default: offline runs take tens of milliseconds for representative inputs; online runs are slower, but still fast enough to outpace typical CI/CD setups. In other words: people very rarely wait for zizmor to complete, or in the worst case wait no longer than they’re already waiting for the rest of their CI/CD. It strikes the right balance with its persona design: the default (“ regular ”) persona prioritizes signal over noise, while letting users opt into more noisy personae (like pedantic and auditor ) as they please. One outcome from this choice which has been (pleasantly) surprising is seeing users opt into a more sensitive persona than the default, including even enforcing zizmor at the pedantic or auditor level in their CI/CD. I didn’t expect this (I expected most users to ignore non-default personae), and it suggests that zizmor’s other usability-first design decisions afford us a “pain budget” that users are willing to spend on more sensitive checks. More interprocedural analysis: zizmor is largely “intraprocedural” in the moment, in the sense the it analyzes individual inputs (workflows or action definitions) in isolation. This approach makes zizmor simple, but it also leaves us with an partial picture of e.g. a repository’s overall CI/CD posture. As a simple example: the unpinned-uses audit will correctly flag any unpinned action usages, but it won’t detect transitively unpinned usages that cross input boundaries. Better support for large-scale users: it’s increasingly clear to me that a significant portion (and increasing) portion of zizmor’s userbase is security teams within bigger open source projects (and companies), who want to use zizmor as part of “estate management” for dozens, hundreds, or even thousands of individual repositories. zizmor itself doesn’t struggle to operate in these settings, but large scales make it harder to triage and incrementally address zizmor’s findings. I’m not sure exactly what an improved UX for bulk triage will look like, but some kind of GitHub App integration seems worth exploring 6 . A bigger architectural reevaluation: zizmor’s current architecture is naïve in the sense that individual audits don’t share or re-use computation between each other. For example, two audits that both evaluate GitHub Actions expressions will independently re-parse those expressions rather than caching that reusing that work. This is not a performance issue at zizmor’s current audit count, but will likely eventually become one. When this happens, I’ll likely need to think about a larger architectural change that allows audits to either share computed analysis state or push more analysis state into zizmor’s input collection phase. Another (unrelated) architectural change that’ll likely eventually need to happen involves , which zizmor currently uses extensively: its deprecated status is a long-term maintenance risk. My hope is that alternatives (like saphyr , which I’ve been following) will become sufficiently mature and feature-rich in the medium-long term to enable fully replacing (and potentially even some of our use of in e.g. and .). It depends on how you count: zizmor’s first commit was roughly 13 months ago, while its first tagged release (v0.1.0) was roughly 11 months ago. So I’m splitting the two and saying it’s been one year.  ↩ Number #9,047 as of time of writing.  ↩ Read: product leadership. I have a great deal of faith in and respect for GitHub’s engineers, who seem to be uniformly upset about the slow-but-accelerating degradation of GitHub’s product quality.  ↩ A more cynical framing would be that GitHub has entered the “value strip-mining” phase of its product lifecycle, where offerings like GitHub Actions are kept alive just enough to meet contractual obligations and serve as a staging bed for the next round of “innovation” (read: AI in more places). Fixing longstanding flaws in GitHub Actions’ design and security model would not benefit this phase, so it is not prioritized.  ↩ Off the top of my head: immutable actions/releases , fixing action policies , acknowledging how dangerous is, &c.  ↩ This avenue of exploration is not without risks: my experience has been that many security tools that choose to go the “app” integration route end up as Yet Another Infernal Dashboard that security teams dread actually using.  ↩

Open Source Security DevOps Rust C++

▲ 0

0 views

ENOSUCHBLOG 2 months ago

Fun with finite state transducers

I recently 1 solved an interesting problem inside with a type of state machine/automaton I hadn’t used before: a finite state transducer (FST) . This is just a quick write-up of the problem and how I solved it. It doesn’t go particularly deep into the data structures themselves. For more information on FSTs themselves, I strongly recommend burntsushi’s article on transducers (which is what actually led me to his crate). TL;DR: I used the crate to build a finite state transducer that maps GitHub Actions context patterns to their logical “capability” in the context of a potential template injection vulnerability. This ended up being an order of magnitude smaller in terms of representation (~14.5KB instead of ~240 KB) and faster and more memory efficient than my naïve initial approaches 2 (tables and prefix trie walks). It also enabled me to fully precompute the FST at compile time, eliminating the startup cost of priming a trie- or table-based map. is a static analysis tool for GitHub Actions. One of the categories of weaknesses it can find is template injections , wherein the CI author uses a GitHub Actions expression in a shell or similar context without realizing that the expression can escape any shell-level quoting intended to “defuse” it. Here’s an example, derived from a pattern that gets exploited   over   and over   again : If this step is part of a workflow that grants elevated privileges to third parties (like ), and attacker can contrive a ref that escapes the shell quoting and runs arbitrary code. For example, the following ref: …would expand 3 as: Fortunately, detects these: There’s a very simple way to detect these vulnerabilities: we could walk every code “sink” in a given workflow (e.g. blocks, action inputs that are known to contain code, &c.) and look for the fences of an expression ( ). If we see those fences, we know that the contents are a potential injection risk. This is appealing for reasons of simplicity, but is unacceptably noisy : There are many actions expressions that are trivially safe, or non-trivial but deductively safe: Any expression that can only expand to a literal: Any expression that can’t expand to meaningful code, e.g. due to the expression’s type: There are many expressions that can appear unsafe by virtue of dataflow or context expansion, but are actually safe because of the context’s underlying type or constraints: generally aims to present low-noise findings, so filtering these out by default is paramount. The first group is pretty easy: we can do a small amount of dataflow analysis to determine whether an expression’s evaluation is “tainted” by arbitrary controllable inputs. The second group is harder, because it requires to know additional facts about arbitrary-looking contexts. The two main facts we care about are type (whether a context expands to a string, a number, or something else) and capability (whether the expansion is fully arbitrary, or constrained in some manner that might make it safe or at least less risky). In practice these both collapse down to capability , since we can categorize certain types (e.g. booleans and numbers) as inherently safe. So, what we want is a way to collect facts about every valid GitHub Actions context. The trick to this lies in remembering that, under the hood, GitHub Actions is driven by GitHub’s webhooks API : most 4 of the context state loaded into a GitHub Actions workflow run is derived from the webhook payload corresponding to the event that triggered the workflow. So, how do we get a list of all valid contexts along with information about their expansion? GitHub doesn’t provide this directly, but we can derive it from their OpenAPI specification for the webhooks API . This comes in the form of a ~4.5MB OpenAPI schema , which is pain in the ass to work with directly 5 : it’s both heavily self-referential (by necessity, since an “unrolled” version with inline schemas for each property would infeasibly large), is heavily telescoped (also by necessity, since GitHub’s API responses themselves are not particularly flat), and makes ample use of OpenAPI constructions like , , and that require careful additional handling. At the bottom of all of this, however, is our reward: detailed information about every property provided by each webhook event, including the property’s type and valuable information about how the property is constrained. For example, here’s the schema for a property: This tells us that is a string, but that its value is constrained to either or . We categorize this as having a “fixed” capability, since we know that the attacker can’t control the structure of the value itself in a meaningful way. Long story short: this is implemented as a helper script within called . This script is run periodically in GitHub Actions and walks the OpenAPI scheme to produces a CSV, , that looks like this: …to the tune of about 4000 unique 6 contexts. So, we have a few thousand contexts, each with a capability that tells us how much of a risk that context poses in terms of template injection. We can just shove these into a map and call it a day, right? Wrong. We’ve glossed over a significant wrinkle, which is that context accesses in GitHub Actions are not themselves always literal. Instead, they can be patterns that can expand to multiple values. A good example of this is : is an array of objects , each of which has a property corresponding to the label’s actual name. To access these, we can use syntaxes that select individual labels or all labels: In both cases, we want to apply the same capability to the context’s expansion. To make things even more complicated exciting, GitHub’s own context access syntax is surprisingly malleable: each of the following is a valid and equivalent way to access the first label’s name: ..and so forth. In sum, we have two properties that blow a hole in our “just shove it in a map” approach: To me, this originally smacked of a prefix/radix trie problem: there are a a large number of common prefixes/segments in the pattern set, meaning that the trie could be made relatively compact. However, tries are optimized for operations that the problem doesn’t require: Finally, on a more practical level: I couldn’t find a great trie/radix trie crate to use. Some of this might have been a discovery failure on my part, but I couldn’t find one that was already widely used and still actively maintained. came the closest, but hasn’t been updated in nearly 5 years. While reading about other efficient prefix representation structures, I came across DAFSAs (also sometimes called DAWGs). These offer a significantly more compact representation of prefixes than a trie, but at a cost: unlike a trie, a DAFSA cannot contain auxiliary data. This makes them great for inclusion checking (including of prefixes), but not so great for my purpose of storing each pattern’s associated capability. That brought me to transducers as a class of finite state machines: unlike acceptors (DAFSAs, but also normal DFAs and NFAs) that map from an input sequence to a boolean accept/reject state, transducers map an input sequence to an output sequence. That output sequence can then be composed (e.g. via summation) into an output value. In effect, the “path” an input takes through the transducer yields an output value. In this way, FSTs can behave a lot like a map (whether backed by a prefix trie or a hash table), but with some appealing additional properties: These desirable properties come with downsides too: optimal FST construction requires memory proportional to the total input size, and requires ordered insertion of each input. Modifications to an FST are also limited: optimal insertions must be ordered, while deletions or changes to an associated value require a full rebuild of the FST. In practice, this that FST construction is a static affair over a preprocessed input set. But that’s perfectly fine for my use case! As it turns out, using the crate to construct an FST at build time is pretty simple. Here’s the totality of the code that I put in to transform : …and then, loading and querying it: (where the input has been normalized, e.g. from to ). Everything above was included in zizmor 1.9.0 , as part of a large-scale refactor of the template-injection audit. Was this overkill? Probably: I only have about ~4000 valid context patterns, which would have comfortably fit into a hash table. However, using an FST for this makes the footprint ludicrously and satisfyingly small : each pattern takes less than 4 bytes to represent in the serialized FST, well below the roughly linear memory footprint of loading the equivalent data from . Using an FST also unlocked future optimizations ideas that I haven’t bothered to experiment with yet: A few months ago.  ↩ Spoiler: I didn’t bother benchmarking this at the time.  ↩ I’ve replaced with a single literal space for readability.  ↩ But not all: the and namespaces, for example, are populated by the GitHub Actions runner itself.  ↩ I can only assume that these kinds of huge generated schemas are actually useful to others. I think I’ve yet to encounter a JSON or OpenAPI schema that actually made my task at hand easier.  ↩ There are significantly more possible contexts than these, but most are overlapping. For example, is shared across all webhook types. In theory we expect these variants of the same context to converge on the same capability — shouldn’t really change type or structure with the webhook type. However, we can’t guarantee that, so in practice we unify duplicate contexts into their “strongest” capability: becomes , for example. This is conceptually an overapproximation, but whatever.  ↩ At least, not without breaking out “offline first” policy. Plus, there are other resources whose cardinality GitHub won’t easily tell us.  ↩ There are many actions expressions that are trivially safe, or non-trivial but deductively safe: Literals, e.g. or ; Any expression that can only expand to a literal: Any expression that can’t expand to meaningful code, e.g. due to the expression’s type: There are many expressions that can appear unsafe by virtue of dataflow or context expansion, but are actually safe because of the context’s underlying type or constraints: is populated by GitHub’s backend and can only expand to or , but requires us to know a priori that it’s a “safe” context; is an arbitrary string, but is limited in structure to characters that make it infeasible to perform a useful injection with (no semicolons, , &c.). Contexts are patterned , and can’t be expanded into a static finite enumeration of simplified contexts. For example, we can’t know how many labels a repository has 7 , so we can’t statically unfold the context into contexts that match everything the user might write in a workflow. Contexts can be expressed through multiple syntaxes. We can keep things simple on the capability extraction side by only using the -ish syntax, but we still need to normalize any contexts as they appear in workflows. This is not very difficult, but it makes a single map lookup even less appealing. We have a few thousands contexts, each of which is really a pattern that can match one or more “concrete” context usages as they appear in a user’s workflow. Each of these context patterns has an associated capability, as one of , indicating how much of a risk the context poses in terms of template injection. Our goal is to efficiently match these patterns against the contexts as they appear in a user’s workflow, and return the associated capability. Tries are optimized to grow and shrink at runtime, but we don’t need that: the set of context patterns is static, and we’d ideally trade off some compile-time cost for runtime size and speed. Tries can perform efficient prefix and exact matches, but (typically) at the cost of a larger runtime memory footprint. A finite state transducer can compress its entire input, unlike a prefix trie (which only compresses the prefixes). That means a more compact representation. A finite state transducer can share duplicated output values across inputs, unlike a prefix trie or hash table (which would store the same output value multiple times). This also means a more compact representation, and is particularly appealing in our case as we only have a small set of possible capabilities. The FST is currently searched using a normalization of each context as it appears in a workflow. However, FSTs can be searched by any DFA, meaning that I could in theory convert each context into a regular expression instead. I’m unclear on whether this would bring a performance advantage, since the context-to-regex-to-DFA conversion itself is not necessarily cheap. In principle, the FST’s size could be squeezed down even further by splitting the context patterns into segments , rather than decomposing into sequences of bytes. This comes from the observation that many vertices in the FST are shared and singular, meaning that they only have one incoming edge and one outgoing edge. I don’t think the crate supports this natively, but it would yield the prefix deduplication benefits of a prefix trie while still preserving the compression benefits of an FST. A few months ago.  ↩ Spoiler: I didn’t bother benchmarking this at the time.  ↩ I’ve replaced with a single literal space for readability.  ↩ But not all: the and namespaces, for example, are populated by the GitHub Actions runner itself.  ↩ I can only assume that these kinds of huge generated schemas are actually useful to others. I think I’ve yet to encounter a JSON or OpenAPI schema that actually made my task at hand easier.  ↩ There are significantly more possible contexts than these, but most are overlapping. For example, is shared across all webhook types. In theory we expect these variants of the same context to converge on the same capability — shouldn’t really change type or structure with the webhook type. However, we can’t guarantee that, so in practice we unify duplicate contexts into their “strongest” capability: becomes , for example. This is conceptually an overapproximation, but whatever.  ↩ At least, not without breaking out “offline first” policy. Plus, there are other resources whose cardinality GitHub won’t easily tell us.  ↩

Shell DevOps Programming JSON

▲ 0

0 views

ENOSUCHBLOG 4 months ago

A new adventure

This is a personal announcement post: after 7 years at Trail of Bits , I’m leaving to do something new. Specifically, I’ll be joining Astral to help them build the next generation of Python developer tooling. Trail of Bits has been a really wonderful place to work: they hired me right out of college, and gave me the kind of high-trust, applied-lab-like environment that I thrive in. That environment enabled me to build the company’s open source engineering practice from the ground up and drive all kinds of high-visibility , high-impact projects across the worlds of Python , cryptography , fault injection , fuzzing , and too much more to summarize. Finally, I’ve had the immense privilege of working with some of the smartest, friendliest, and most driven people in the world of cybersecurity. It feels cliche to say, but there really are too many to name. Company photo from 2019, my first offsite.

Python Security

▲ 0

0 views

ENOSUCHBLOG 4 months ago

Bypassing GitHub Actions policies in the dumbest way possible

TL;DR : GitHub Actions provides a policy mechanism for limiting the kinds of actions and reusable workflows that can be used within a repository, organization, or entire enterprise. Unfortunately, this mechanism is trivial to bypass . GitHub has told me that they don’t consider this a security issue (I disagree), so I’m publishing this post as-is. Update 2025-06-13 : GitHub has silently updated the actions policies documentation to note the bypass in this post: Policies never restrict access to local actions on the runner filesystem (where the path start with ). GitHub Actions is GitHub’s CI/CD offering. I’m a big fan of it, despite its spotty security track record . Because a CI/CD offering is essentially arbitrary code execution as a service , users are expected to be careful about what they allow to run in their workflows, especially privileged workflows that have access to secrets and/or can modify the repository itself. That, in effect, means that users need to be careful about what actions and reusable workflows they trust. Like with other open source ecosystems, downstream consumers (i.e., users of GitHub Actions) retrieve their components (i.e., action definitions) from an essentially open index (the “Actions Marketplace” ​ 1 ). To establish trust in those components, downstream users perform all of the normal fuzzy heuristics: they look at the number of stars, the number of other user, recency of activity, whether the user/organization is a “good” one, and so forth. Unfortunately, this isn’t good enough along two dimensions: Even actions that satisfy these heuristics can be compromised. They’re heuristics after all, not verifiable assertions of quality or trustworthiness. The recent tj-actions attack typifies this: even popular, widely-used actions are themselves software components, with their own supply chains (and CI/CD setups). This kind of acceptance scheme just doesn’t scale , both in terms of human effort and system complexity: complex CI/CD setups can have dozens (or hundreds) of workflows, each of which can contain dozens (or hundreds) of jobs that in turn employ actions and reusable workflows. These sorts of large setups don’t necessarily have a single owner (or even a single team) responsible for gating admission and preventing a the introduction of unvetted actions and reusable workflows. The problem (as stated above) is best solved by eliminating the failure mode itself: rather than giving the system’s committers the ability to introduce new actions and reusable workflows without sufficient review, the system should prevent them from doing so in the first place . To their credit, GitHub understands this! They have a feature called “Actions policies 2 ” that does exactly this. From the Manage GitHub Actions settings documentation: You can restrict workflows to use actions and reusable workflows in specific organizations and repositories. Specified actions cannot be set to more than 1000. (sic) To restrict access to specific tags or commit SHAs of an action or reusable workflow, use the same syntax used in the workflow to select the action or reusable workflow. For an action, the syntax is . For example, use to select a tag or to select a SHA. For more information, see Using pre-written building blocks in your workflow. For a reusable workflow, the syntax is . For example, . For more information, see Reusing workflows. You can use the wildcard character to match patterns. For example, to allow all actions and reusable workflows in organizations that start with , you can specify . To allow all actions and reusable workflows in repositories that start with , you can use . For more information about using the wildcard, see Workflow syntax for GitHub Actions. Use to separate patterns. For example, to allow and , you can specify . GitHub also provides special “preset” cases for this functionality, such as allowing only actions and reusable workflows that belong to the same organization namespace as the repository itself. Here’s what that looks like on a dummy organization and repository of mine: …and here’s what happens when I try to violate that policy, e.g. by using in a workflow: This is fantastic, except that it’s trivial to bypass. Let’s see how. To understand how we’re going to bypass this, we need to understand a few of the building blocks underneath actions and reusable workflows. In particular: These four aspects of GitHub Actions compose together into the world’s dumbest policy bypass : instead of doing , the user can (or otherwise fetch) the repository into the runner’s filesystem, and then use to run the very same action. Here’s what that looks like in practice: (The actual block of the step is inconsequential — I just used that repository for the demo, but anything would work.) And naturally, it works just fine: The fix for this bypass is simple, if potentially somewhat painful: GitHub Actions could consider “local” references to be another category for the purpose of policies, and reject them whenever the policy doesn’t permit them. This would seal off the entire problem, since would just stop working. The downside is that it would potentially break existing users of policies who also use local actions and reusable workflows, assuming there are significant numbers of them 4 . The other option would be to leave it the way it is, but explicitly document local references as a limitation of this policy mechanism. I honestly think this would be perfectly fine; what matters is that users 5 are informed of a feature’s limitations, not necessarily that the feature lacks limitations. First, I’ll couch this again: this is not exactly fancy stuff. It’s a very dumb bypass, and I don’t think it’s critical by any means. At the same time, I think this matters a great deal : ineffective policy mechanisms are worse than missing policy mechanisms, because they provide all of the feeling of security through compliance while actually incentivizing malicious forms of compliance . In this case, the maliciously complying party is almost certainly a developer just trying to get their job done: like most other developers who encounter an inscrutable policy restriction, they will try to hack around it such that the policy is satisfied in name only. For that reason alone I think GitHub should fix this bypass, either by actually fixing it or at least documenting its limitations. Without either of those, projects and organizations are likely to mistakenly believe that these sorts of policies provide a security boundary where none in fact exists . Technically “publishing” an action to the Actions Marketplace is not required; anybody can do to fetch the action defined in even if it isn’t published. All publishing does is give the action a marketplace page and the potential for a little blue checkmark of unclear security value.  ↩ Actually, I don’t know what this feature is called. It’s titled “Policies” under the “Actions” section of the repo/org/enterprise settings and is documented under “Github Actions policies” in the Enterprise documentation, but I’m not sure if that’s an umbrella term or not. I’m just going to keep calling it “Actions policies” for now.  ↩ Where “owner” is an individual owner or an organization, which in turn might be controlled by an enterprise. But that last bit isn’t visible in the namespace.  ↩ I honestly have no idea how widely used this policy feature is.  ↩ Here, policy authors and enforcers.  ↩ Even actions that satisfy these heuristics can be compromised. They’re heuristics after all, not verifiable assertions of quality or trustworthiness. The recent tj-actions attack typifies this: even popular, widely-used actions are themselves software components, with their own supply chains (and CI/CD setups). This kind of acceptance scheme just doesn’t scale , both in terms of human effort and system complexity: complex CI/CD setups can have dozens (or hundreds) of workflows, each of which can contain dozens (or hundreds) of jobs that in turn employ actions and reusable workflows. These sorts of large setups don’t necessarily have a single owner (or even a single team) responsible for gating admission and preventing a the introduction of unvetted actions and reusable workflows. For an action, the syntax is . For example, use to select a tag or to select a SHA. For more information, see Using pre-written building blocks in your workflow. For a reusable workflow, the syntax is . For example, . For more information, see Reusing workflows. Actions and reusable workflows share the same namespace as the rest of GitHub, i.e. 3 ; When a user writes something like in a workflow, GitHub resolves that reference to mean “the file defined at tag in the repository”; keywords can also refer to relative paths on the runner itself. For example, runs the step with the in the current directory. Relative paths from the runner are not inherently part of the repository state itself: the runner is can contain any state introduced by previous steps within the same job. Technically “publishing” an action to the Actions Marketplace is not required; anybody can do to fetch the action defined in even if it isn’t published. All publishing does is give the action a marketplace page and the potential for a little blue checkmark of unclear security value.  ↩ Actually, I don’t know what this feature is called. It’s titled “Policies” under the “Actions” section of the repo/org/enterprise settings and is documented under “Github Actions policies” in the Enterprise documentation, but I’m not sure if that’s an umbrella term or not. I’m just going to keep calling it “Actions policies” for now.  ↩ Where “owner” is an individual owner or an organization, which in turn might be controlled by an enterprise. But that last bit isn’t visible in the namespace.  ↩ I honestly have no idea how widely used this policy feature is.  ↩ Here, policy authors and enforcers.  ↩

Ci Cd Security

▲ 0

0 views

ENOSUCHBLOG 5 months ago

A Discord server and new GitHub organization for zizmor

TL;DR : now has a Discord server and a new GitHub organization ( @zizmorcore ). Feel free to join the Discord server, and be on the lookout for an official transition of @woodruffw/zizmor to the new organization in the coming weeks! (Click me for an invite!) is a static analysis tool for GitHub Actions. I’ve been building it for the last few months (along with the help of a lot of fantastic contributors), and the impact it’s had on the overall security of the GitHub Actions ecosystem has been very gratifying to see. You can read more about in some earlier posts: However, this post isn’t a technical post about itself. It’s more of a PSA for some organizational changes that are coming to qua project . These changes won’t affect ordinary users of in any way, but they will hopefully make it easier for me to grow ’s community and coordinate with contributors and users looking for help. now has a Discord server ! This server is intended as a place for both users and maintainers/contributors to talk (both about and CI/CD security more generally), ask questions, and discuss ideas that need to “pre-germinate” before they can be turned into normal GitHub issues or PRs. I’m hoping that this server will complement the existing GitHub issue tracker and discussion board, rather than replace them — the goal is to make it easier for people to self-select into different “tracks” for discussion: Or in other words: if you’re new or just don’t know where to start with a topic, pop into Discord and ask! Most things will still happen on GitHub, but Discord is a great place to get started. Finally: I’m not exactly an expert when it comes to Discord; this is my first time really using it, particularly in an administrative capacity. I’m very open to feedback and suggestions on how to improve the server and make it a better place for discussion. A PSA for the Discord server is also tracked in zizmor#756 . has a new GitHub organization: @zizmorcore ! This organization will soon be the new home of @woodruffw/zizmor , along with all current other -related repositories (like @woodruffw/yamlpath and @woodruffw/github-actions-models ). This new organization will also be the home of new projects under the umbrella, including dedicated repositories for workflows/actions, IDE plugins, and so forth. Stay tuned for dedicated announcements there! In practice, this will have little to no effect on most users of : commands like and will continue to work exactly as they do now, and GitHub will transparently handle the repository redirect itself. The main visible changes from this will be: The current plan is to perform this transfer with the release of , i.e. the following minor release (since is already planned for release in the coming days). This will give me time to figure out how bad the link breakage will be and, if unacceptable, to consider workarounds (like redirects, or possibly a whole new domain name) before sealing the deal. This change is also being tracked in zizmor#758 . October 2024: Introducing zizmor: now you can have beautiful clean workflows December 2024: zizmor would have caught the Ultralytics workflow vulnerability January 2025: zizmor 1.0 “Ordinary” user questions (things like “how do I use ?” and “why doesn’t this flag work?”) belong in Discord, since they’re often ephemeral and don’t require permanent tracking. Early technical discussions (things like “I have an idea for a new audit, but I have no idea where to get started”) belong in Discord, since they often require back-and-forth discussion and don’t need to be tracked in the issue tracker until firmed up. Just about everything else (things like qualified bug reports and feature requests) belong in the GitHub issue tracker, so that they don’t get lost in the backlog of a chat and can be categorized and triaged. The GitHub slug will go from @woodruffw/zizmor to @zizmorcore/zizmor (which doesn’t exist yet). Contributors may want to upgrade their remotes to point to the new organization, although GitHub will also handle the redirect transparently. The current documentation site ( https://woodruffw.github.io/zizmor/ ) will be moved to https://zizmorcore.github.io/zizmor/ , and all references to the former (e.g. in zizmor’s outputs) will be moved to the latter. GitHub unfortunately doesn’t support redirects for GitHub Pages like they do for repository renames, so users on older versions of may experience some link breakage.

Open Source

▲ 0

0 views

ENOSUCHBLOG 9 months ago

Be aware of the Makefile effect

Update 2024-01-12 : Ken Shirriff has an an excellent blog post on why “cargo cult” is a poor term of art. I’m not aware of a perfect 1 term for this, so I’m making one up: the Makefile effect 2 . The Makefile effect boils down to this: Tools of a certain complexity or routine unfamiliarity are not run de novo , but are instead copy-pasted and tweaked from previous known-good examples. You see this effect frequently with engineers of all stripes and skill/experience levels, with Make being a common example 3 : On one level, this is a perfectly good (even ideal) engineering response at the point of solution : applying a working example is often the parsimonious thing to do, and runs a lesser (in theory) risk of introducing bugs, since most of the work is unchanged. However, at the point of design , this suggests a tool design (or tool application 5 ) that is flawed : the tool (or system) is too complicated (or annoying) to use from scratch. Instead of using it to solve a problem from scratch, users repeatedly copy a known-good solution and accrete changes over time. Once you notice it, you start to see this pattern all over the place. Beyond Make: In many cases, perhaps not. However, I think it’s worth thinking about, especially when designing tools and systems: Tools and systems that enable this pattern often have less-than-ideal diagnostics or debugging support: the user has to run the tool repeatedly, often with long delays, to get back relatively small amounts of information. Think about CI/CD setups, where users diagnose their copy-pasted CI/CD by doing print-style debugging over the network with a layer of intermediating VM orchestration. Ridiculous! Tools that enable this pattern often discourage broad learning : a few mavens know the tool well enough to configure it, and others copy it with just enough knowledge to do targeted tweaks. This is sometimes inevitable, but often not: dependency graphs are an inherent complexity of build systems, but remembering the difference between and in Make is not. Tools that enable this pattern are harder to use securely : security actions typically require deep knowledge of the why behind a piece of behavior. Systems that are subject to the Makefile effect are also often ones that enable confusion between code and data (or any kind of in-band signalling more generally), in large part because functional solutions are not always secure ones. Consider, for example, about template injection in GitHub Actions. In general, I think well-designed tools (and systems) should aim to minimize this effect. This can be hard to do in a fully general manner, but some things I think about when designing a new tool: The Makefile effect resembles other phenomena, like cargo culting, normalization of deviance, “write-only language,” &c. I’ll argue in this post that it’s a little different from each of these, insofar as it’s not inherently ineffective or bad and concerns the outcome of specific designs .  ↩ Also note: the title is “be aware,” not “beware.” The Makefile effect is not inherently bad! It’s something to be aware of when designing tools and systems.  ↩ Make is just an example, and not a universal one: different groups of people master different tools. The larger observation is that there are classes of tools/systems that are (more) susceptible to this, and classes that are (relatively) less susceptible to it.  ↩ I’ve heard people joke about their “heritage” s, i.e. s that were passed down to them by senior engineers, professors, &c. The implication is that these forebearers also inherited the , and have been passing it down with small tweaks since time immemorial.  ↩ Complex tools are a necessity; they can’t always be avoided. However, the occurrence of the Makefile effect in a simple application suggests that the tool is too complicated for that application.  ↩ A task (one of a common shape) needs completing. A very similar (or even identical) task has been done before. Make (or another tool susceptible to this effect) is the correct or “best” (given expedience, path dependencies, whatever) tool for the task. Instead of writing a , the engineer copies a previous (sometimes very large and complicated 4 ) from a previous instance of the task and tweaks it until it works in the new context. CI/CD configurations like GitHub Actions and GitLab CI/CD, where users copy their YAML spaghetti from the last working setup and tweak it (often with repeated re-runs) until it works again; Linter and formatter configurations, where a basic set of rules gets copied between projects and strengthened/loosened as needed for local conditions; Build systems themselves, where everything non-trivial begins to resemble the previous build system. Tools and systems that enable this pattern often have less-than-ideal diagnostics or debugging support: the user has to run the tool repeatedly, often with long delays, to get back relatively small amounts of information. Think about CI/CD setups, where users diagnose their copy-pasted CI/CD by doing print-style debugging over the network with a layer of intermediating VM orchestration. Ridiculous! Tools that enable this pattern often discourage broad learning : a few mavens know the tool well enough to configure it, and others copy it with just enough knowledge to do targeted tweaks. This is sometimes inevitable, but often not: dependency graphs are an inherent complexity of build systems, but remembering the difference between and in Make is not. Tools that enable this pattern are harder to use securely : security actions typically require deep knowledge of the why behind a piece of behavior. Systems that are subject to the Makefile effect are also often ones that enable confusion between code and data (or any kind of in-band signalling more generally), in large part because functional solutions are not always secure ones. Consider, for example, about template injection in GitHub Actions. Does it need to be configurable? Does it need syntax of its own? As a corollary: can it reuse familiar syntax or idioms from other tools/CLIs? Do I end up copy-pasting my use of it around? If so, are others likely to do the same? The Makefile effect resembles other phenomena, like cargo culting, normalization of deviance, “write-only language,” &c. I’ll argue in this post that it’s a little different from each of these, insofar as it’s not inherently ineffective or bad and concerns the outcome of specific designs .  ↩ Also note: the title is “be aware,” not “beware.” The Makefile effect is not inherently bad! It’s something to be aware of when designing tools and systems.  ↩ Make is just an example, and not a universal one: different groups of people master different tools. The larger observation is that there are classes of tools/systems that are (more) susceptible to this, and classes that are (relatively) less susceptible to it.  ↩ I’ve heard people joke about their “heritage” s, i.e. s that were passed down to them by senior engineers, professors, &c. The implication is that these forebearers also inherited the , and have been passing it down with small tweaks since time immemorial.  ↩ Complex tools are a necessity; they can’t always be avoided. However, the occurrence of the Makefile effect in a simple application suggests that the tool is too complicated for that application.  ↩

Programming DevOps Yaml Design

▲ 0

0 views

ENOSUCHBLOG 9 months ago

zizmor 1.0

Happy New Year! I’m pleased to announce that 1.0 is now released. Read the release notes here . For those unacquainted: is a static analysis tool for GitHub Actions. It can find security issues in GitHub Actions workflow and action definitions, including the kinds of weaknesses that compromised a major machine learning project just last month. I released to the public about 2 months ago, and interest and adoption have been meteoric: over 1600 people have starred the repository (as of writing), and major projects like Python, PyPI, cURL, urllib3, and RubyGems are using it to secure their CI/CD setups 1 . The docs contain lots of additional details, including how to install (it’s very simple!), as well as how to get started with auditing your own and others’ repositories. Finally, some words of thanks: A full list of projects and companies using can be found in the trophy case .  ↩ Thank you to Alex Gaynor , Paul Kehrer , Mike Fiedler for being early adopters and alpha testers. Thank you to Hugo van Kemenade and Alyssa Coghlan for their detailed bug reports and adoption efforts. Thank you to Ben Cotton for being the project’s first (and so far, only!) sponsor. Thank you to the 20+ people who have contributed to so far, including the many more who contributed feedback and bug reports that don’t show up in GitHub contribution statistics. And finally, a huge thank you to Ubiratan Soares , who has both added and significantly improved ’s set of audits ! A full list of projects and companies using can be found in the trophy case .  ↩

Open Source Security DevOps Python

▲ 0

0 views

ENOSUCHBLOG 10 months ago

zizmor would have caught the Ultralytics workflow vulnerability

TL;DR : would have caught the vulnerability that caused this…mostly. Read on for details. Important: I’m writing this post in real time as I learn more about what happened here. I’ll be updating it throughout the day. EDIT : I’ve reached the point here where I feel comfortable making some inferences/conclusions. These are in the Conclusions section. EDIT 2024-12-07 : Today is the last day I’ll be making updates to this post. The Conclusions are now fully updated, and I’ve added a rough (but comprehensive) timeline of events as an appendix. Yesterday, someone exploited Ultralytics , which is a very popular machine learning package for vision stuff™. The attacker appears to have compromised Ultralytics’ CI, and then pivoted to making a malicious PyPI release (v8.3.41, now deleted 1 ), which contained a crypto miner. It appears as though a subsequent release (v8.3.42, also now deleted) was also malicious. UPDATE 2024-12-07 : Ultralytics was compromised again , within 36 hours of the last compromise. This latest compromise appears to have resulted in two more malicious releases, v8.3.45 and v8.3.46, both of which were released directly to PyPI and have since been deleted. These releases appear to have been pushed directly by the attacker using an API token stolen from the ultralytics/ultralytics CI/CD, likely at the same time as they conducted the original exfiltration and cache poisoning attack. Here’s the rough flow of what happened: The @openimbot account opens a PR, #18020 , against the upstream ultralytics/ultralytics repository. PR #18020 has a malicious branch name: or formatted, with expanded with spaces: NOTE : I haven’t been able to get my hands on this payload yet; if you have access to it, ping me! This gets picked up by the workflow , which has a fundamentally dangerous workflow trigger ( ). calls a custom action, defined in : The custom action is a composite action with shell steps in its , including the following: Line above is a classic GitHub Actions template injection : the expansion of is injected directly into the shell’s context, with no quoting or interpolation. I believe this is where the malicious branch name gets injected, resulting in the payload (inside ) being run. From here, the attacker is running code of their choice in a context, meaning that (by default) they have access to anything a normal privileged workflow can do. In particular, that means they (almost certainly) had (or still have) push access to ultralytics/ultralytics itself, as well as to the repository’s privileged caches. Either one of these was/is an effective vector for compromising the repository contents and/or the contents of its artifacts. I don’t know yet how the attacker pivoted from this context to compromising the PyPI package that was then uploaded and eventually noticed by users in #18027 . Their underlying technique there would probably be revealed by the payload inside of the shell script above. EDIT : My colleague Will Tan pointed out that one of the fix PRs, #18052 , also removes from one of the sites. This suggests that the underlying vector was indeed a poisoned cache, introduced after the code injection. EDIT : Seth Larson and Ee Durbin point out that the Ultralytics is using Trusted Publishing with PyPI, but they aren’t using a configured deployment environment. That means that they have/had no additional signoff or other environment protections on PyPI releases, which in turn suggests that workflow compromise was sufficient to induce the publish event. EDIT : Adnan Khan has pointed out the likely cache entry that was poisoned to compromise the build in this case. Combined with the exfiltration observed below, this very strongly suggests that the attacker used a poisoned cache. Regardless of how the pivot occurred, it appears as though is the triggering commit for the first malicious release: it bumps the version to the first known malicious version (v8.3.41) and, critically , removes a check that limited who could do triggers from the branch: At this point, the workflow is triggered on a event for the branch, resulting in an action run that published v8.3.41 to PyPI. I’ve uploaded the action log here . The sdist and wheel for v8.3.41 can also be found in Sigstore’s transparency log as 153415338 and 153415340 respectively. Adnan Khan also points out that 153589717 is the Sigstore transparency log entry for one of v8.3.42’s distributions. All of these log entries show a event for . All in all, malicious version of Ultralytics were available on PyPI for about 13 hours . The discussion in #18027 has lots of additional details, including some analysis of the payload in the Python package itself (which I haven’t analyzed yet). I’m breaking this section out because it’s proving to be independently interesting. This is the initial payload, which now 404s presumably because GitHub has taken it down: Andy Lindeman conducted a search for similar branch names pushed by other users, and discovered that @jiwuwgknvm (user ID 190546325, now deleted) created a branch with a similar name at . This branch references a different payload, this time in a Gist: That URL 404s, but the underlying Gist repository is still live: …and contains a standard stealer: The for the gist suggests that the @jiwuwgknvm identity is also in control of , which was one of the original IOCs for the dropped miner. EDIT : Seth Larson observes that the stealer also steals the , providing more circumstantial evidence for the cache poisoning hypothesis. Andy Lindeman also discovered an earlier GitHub identity, @jeficmer456 (also deleted), which appears to have been experimenting with the template injection even earlier (circa ): This led to another user Gist: The for this Gist suggests the identity goes back to 2024-12-01: This email identity is also separate from the one associated with the Gist, per above ( ). This is what I immediately wondered upon seeing this. Let’s find out! Here is what reports for v8.3.40 , i.e. the release state right before the attack: NOTE : Passing directly is supported on , but isn’t in a release of yet. To reproduce with a released version, you can Ultralytics and run on a checkout of v8.3.40 . I’ve excerpted the output substantially, to remove findings that Ultralytics should fix but are not immediately relevant to this particular exploit: All told, detects the key parts of the exploit chain: At the same time, does not detect the template injection within yet, since I haven’t yet added support for auditing composite actions. This is being tracked in zizmor#173 , and this incident is as good an impetus as any for me to begin work on this! Based on everything above, here’s how it likely went down: The attacker obtained code execution in the parent ( ultralytics/ultralytics ) CI context via an insecure workflow trigger ( ) combined with a template injection in a custom composite GitHub Action. In other words, they performed a traditional “pwn request” of the sort that’s been well-understood since 2021. The attacker used @openimbot as a puppet account, but also used the @jiwuwgknvm identity while developing their exploit. They may or may not also be the @jeficmer456 identity, which tried to exploit a different repository shortly before with a similar shell-script-in-Gist technique. At this point, it’s unclear whether @openimbot is controlled by the attacker or not. Once the attacker had code execution, they used a ready-made token exfiltration script that they took directly from Adnan Khan’s excellent post on GitHub Actions cache poisoning . This script probably posted back to a webhook panel on , one that we don’t have access to (since we only have their earlier attempt, not the final one). With the stolen cache token, they likely effected a cache poisoning attack on the cache used by , injecting their changes into the release distributions. Those changes were a client-side downloader stage (patched into the function) and the client side miner execution (patched into the function). There are still some loose ends here (like the actual payload), but the circumstantial evidence very strongly suggests the above. Here is the rough overall timeline of what occurred, as best as I can figure it. I’ve left out events for the @jiwuwgknvm identity’s setup and weaponization phase, since I don’t have a complete view of that yet. UPDATE : Gaëtan Ferry and Guillaume Valadon at GitGuardian have put together a reconstructed analysis of the attack , including a recovery of the final payload used by the attacker. Their analysis confirms that the attacker succeeded in their first PR ( #18018 ) but never pushed the commit that would have enabled their second PR ( #18020 ). Moreover, thanks to their repository monitoring, they were able to recover the payload itself, which is a modified variant of the the one used in the exploratory phase: 2024-08-14: Adnan Khan reports GHSA-7x29-qqmq-v6qc to the Ultralytics maintainers. This contains a template injection vulnerability virtually identical to the one used by the attacker. 2024-08-14: v0.0.3 of is released with a fix for GHSA-7x29-qqmq-v6qc . 2024-08-24: v0.0.24 of reintroduces the vulnerability in . 2024-12-03 22:28:49: the @jiwuwgknvm identity begins experimenting with a weaponized attack against ultralytics/ultralytics . It’s unclear whether this identity is the same person or people as others who attempted to weaponize code injection via branches as early as 2024-10-02. 2024-12-03 22:33:47: the @jiwuwgknvm identity submits PR #17984 (now deleted) to ultralytics/ultralytics , which contains the token stealer and exfiltration script above. From this point on, the repository should be considered to be compromised : the attacker is assumed to have access to everything in the context, including any GitHub PATs and the PyPI API token (which was not in use, since Ultralytics had switched to Trusted Publishing ). 2024-12-04 19:33: the @openimbot identity submits PR #18018 to ultralytics/ultralytics , containing a template injection via a crafted branch name. Branch payload: 2024-12-04 19:57: the @openimbot identity submits PR #18020 to ultralytics/ultralytics , containing a different template injection via a crafted branch name. Branch payload: 2024–12–04 20:50: Release for v8.3.41 is triggered by the @UltralyticsAssistant identity via commit , which also disables the actor release restriction for @glenn-jocher . This strongly suggests that the attacker is in full control of the @UltralyticsAssistant identity at this point. Workflow run: https://github.com/ultralytics/ultralytics/actions/runs/12168072999/job/33938058724 . 2024-12-04 20:51: Sigstore’s transparency log records 153415338 and 153415340 indicating two attestations, one for each of the distributions of v8.3.41 that will appear on PyPI ( and ). 2024-12-04 20:51: v8.3.41 is uploaded to PyPI with a Trusted Publisher and valid attestations for each distribution, matching the ultralytics/ultralytics Trusted Publisher identity. 2024-12-05 06:34: #18027 is opened on GitHub by Eric Johnson , identifying v8.3.41 as malicious. 2024-12-05 09:15:06: v8.3.41 is removed from PyPI (approx. 12 hours after introduction). 2024-12-05 12:46: Release for v8.3.42 is triggered by @glenn-jocher via commit . At this point the actor release restriction is back in place, but has no effect as @glenn-jocher is the one making the release. Workflow run: https://github.com/ultralytics/ultralytics/actions/runs/12180037832/job/33973482495 . 2024-12-05 12:47: Sigstore’s transparency log records 153589716 and 153589717 indicating two attestations, one for each of the distributions of v8.3.42 that will appear on PyPI ( and ). 2024-12-05 12:47: v8.3.42 is uploaded to PyPI with a Trusted Publisher and valid attestations for each distribution. 2024-12-05 13:03: @renzhexigua announces on #18027 that v8.3.42 is still malicious. 2024-12-05 13:47:30: v8.3.42 is removed from PyPI (approx. 1 hour after introduction). 2024-12-05 15:17: @glenn-jocher announces on #18027 that the @openimbot identity is banned from interacting with Ultralytics . 2024-12-07 01:41:45: v8.3.45 is directly released to PyPI, with no CI/CD or repository activity from ultralytics/ultralytics . This was done with an API token and not a Trusted Publisher, and therefore has no attestations. This strongly suggests that the attacker either obtained a PyPI API token from the original secrets exfiltration phase or is in full control pypi/u/glenn-jocher . 2024-12-07 02:27:14: v8.3.46 is directly released to PyPI, with no CI/CD or repository activity from ultralytics/ultralytics . Like with v8.3.45, this release is done via API token and has no attestation. 2024-12-07 04:00: Adnan Khan announces on #18027 that v8.3.45 and v8.3.46 are still malcious, albiet with a different (simpler) miner payload. 2024-12-07 10:08:32: v8.3.45 is removed from PyPI (approx. 8 hours after introduction). 2024-12-07 10:09:08: v8.3.46 is removed from PyPI (approx. 7.5 hours after introduction). In summary: A total of 4 different releases of Ultralytics were malicious: v8.3.41, v8.3.42, v8.3.45, and v8.3.46. All evidence strongly suggests that the attacker was able to fully exfiltrate all of the configured repository secrets within ultralytics/ultralytics . As such, the Ultralytics maintains must consider these credentials compromised and revoke them immediately to avoid further compromise . The failure to immediately revoke a stale API token may have been what enabled the follow-on direct releases of v8.3.45 and v8.3.46. All evidence strongly suggests that the attacker was in full control of the @UltralyticsAssistant bot account, and may still have control over it . The basic vulnerability that enabled this attack has been present in since 2024-08-24 and was reintroduced after a previous disclosure. This strongly suggests a lack of adequate security controls and reviews within Ultralytics’ processes. The release has been deleted and can no longer be resolved, but the malicious distribution is still available on PyPI’s storage: (replace with ).  ↩ The @openimbot account opens a PR, #18020 , against the upstream ultralytics/ultralytics repository. This appears to be a bot account, but presumably has some underlying functionality that allowed a human being to open a PR under its name. PR #18020 has a malicious branch name: or formatted, with expanded with spaces: NOTE : I haven’t been able to get my hands on this payload yet; if you have access to it, ping me! This gets picked up by the workflow , which has a fundamentally dangerous workflow trigger ( ). calls a custom action, defined in : The custom action is a composite action with shell steps in its , including the following: Line above is a classic GitHub Actions template injection : the expansion of is injected directly into the shell’s context, with no quoting or interpolation. I believe this is where the malicious branch name gets injected, resulting in the payload (inside ) being run. From here, the attacker is running code of their choice in a context, meaning that (by default) they have access to anything a normal privileged workflow can do. In particular, that means they (almost certainly) had (or still have) push access to ultralytics/ultralytics itself, as well as to the repository’s privileged caches. Either one of these was/is an effective vector for compromising the repository contents and/or the contents of its artifacts. I don’t know yet how the attacker pivoted from this context to compromising the PyPI package that was then uploaded and eventually noticed by users in #18027 . Their underlying technique there would probably be revealed by the payload inside of the shell script above. EDIT : My colleague Will Tan pointed out that one of the fix PRs, #18052 , also removes from one of the sites. This suggests that the underlying vector was indeed a poisoned cache, introduced after the code injection. EDIT : Seth Larson and Ee Durbin point out that the Ultralytics is using Trusted Publishing with PyPI, but they aren’t using a configured deployment environment. That means that they have/had no additional signoff or other environment protections on PyPI releases, which in turn suggests that workflow compromise was sufficient to induce the publish event. EDIT : Adnan Khan has pointed out the likely cache entry that was poisoned to compromise the build in this case. Combined with the exfiltration observed below, this very strongly suggests that the attacker used a poisoned cache. It flags as using a fundamentally insecure trigger ( ); It flags other sources of template/code injection in the Ultralytics workflows, including identical uses of the pattern that made their custom action exploitable. The attacker obtained code execution in the parent ( ultralytics/ultralytics ) CI context via an insecure workflow trigger ( ) combined with a template injection in a custom composite GitHub Action. In other words, they performed a traditional “pwn request” of the sort that’s been well-understood since 2021. The attacker used @openimbot as a puppet account, but also used the @jiwuwgknvm identity while developing their exploit. They may or may not also be the @jeficmer456 identity, which tried to exploit a different repository shortly before with a similar shell-script-in-Gist technique. At this point, it’s unclear whether @openimbot is controlled by the attacker or not. Once the attacker had code execution, they used a ready-made token exfiltration script that they took directly from Adnan Khan’s excellent post on GitHub Actions cache poisoning . This script probably posted back to a webhook panel on , one that we don’t have access to (since we only have their earlier attempt, not the final one). With the stolen cache token, they likely effected a cache poisoning attack on the cache used by , injecting their changes into the release distributions. Those changes were a client-side downloader stage (patched into the function) and the client side miner execution (patched into the function). The poisoned cache key is almost certainly . 2024-08-14: Adnan Khan reports GHSA-7x29-qqmq-v6qc to the Ultralytics maintainers. This contains a template injection vulnerability virtually identical to the one used by the attacker. 2024-08-14: v0.0.3 of is released with a fix for GHSA-7x29-qqmq-v6qc . 2024-08-24: v0.0.24 of reintroduces the vulnerability in . 2024-12-03 22:28:49: the @jiwuwgknvm identity begins experimenting with a weaponized attack against ultralytics/ultralytics . It’s unclear whether this identity is the same person or people as others who attempted to weaponize code injection via branches as early as 2024-10-02. 2024-12-03 22:33:47: the @jiwuwgknvm identity submits PR #17984 (now deleted) to ultralytics/ultralytics , which contains the token stealer and exfiltration script above. From this point on, the repository should be considered to be compromised : the attacker is assumed to have access to everything in the context, including any GitHub PATs and the PyPI API token (which was not in use, since Ultralytics had switched to Trusted Publishing ). 2024-12-04 19:33: the @openimbot identity submits PR #18018 to ultralytics/ultralytics , containing a template injection via a crafted branch name. Branch payload: 2024-12-04 19:57: the @openimbot identity submits PR #18020 to ultralytics/ultralytics , containing a different template injection via a crafted branch name. Branch payload: 2024–12–04 20:50: Release for v8.3.41 is triggered by the @UltralyticsAssistant identity via commit , which also disables the actor release restriction for @glenn-jocher . This strongly suggests that the attacker is in full control of the @UltralyticsAssistant identity at this point. Workflow run: https://github.com/ultralytics/ultralytics/actions/runs/12168072999/job/33938058724 . 2024-12-04 20:51: Sigstore’s transparency log records 153415338 and 153415340 indicating two attestations, one for each of the distributions of v8.3.41 that will appear on PyPI ( and ). 2024-12-04 20:51: v8.3.41 is uploaded to PyPI with a Trusted Publisher and valid attestations for each distribution, matching the ultralytics/ultralytics Trusted Publisher identity. A detailed sub-item timeline for this upload is available in this comment . 2024-12-05 06:34: #18027 is opened on GitHub by Eric Johnson , identifying v8.3.41 as malicious. 2024-12-05 09:15:06: v8.3.41 is removed from PyPI (approx. 12 hours after introduction). 2024-12-05 12:46: Release for v8.3.42 is triggered by @glenn-jocher via commit . At this point the actor release restriction is back in place, but has no effect as @glenn-jocher is the one making the release. Workflow run: https://github.com/ultralytics/ultralytics/actions/runs/12180037832/job/33973482495 . 2024-12-05 12:47: Sigstore’s transparency log records 153589716 and 153589717 indicating two attestations, one for each of the distributions of v8.3.42 that will appear on PyPI ( and ). 2024-12-05 12:47: v8.3.42 is uploaded to PyPI with a Trusted Publisher and valid attestations for each distribution. 2024-12-05 13:03: @renzhexigua announces on #18027 that v8.3.42 is still malicious. 2024-12-05 13:47:30: v8.3.42 is removed from PyPI (approx. 1 hour after introduction). 2024-12-05 15:17: @glenn-jocher announces on #18027 that the @openimbot identity is banned from interacting with Ultralytics . 2024-12-07 01:41:45: v8.3.45 is directly released to PyPI, with no CI/CD or repository activity from ultralytics/ultralytics . This was done with an API token and not a Trusted Publisher, and therefore has no attestations. This strongly suggests that the attacker either obtained a PyPI API token from the original secrets exfiltration phase or is in full control pypi/u/glenn-jocher . 2024-12-07 02:27:14: v8.3.46 is directly released to PyPI, with no CI/CD or repository activity from ultralytics/ultralytics . Like with v8.3.45, this release is done via API token and has no attestation. 2024-12-07 04:00: Adnan Khan announces on #18027 that v8.3.45 and v8.3.46 are still malcious, albiet with a different (simpler) miner payload. 2024-12-07 10:08:32: v8.3.45 is removed from PyPI (approx. 8 hours after introduction). 2024-12-07 10:09:08: v8.3.46 is removed from PyPI (approx. 7.5 hours after introduction). A total of 4 different releases of Ultralytics were malicious: v8.3.41, v8.3.42, v8.3.45, and v8.3.46. Of these releases, two were made via GitHub Actions and used Trusted Publishing + attestations, and two were made via an API token that the attacker likely either exfiltrated from a developer’s machine or from the GitHub Actions context (where it may have been forgotten about after Trusted Publishing was enabled). All evidence strongly suggests that the attacker was able to fully exfiltrate all of the configured repository secrets within ultralytics/ultralytics . As such, the Ultralytics maintains must consider these credentials compromised and revoke them immediately to avoid further compromise . The failure to immediately revoke a stale API token may have been what enabled the follow-on direct releases of v8.3.45 and v8.3.46. In particular, in addition to any PyPI API tokens, the attacker may have access to a custom GitHub PAT configured as in the Ultralytics CI. This must be considered compromised and revoked immediately. All evidence strongly suggests that the attacker was in full control of the @UltralyticsAssistant bot account, and may still have control over it . The basic vulnerability that enabled this attack has been present in since 2024-08-24 and was reintroduced after a previous disclosure. This strongly suggests a lack of adequate security controls and reviews within Ultralytics’ processes. The release has been deleted and can no longer be resolved, but the malicious distribution is still available on PyPI’s storage: (replace with ).  ↩

DevOps AI Python Shell

▲ 0

0 views

ENOSUCHBLOG 11 months ago

Security means securing people where they are

Standard disclaimer : These are my personal opinions, not the opinions of my employer, PyPI, or any open source I projects I participate in (either for funsies or because I’m paid to). In particular, nothing I write below can be interpreted to imply ( or imply the negation of) similar opinions by any of the above, except where explicitly stated. TL;DR: If you don’t bother to read the rest of the post, here is the gloss: being serious about security at scale means meeting users where they are . In practice, this means deciding how to divide a limited pool of engineering resources such that the largest demographic of users benefits from a security initiative. This results in a fundamental bias towards institutional and pre-existing services, since the average user belongs to these institutional services and does not personally particularly care about security. Participants in open source can and should work to counteract this institutional bias, but doing so as a matter of ideological purity undermines our shared security interests. I was sniped into writing encouraged to write this by Seth Larson , following voluminous public discourse about PEP 740 and its recently announced implementation on PyPI. Many people were concerned about decisions that went into the implementation of PEP 740 on PyPI, and expressed these these concerns in a wide variety of ways. A sampling of shpilkes, from “eminently reasonable” to “unhinged”: A sub-variant of this criticism is “intentional,” i.e. “attestations are intended to cause lock-in” versus “double-effect,” i.e. “there’s a risk of vendor dependence, but the goal itself is building out a new security feature for the ecosystem.” The former is in effect a way of accusing the people who did this work of having evil motives, while the latter is a reasonable expression that the feature didn’t sufficiently consider vendor dependency. These concerns range from containing reasonable (and concerning!) inferences to being nakedly factually incorrect. In the interest of establishing a factual baseline, here’s my list of priors: Trusted Publishing is not limited to GitHub . A persistent form of misinformation around PyPI’s support for attestations stems from misinformation about Trusted Publishing, as the layer beneath it. When Trusted Publishing was originally released on PyPI, it originally only supported GitHub. Other providers ( GitLab , Google Cloud , ActiveState   2 ) came a few months later, but are now fully supported as Trusted Publishing providers. The reason for this approach (GitHub first, then others) had nothing to do with a sinister Microsoft plot (as was insinuated then), but instead came from the exact same reasoning that will be elaborated in this post: the largest demographic that stood to immediately benefit from Trusted Publishing’s usability and security benefits was on GitHub, so they were targeted first. Trusted Publishing and PEP 740 are built on open standards . More precisely, both are built on top of OpenID Connect , which allows independent services to federate with each other via claims that are signed with public-key cryptography. This underlying technical choice is what made onboarding GitLab, &c., relatively easy: there was no vendor or otherwise closed dependency that needed to be removed or replaced. This remains true to this day. Adding a new Trusted Publisher and/or attestation source is not hard, but also not trivial. Adding a new Trusted Publishing provider is not as trivial as adding a well-known OIDC discovery URL to PyPI’s codebase: each new provider needs to be reviewed for claim contents, to ensure that the provider’s principals can be distinguished from each other in a way that PyPI can model. In other words: it would be catastrophic for PyPI to support an OIDC IdP that can’t distinguish between its users, or permitted claim malleability such that users could impersonate each other. Ensuring that each accepted IdP meets these conditions requires a nontrivial time commitment that gets balanced against the expected real-world usage of a given IdP: an IdP with one-to-few users is not worth the tradeoff in review time. Not everything makes sense as a Trusted Publisher/attestation provider . As a corollary to the point above: it doesn’t make sense (for either PyPI, or individual project maintainers) to attempt to do all package uploading via Trusted Publishing. OIDC fundamentally benefits from scale, and it doesn’t make sense (in terms of operational complexity 3 and diminished rewards 4 ) for every individual maintainer to run their own OIDC IdP. Neither Trusted Publishing nor PEP 740 increases trust in an already-used CI/CD provider. This one can be a little unintuitive, but it follows from existing workflows: if you were already using GitHub/GitLab/&c. to publish with a plain old API token, then you were already trusting your CI/CD provider to securely store that credential (and only use it when you want it used). In a broader sense, Trusted Publishing and PEP 740 reduce unnecessary trust in the CI/CD provider, since they force the provider to make a publicly auditable and verifiable claim in order to receive a temporary API token. This is the baseline, as I see it. Now let’s talk a bit about why PyPI’s initial attestations rollout focused on GitHub (like what happened with Trusted Publishing), and why it was (1) not a conspiracy, and (2) the strategic thing to do . I’ll then end with some thoughts on how we can better address the unfair social pressure case. And they shouldn’t have to care. This is the hard truth beneath everything else: most open source maintainers are not security experts (they’re experts in other things , like the projects they maintain), and they don’t want to become security experts. Security is a hump that people get over while attempting to achieve their actual goals. At the same time, expectations change over time: MFA was a relative rarity a decade ago, and is now mandatory across a wide swath of popular OSS-adjacent services (or mandatory for demographic subsets, such as “critical” package maintainers on NPM and RubyGems). This sets up a fundamental tension : most maintainers want to just keep doing whatever has always worked, while security is a moving target that sometimes requires universal change. There aren’t many ways to eliminate this tension, but there are (at least) two ways to ameliorate it: For the Python ecosystem, in 2024, that service is overwhelmingly GitHub. The history of open source on the public internet has long favored a small and stable (but not static), group of watering holes at which the overwhelming majority of projects concentrate. Past watering holes include SourceForge and Google Code , along with specialized project hosts like Savannah . Today, that watering hole is GitHub. Using last week’s dump : Of the 447,148 packages that have URLs 7 , a full 378,613 list in their metadata. That’s 84.7% of all projects that list URLs in their metadata. By contrast, here are the next 10 most popular hosts: The drop-off is stark: GitLab is #2, but with only 1.99% of all projects 8 . This tells an important baseline story: if PyPI builds a security feature that needs to interoperate with source forges or CI/CD providers, then the overwhelming majority of its packages can be best served by starting with GitHub. That doesn’t mean that PyPI should stop with just GitHub, or GitHub plus GitLab, or anything else of the sort. It just tells us where the starting point should be . This finally gets us to the point of this post: The conclusion: if a new feature needs to interact with services outside of PyPI itself, then the purely practical course to take is to start with the services that will yield the most immediate benefit to the Python community. A recurring strain of thought in conversations around PEP 740 (and centralized infrastructure more generally) is whether the ethics of open source impute a similar ethic 9 of independence and decentralization. Or in other words: does PyPI (or OSS more generally) have a responsibility to try and avoid corporate-associated integrations? I would argue no : PyPI’s primary responsibility is to the community that uses it, both upstream and downstream, and that community is best served by using open standards to interoperate with the services the community overwhelmingly uses . This does not however imply that PyPI should ignore smaller opportunities for integration, such as adding Trusted Publishing providers for independent GitLab hosts with large user bases, or Codeberg instances, or anything else. On the contrary: I would like to see PyPI integrate more of these as Trusted Publishing providers, provided that the usage statistics and operational complexity for each actually benefit the community as a whole. Enrolling a few thousand projects on a single self-hosted forge would be great; having to review dozens of forges with under a dozen users would not be. I would like to see a similar thing occur for attestations. In sum: PyPI shouldn’t (and doesn’t) pick winners, but it should (and does) pick battles to fight and the order in which it fights them . There’s a flip side to all of this: despite effusive attempts to emphasize that attestations are not a “trusted bit” and that consumers shouldn’t treat them as a signal of package quality or security, we are almost certainly going to see people (and companies 10 ) do exactly that. In practice, that means that maintainers who do everything right but not in a way that’s currently legible to the attestations feature are going to receive annoying emails, issues, &c. asking them why they’re “less secure” 11 than other packages. In the medium term, I think the way to address this is to: Those two, combined, should address the overwhelming majority of the remainder : people who can’t (or simply don’t want to) use Trusted Publishing, and those who do but can’t yet. I’ll be working on those. At this point, I consider “let’s bring back PGP” to be an unserious contribution to the conversation. The rest of the post assumes that something resembling identity-based signing is going to be way forwards.  ↩ …and almost certainly more to come in the future.  ↩ Maintaining an OIDC IdP is a PKI-shaped problem: to be done responsibly, it requires offline keys, on-call staff and rotation schedules, as well as incident response capacity and the ability to run fire drills to keep operational processes (such as emergency key rotations) from bitrotting. It would be horrifically onerous to impose these requirements on every single unpaid open source maintainer.  ↩ At small scales, there’s virtually no difference between Trusted Publishing and plain old API tokens: running an OIDC IdP means maintaining (and rotating) a different but equally valuable set of credentials. The numbers only come out in Trusted Publishing’s favor at large scales: 100, 1000, or more independent projects per IdP.  ↩ Where “you” is my security brethren, even the misguided PGP ones.  ↩ From personal experience, and from talking to literally every other maintainer I know .  ↩ This excludes packages with no URLs in their metadata at all, which is about 21% of all packages (based on the dump listing 566,404 packages total). A large part of this is probably packages that predate free-form URLs in the standard metadata, but even at a fractional linear extrapolation GitHub-hosted projects remain the overwhelming majority.  ↩ Which, to be clear, is still a significant number, and a strong datapoint for GitLab being a Trusted Publishing provider!  ↩ Which exact ethic is hard to pin down: there’s a lot of general bellyaching whenever PyPI (or any other packaging ecosystem) interoperates with anything that has a company behind it (even if via an open standard), but it’s hard to infer any particular or consistent ideology beneath that.  ↩ And especially “supply chain” companies, which continue to demonstrate a marked lack of shame around spamming the Python community.  ↩ They’re not.  ↩ PyPI’s sourcing of attestations from large IdPs like GitHub will result in unfair social pressure on projects that do everything right but on their own infrastructure , which includes major OSS projects that run their own Jenkins, private CI/CD, &c. PyPI’s decision to enable GitHub-based attestations before others is effectively a form of vendor bias , and encourages the OSS community to deepen its dependency on GitHub. A sub-variant of this criticism is “intentional,” i.e. “attestations are intended to cause lock-in” versus “double-effect,” i.e. “there’s a risk of vendor dependence, but the goal itself is building out a new security feature for the ecosystem.” The former is in effect a way of accusing the people who did this work of having evil motives, while the latter is a reasonable expression that the feature didn’t sufficiently consider vendor dependency. Attestations are just plain bad™ and PyPI should go back to (weakly) tolerating long-lived PGP signing keys since, despite all evidence to the contrary , people swear that these signatures are being verified and form a security boundary somewhere 1 . PyPI has been captured by the Micro$oft/NSA/Unit 8200 and has developed attestations to complete this year’s . Trusted Publishing is not limited to GitHub . A persistent form of misinformation around PyPI’s support for attestations stems from misinformation about Trusted Publishing, as the layer beneath it. When Trusted Publishing was originally released on PyPI, it originally only supported GitHub. Other providers ( GitLab , Google Cloud , ActiveState   2 ) came a few months later, but are now fully supported as Trusted Publishing providers. The reason for this approach (GitHub first, then others) had nothing to do with a sinister Microsoft plot (as was insinuated then), but instead came from the exact same reasoning that will be elaborated in this post: the largest demographic that stood to immediately benefit from Trusted Publishing’s usability and security benefits was on GitHub, so they were targeted first. Trusted Publishing and PEP 740 are built on open standards . More precisely, both are built on top of OpenID Connect , which allows independent services to federate with each other via claims that are signed with public-key cryptography. This underlying technical choice is what made onboarding GitLab, &c., relatively easy: there was no vendor or otherwise closed dependency that needed to be removed or replaced. This remains true to this day. Adding a new Trusted Publisher and/or attestation source is not hard, but also not trivial. Adding a new Trusted Publishing provider is not as trivial as adding a well-known OIDC discovery URL to PyPI’s codebase: each new provider needs to be reviewed for claim contents, to ensure that the provider’s principals can be distinguished from each other in a way that PyPI can model. In other words: it would be catastrophic for PyPI to support an OIDC IdP that can’t distinguish between its users, or permitted claim malleability such that users could impersonate each other. Ensuring that each accepted IdP meets these conditions requires a nontrivial time commitment that gets balanced against the expected real-world usage of a given IdP: an IdP with one-to-few users is not worth the tradeoff in review time. Not everything makes sense as a Trusted Publisher/attestation provider . As a corollary to the point above: it doesn’t make sense (for either PyPI, or individual project maintainers) to attempt to do all package uploading via Trusted Publishing. OIDC fundamentally benefits from scale, and it doesn’t make sense (in terms of operational complexity 3 and diminished rewards 4 ) for every individual maintainer to run their own OIDC IdP. Neither Trusted Publishing nor PEP 740 increases trust in an already-used CI/CD provider. This one can be a little unintuitive, but it follows from existing workflows: if you were already using GitHub/GitLab/&c. to publish with a plain old API token, then you were already trusting your CI/CD provider to securely store that credential (and only use it when you want it used). In a broader sense, Trusted Publishing and PEP 740 reduce unnecessary trust in the CI/CD provider, since they force the provider to make a publicly auditable and verifiable claim in order to receive a temporary API token. Make security features into usability features. This was one of the core objectives behind Trusted Publishing’s design: users found the experience of context-switching between PyPI and their CI/CD frustrating, so we found a way to eliminate those context switches while improving the security of the credentials involved. Delegate some (if not all) responsibility for security to services. The reasoning behind this is intuitive: big services have both the staff and the financial incentive to maintain a strong default security posture, as well as keep up with the latest changes in baseline security expectations. This, too, has a usability angle: it’s just plain easier 6 to maintain a project when an external service hums along and provides source control, CI/CD, release management, &c. for you. Most maintainers (reasonably!) don’t especially care about security and, as a corollary, have selected infrastructure and services that compartmentalize most of the boring, operational aspects of open source security (like maintaining a set of trusted committers and a secure CI/CD); GitHub is overwhelmingly the target of that selection process. Support email identities for attestations, since PyPI already has a notion of “verified” email to cross-check attestations against. Continue to widen the number of Trusted Publishing providers and enable attestation support for each, within reason. At this point, I consider “let’s bring back PGP” to be an unserious contribution to the conversation. The rest of the post assumes that something resembling identity-based signing is going to be way forwards.  ↩ …and almost certainly more to come in the future.  ↩ Maintaining an OIDC IdP is a PKI-shaped problem: to be done responsibly, it requires offline keys, on-call staff and rotation schedules, as well as incident response capacity and the ability to run fire drills to keep operational processes (such as emergency key rotations) from bitrotting. It would be horrifically onerous to impose these requirements on every single unpaid open source maintainer.  ↩ At small scales, there’s virtually no difference between Trusted Publishing and plain old API tokens: running an OIDC IdP means maintaining (and rotating) a different but equally valuable set of credentials. The numbers only come out in Trusted Publishing’s favor at large scales: 100, 1000, or more independent projects per IdP.  ↩ Where “you” is my security brethren, even the misguided PGP ones.  ↩ From personal experience, and from talking to literally every other maintainer I know .  ↩ This excludes packages with no URLs in their metadata at all, which is about 21% of all packages (based on the dump listing 566,404 packages total). A large part of this is probably packages that predate free-form URLs in the standard metadata, but even at a fractional linear extrapolation GitHub-hosted projects remain the overwhelming majority.  ↩ Which, to be clear, is still a significant number, and a strong datapoint for GitLab being a Trusted Publishing provider!  ↩ Which exact ethic is hard to pin down: there’s a lot of general bellyaching whenever PyPI (or any other packaging ecosystem) interoperates with anything that has a company behind it (even if via an open standard), but it’s hard to infer any particular or consistent ideology beneath that.  ↩ And especially “supply chain” companies, which continue to demonstrate a marked lack of shame around spamming the Python community.  ↩ They’re not.  ↩

Python Open Source Security

▲ 0

0 views