tekin.co.uk 2 weeks ago

10 Things You Might Not Know About Rails i18n

This is mostly a transcript of my recent lightning talk on i18n in Rails , go watch that if you prefer your content in video form. Otherwise read on to find out why I believe understanding i18n in Rails can be useful to you, a Rails developer, even if you’re not translating your applications into other languages. I’m going to share ten things I think are interesting in i18n, and explain how we can take advantage of them in our single language app code. But first a brief primer on i18n… i18n is shorthand for Internationalisation When we’re talking about i18n, it’s useful to understand the difference between Internationalisation and Localisation : The take away here is that although i18n is primarily a framework for adapting applications to support other languages, the core idea of separating content from applications code has useful properties that we can take advantage of it to improve our Rails applications, even if they only support a single language. Rails ships with the i18n gem , and there are two main components: Rails aliases these helpers in views and helper modules to the convenient shorthands of and respectively. So the most basic usage of i18n would look something like this: Primer over! On to the ten things… Even if you’re not localising your app you’re almost certainly still using i18n. That’s because it’s baked into the very fabric of the framework. For example Active Record uses i18n under the hood to generate error messages for its validations. It does so using a YAML file that ship with Active Model : Whenever you see something like “Title cannot be blank”, this is the i18n plumbing that was used to construct it. Rails i18n makes it easy to override these default errors messages. Just replicate the same key structure in your application’s locale file with your chosen phrasing: Rails will check your application’s locale file first, falling back to the defaults only if the message isn’t defined there. As well as overriding this global phrasing, it’s also possible to override the error messages for a given attribute name: Or you can get even more granular and override specific attributes on a given model. For example if we have the following model and validation: It’s possible to set a bespoke error message for this specific model and attribute: As well as overriding individual validation messages globally and for specific attributes, it’s also possible to change the actual format of error messages. By default, errors are constructed with the familiar format of . This is defined in the same YAML that ships with Active Model alongside the default error messages: This format works as a reasonable default to ship with the framework as it gives us mostly coherent error messages out of the box without us having to do anything, but it lacks fidelity and can result in some clunky and unfriendly sounding messages. For example, take this validation: If we go with the default format we end up with . Which is fine, but we can definitely do better. My personal preference on the apps I work with is to remove the attribute prefix entirely: This frees me from having to make my error messages work with attribute’s name upfront, allowing me to write more user-friendly and coherent error messages: Now changing the default format does come at a cost: without the attribute name prefix, the default messages that ship with Rails no longer form complete sentences. This forces me to write bespoke error messages for each validation I add to my app: Personally I’m happy to pay this cost for the benefit of improving the user experience. So whilst you may not want to do this on an existing app with thousands of error messages to backfill, it might be something you consider next time you run and are starting from a clean slate. Update : There’s actually a fairly neat way to avoid this cost, check out my follow-up post on how . As well using i18n to manage and override the error messages for the built-in validations, you can also use it to manage the messages for your custom validations. Let’s look at a custom model validation: Here the error message is inlined as part of the validation code. Instead of doing this, we can pass in a symbol identifying the error, and define the corresponding message in the locale file using the same key structure as the built-in validations: This results in more concise and compact model code, but also equally as nice is that it removes a presentational concern — the specific phrasing of the message — our the business logic. It’s also possible to interpolate both the attribute name and the value of the attribute directly into the error message, which is useful for custom validations that are used across multiple attributes and/or models: Another place where Rails makes use of i18n under the hood is the form helpers: specifically, the label helper methods. By default the label helper methods humanize the attribute name to arrive at the label text: One way to override the label text is to pass in a string to the helper call like so: I think a better way is to define label overrides in the application’s locale file: Not only does this keep the view code more concise and less noisy, but it has the added advantage of defining this label override in one place: now any other forms that display the same label will automatically get the overridden text without you having to manually copy the literal string to every template. And as with form errors, label overrides can be defined per-model as well as globally: i18n has sophisticated support for pluralisation that we can leverage to simplify our app code. As an example, we can display the stock availability of a product by calling out to i18n like so: And then define the phrasing based on the number of items in stock in our locale file: Much neater than a bunch of conditional logic in a helper method! As well as being available in Rails controllers and views, i18n is also integrated into Action Mailer, allowing you to organise content for mailer templates in local files. Mailer get one extra feature of controllers: the ability to specify email subject lines in locale files: Again, this enables us to move another presentational concern out of our application code and put it somewhere more appropriate. There is also a meta benefit to defining all these bits of content (mailer subject lines, form labels, validation error messages) in an application’s locale file: it puts them all in one centralised place, making them easier to change, check for consistency and generally manage. It also makes the content more accessible to non-developers such as designers and product folks. Right at the top I mentioned the method that is used to output localised times and dates. The way this works is by defining our own time and date formats in our locale file using the same format specification as Ruby’s : Here we’ve defined a short date format that will be familiar to my UK-based audience: day/month/year. To render a date using this format we simply pass in the format identifier to (or its alias): or with the shorthand: Now my American readers may be confused by this short format, as across the pond the preferred way to display dates is actually month/day/year. So whilst you may not need to localise your entire application, you might want to consider localising your date rendering, especially if you have customers both sides of the Atlantic. To do this you would define a US region specific English locale file: Then update your controller code to automatically switch locale so the user gets dates formatted appropriately for them. There are many ways to do actually do the locale switching, but at a high-level, you identify the user’s specific locale and then make sure your controller actions are executed with that locale. Here’s an example of doing that by sniffing the user’s locale from their browser headers: (The exact mechanics of the method is left as a separate exercise, but you could do worse than use the http_accept_language library) Little known fact: the i18n library supports locale files defined using Ruby! Now why would you do this, other than a burning hatred for YAML? Well one reason is that Ruby-based locale files allows you to make use of procs to dynamically generate content. Below is an excerpt from a Ruby-based locale file from one of my applications. It uses a proc to format dates as academic years, changing the output depending on which side of September the date is: Now this could just as easily be achieved using a standard Rails helper method. But for me I like the consistency and clarity of using a single mechanism ( ) for formatting dates across my application code, rather than with a mishmash of calls to bespoke helpers, , , etc. This is the part of the lightning talk where I make a meta point about what I’ve covered. I won’t write that up here, but if you’re interested go watch the video from minute 9. Internationalisation — the process of abstracting content and other locale-specific things away from application code itself Localisation — the process of adapting software to support different languages and regions (maybe using i18n) locale files where textual content is organised and stored per-language (normally as YAML) two helper methods: for outputting text content from the locale files for localising dates and times

Yaml Rails Ruby Tutorial

▲ 0

0 views

Simon Willison 1 months ago

Have your agent record video demos of its work with shot-scraper video

shot-scraper video is a new command introduced in today's shot-scraper 1.10 release which accepts a file defining a routine to run against a web application and uses Playwright to record a video of that routine. I've written before about the importance of having coding agents produce demos of their work; this is my latest attempt at enabling them to do that. Here's an example video created using , exercising a still in development feature adding the ability to create new tables in Datasette from pasted CSV, TSV or JSON data: That video was created by running this command : (That JSON file contains a cookie , as described here in the documentation.) Here's the file: The video command documentation includes simpler examples, but for the purpose of this post I thought I'd go with something more comprehensive. That demo YAML storyboard was constructed entirely by GPT-5.5 xhigh running in Codex Desktop, using the following prompt run inside my checkout of this branch : Now that I've released the feature the prompt could say " " instead and it should achieve the same result. I really like this pattern where the output for a command provides enough detail that a coding agent can use it - it works kind of like bundling a file directly inside the tool. I used the same pattern for showboat and rodney . started as an experimental prototype. is built on top of Playwright , and the key feature it needed was for Playwright to be able to record video of browser sessions with enough control to create the desired demo. I first tried this a few years ago and found that the Playwright-produced videos included additional chrome that was useful for debugging a test failure but unwanted for a product demo. They fixed that a while ago, but there were still some minor blockers. In particular I was getting a few white frames at the start of the videos , since the recording mechanism kicked in before the first URL was loaded by the browser. Playwright 1.59 added a new screencast mechanism providing much more finely grained control over video recording. This was very nearly what I needed, but the resulting videos were fixed at 800px wide. I found a landed PR fixing that but it wasn't yet in a release. Then yesterday they shipped it in playwright-python 1.61.0 and I was finally unblocked to finish implementing the feature! The code itself was all written by GPT-5.5 xhigh in Codex Desktop. I had it write the documentation as well which gave me a very useful frame for reviewing the design - much of the iteration on the feature came from reviewing that documentation, spotting things that were redundant, inconsistent or confusing, and requesting (or dictating) a better design. The YAML format itself was mostly defined by the coding agent. I had it use Pydantic to both define and validate the format, partly to make the design easier to review. This is a great example of the kind of feature that I almost certainly wouldn't have taken on without coding agent support. I filed the original issue in February 2024, and had difficulty finding the necessary time to solve this in amongst all of my other projects. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options .

Yaml Web Development JSON Python

▲ 0

0 views

Jack Vanlightly 2 months ago

Kafka Share Groups and Parallelizing Consumption - Part 2: Producer Batches and share.acquire.mode

All tests were executed against Kafka 4.3.0 using Dimster .  In the last post we used simulated consumer processing time to reveal how important it is to set an appropriate value for to ensure the consumer parallelism that we expect. With a uniform distribution of messages over partitions, the rule of thumb was a value somewhat lower than: But there’s more to parallel consumption than . The size of producer batches also plays a role when using the default ( ). Share group members are assigned to partitions like consumer group members are, except that share group assignment allows multiple consumers to be assigned to the same partition. If the number of share consumers is less than the partition count, then each consumer will be assigned multiple partitions. If the consumer count matches or exceeds the partition count, then each consumer will be assigned one partition. Fig 1. Share consumer assignments. Left: consumer count < partition count. Right: consumer count > partition count. When a consumer is assigned only one partition, it will always be fetching from one broker. If a consumer is assigned multiple partitions, it may fetch from multiple brokers concurrently. There are two values for : The Javadoc says the following: The application chooses between the two modes using the consumer share.acquire.mode configuration property. If the application sets the property to batch_optimized or does not set it at all, the share consumer fetches records based on batch boundaries which may mean that the number of records returned may exceed the max.poll.records configuration property. The share consumer may also prefetch records and buffer them temporarily awaiting the application's next call to poll(Duration). If the application sets the property to record_limit, the share consumer fetches no more than records at a time and does not prefetch. This is slower but gives the application tighter control on how many records are fetched and when the acquisition locks begin. So why two modes?  It comes down to efficiency ( ) and consumer control ( ). First of all the sentence “ the share consumer fetches records based on batch boundaries” is correct but a little misleading. No matter what mode is used, whole batches are returned to the consumer over the wire . In other words, the data sent over the network is always based on batch boundaries as the record batch is the unit of data delivery.  What that sentence refers to is what records are acquired by the consumer and returned to the application: With , the config is a soft cap. The consumer acquires any batches (in their entirety) that are covered by the offset range determined by . These acquired batches are returned to the consumer, and the consumer returns the records of those batches to the calling application (that invoked ). With , the config is a strict cap. The consumer only acquires the records that are covered by the offset range determined by (though less if less records are available). However, the unit of data delivery is the record batch, so the consumer receives whole batches but only returns a specific offset range to the calling application. For example, in the figure below we have three consumers sending fetch requests with and . Fig 2. Three consumers fetching with batch_optimized Despite asking for only one record, each consumer acquires and receives records along batch boundaries. The result of consumer.poll(Duration) for c1 is three records, not one. If we rerun this scenario with record_limit: c1 acquires record 0 c2 acquires record 1 c3 acquires record 2 However, the batch is the unit of data delivery, so batch 1 is sent in its entirety to each consumer (the consumer internals only returns the acquired records of the batch to the application). Fig 3. Three consumers fetching with record_limit This is obviously less efficient… We just sent the same batch three times! Nonetheless, exists because sometimes that inefficiency over the wire is countered by other concerns (one of which is covered in this post). Another efficiency gain that has is that because each batch is only sent to one consumer, Kafka only needs to do share group housekeeping of the batch as a whole, not each record individually. This reduces CPU and makes metadata more compact. If we get mixed acknowledgments of the batch records (2 success, 1 reject) only then does the record tracking explode the metadata to be per-record. With , the housekeeping always tracks state per record, which is more expensive. The final difference between the modes is that in mode, a consumer can send concurrent fetches to all the brokers of its assigned partitions. This further increases the number of records that a consumer might receive as is a soft cap per broker. With , the consumer sends one fetch at a time, round-robin between the brokers of its partitions. This difference only manifests when the consumer count is less than the partition count. We’ll cover this aspect more in the next post. The main implications are that: With , the effective consumer parallelism can be impacted by the average number of records per record batch. With , the network throughput will increase in most scenarios as offset ranges are unlikely to align with batch boundaries. If the is larger than the average number of records per batch, then each batch may only be delivered twice. The network throughput can a lot if the is much smaller than the average number of records per record batch. Don’t worry if that isn’t clear yet, we’ll gather some empirical results next which should make it clearer. Let’s test this out with Dimster’s interactive mode, using the same workload as the last post. In the last post, we calculated that the maximum theoretical consumption rate for 300 consumers with a processing time of 5 ms per message would be 60,000 msg/s. By setting to 30 we reached 55,000 msg/s and then finally reached 60,000 with low end-to-end latency by adding an additional 12 consumers (2 per partition). So we use the following workload file (no dimensional stuff in this one as we’re going to use live-interaction): In this test we’re going to make the record batches bigger and see what happens to the consumption rate. First we start Dimster and ensure it’s handling the 60k msg/s. Once it has started and settled in, we see it’s coping well. If I look at the metrics, the current record batch size is around 5KB with 10 records per batch. The average fetch size is 7KB with 14 records. This means some consumers get 1 record batch per fetch and some get 2 record batches per fetch. Let’s increase the batch size. To do this we’ll drop to 1 producer, and set the linger.ms to 10 to reach the default batch.size of 16KB batches. We see that the batch size has risen to the default of 16KB, or 32 records per batch. The consumers should now, on average, receive 32 records per fetch (2 above the max.poll.records). Fig 4. The record batches sent by the producers increase from 5.5 KB to 16 KB The coordinator output shows that the consumers are still coping, as expected. With 500b records, the number of records returned per fetch will be 32 which is close enough to the max.poll.records of 30 to not impact consumption. Now let’s double batch.size to 32786. From a separate terminal window to the coordinator output, we’ll run the following: We see the batch size increase again in the dashboard. Fig 5. The record batches sent by the producers increase from 5.5 KB to 16 KB to 32 KB The coordinator output shows that the consumers are no longer keeping up! Only managing 37K msg/s with a fast growing backlog. The problem is that each partition has an inflight budget of 2000 records and each record batch contains 64 records. That allows up to 31 effective consumers per partition (2000 / 64), leaving 21 consumers starved at any point in time. This explains the 37K msgs/s: We can fix this problem in three ways: Set in the producer. Increase to create a larger inflight budget We already know the default 16KB batch size is ok. Let’s first increase the inflight budget. We’ll double the budget and see what happens. First we’ll stop the producers and remove the processing time on the consumers to drain the backlog. Next we need to update the broker config and restart the brokers. In we add: Then we’ll redeploy Kafka (again from a separate terminal window). Now we’ll start the producers again and apply the 5 ms processing time to the consumers. We’re in business! The consumers are now coping with the larger batch sizes with this increased inflight budget. This time we’ll try . First let’s walk back that inflight budget change by  1) stopping the producers, 2) commenting out the added line to our broker config, 3) redeploying Kafka. While the producers are still stopped, I’ll change the consumers to use : Then start the producers again: In the coordinator, we see that the consumers are now coping with the 60K msg/s. The reason that allows the consumers to keep up, despite the larger record batches, is that each consumer is only allocated a max of 30 records per fetch, even though each batch contains 64 records. However, each batch is now being delivered three times as 30 doesn’t align well with 64. We can see this in the Kafka client metrics. Fig 6. On the left, with the larger inflight budget and batch_optimized. The middle was when we stopped the producers to restart Kafka with the original inflight budget. The right is with record_limit and each batch being sent three times. We could make this more efficient if we increase to 32 to align with the 64 record batches. If I simply change the to 32, we don’t see much of an improvement as most offset ranges of 32 records will touch two batches. But if we stop the producers, ensure there is no backlog at all then set , the fetches will be perfectly aligned. Fig 6. On the left, with unaligned fetches with max.poll.records=32 (each batch delivered 3 times). Right: aligned fetches with max.poll.records=32 (each batch delivered 2 times). Let’s not over-index on this one case. The purpose of this post was to explain the underlying mechanics and back that up with some empirical benchmarks, sticking with the same workload example as the last post. What we’ve learned: Consumer parallelism is impacted by more than just consumer count and . It is also impacted by: Record batch sizes (determined by the producers) The inflight budget ( ) The share consumer config Record acquisition is along batch boundaries with , and record ranges with . Record batches are the unit of delivery, so can cause consumer network bandwidth to increase because fetches likely will not align on batch boundaries causing batches to be delivered at least twice (more if is much smaller than the average number of records per batch). In the next post we’re going to look a bit closer at . ps: you can run this whole scenario with two terminal windows: Window 1 - kick off the benchmark (using the workload yaml described in the post) Window 2 - wait a few minutes then run the following bash script: Happy testing! If the application sets the property to batch_optimized or does not set it at all, the share consumer fetches records based on batch boundaries which may mean that the number of records returned may exceed the max.poll.records configuration property. The share consumer may also prefetch records and buffer them temporarily awaiting the application's next call to poll(Duration). If the application sets the property to record_limit, the share consumer fetches no more than records at a time and does not prefetch. This is slower but gives the application tighter control on how many records are fetched and when the acquisition locks begin. With , the config is a soft cap. The consumer acquires any batches (in their entirety) that are covered by the offset range determined by . These acquired batches are returned to the consumer, and the consumer returns the records of those batches to the calling application (that invoked ). With , the config is a strict cap. The consumer only acquires the records that are covered by the offset range determined by (though less if less records are available). However, the unit of data delivery is the record batch, so the consumer receives whole batches but only returns a specific offset range to the calling application. c1 acquires record 0 c2 acquires record 1 c3 acquires record 2 With , the effective consumer parallelism can be impacted by the average number of records per record batch. With , the network throughput will increase in most scenarios as offset ranges are unlikely to align with batch boundaries. If the is larger than the average number of records per batch, then each batch may only be delivered twice. The network throughput can a lot if the is much smaller than the average number of records per record batch. Don’t worry if that isn’t clear yet, we’ll gather some empirical results next which should make it clearer. Set in the producer. Increase to create a larger inflight budget Consumer parallelism is impacted by more than just consumer count and . It is also impacted by: Record batch sizes (determined by the producers) The inflight budget ( ) The share consumer config Record acquisition is along batch boundaries with , and record ranges with . Record batches are the unit of delivery, so can cause consumer network bandwidth to increase because fetches likely will not align on batch boundaries causing batches to be delivered at least twice (more if is much smaller than the average number of records per batch).

Yaml Bash Performance Backend

▲ 0

0 views

Jack Vanlightly 2 months ago

Benchmarking Apache Kafka Consumer Groups vs Share Groups (overhead test)

In my last blog post I introduced Dimster (DIMensional teSTER), a performance benchmarking tool for Apache Kafka with a specific set of philosophies. In this first share group benchmarking post, we’re going to use share groups as they are not intended to be used, but for a good reason. Share groups allow you to move past partitions as the unit of parallelism by allowing multiple consumers to read from the same partition, using message queue semantics. We’ll run those kinds of tests in the next post. In this post I just want to understand if the mechanics of how share groups work add any additional overhead compared to consumer groups. So we’ll use share groups as if they were consumer groups (by capping consumer count to partition count). Objective : Use synthetic tests to measure the overhead of share groups compared to consumer groups in identical conditions. How : Like-for-like tests which use an identical workload/topology using consumerType (CONSUMER_GROUP|SHARE_GROUP) as a dimension. Given identical producer/consumer counts, producer rate, topic/partition counts, do share groups scale as well as consumer groups? Do they add any latency overhead? These benchmarks are educational , they are not hard numbers, they are not some kind of canonical result (in fact, no such benchmark exists). And again, this is not a realistic test at all, they only serve to understand share group overhead. I ran all these benchmarks on a k3d Kubernetes cluster on my Threadripper 9980X: 64 cores (128 threads) 256 GB DDR5 memory Two Samsung 9100 PRO 8 TB (with one dedicated to the benchmarks) Pretty decent CPU and RAM cooling.  This is not a production setup, but the hardware is more than capable of handling a small to medium sized Kafka cluster with excellent performance. The SSD can sustain around 1.7 GB/s once the SLC cache has filled up and none of these benchmarks exceed that in aggregate across the 3 brokers. All tests were run with TLS between the clients and brokers and between each broker. I prefer to run benchmarks with TLS enabled (though it reduces the numbers) because most people (hopefully?) run Kafka with full TLS.  Dimster uses named environments located in the dimster-config.yaml . Each environment targets a specific k8s cluster (via kubectl context), specifies the Kafka and client versions, sizes the Kafka pods, determines heap sizes, broker and log config files etc, all in one yaml block. This environment uses 36 of 128 CPU threads (16 of 64 cores) and 72 GB of 256 GB of RAM of my workstation, so we’re not pushing the Threadripper too hard. Note, the ‘requests’ field block is applied to both k8s requests and limits. The client pod is over-provisioned with 12 CPU cores (24 threads) and 24 GB RAM to avoid any client bottlenecks causing spurious results. The tests in this post compare consumer groups with share groups. To do that, I tried to isolate other factors as much as possible. Random load skew is one such important factor.  In these tests, I ensured that load was as even as possible over the brokers: Message distribution over the partitions of a given topic was even. I used the Dimster message distributor PINNED_PARTITIONS which ensures the number of producers is divisible by the number of brokers and pins each producer to a set of partitions, and each producer round-robin sends to its partitions directly. Multi-topic tests used a topic count divisible by the number of brokers to ensure even distribution of leaders over brokers. Consumer counts per group were divisible by the number of brokers to ensure even distribution of partitions over consumers. Fig 1. Dimster’s partition pinning for even load distribution This is not like in real-life, but for this post I want to avoid the randomness involved with partition and broker skew so that we can compare consumer group vs share group performance without load skew randomness playing a role. I’ll be writing about and running benchmarks with partition and broker skew in a future post. Link to results as a tarball For the throughput benchmarks, I used Dimster’s explore mode, which probes the cluster to find the highest sustainable throughput while staying under a target end-to-end latency in ms and percentile (50 ms, p75 in this case). It measures e2e latency per-partition and uses the latency of the poorest performing partition as the yardstick.  Explore mode runs in phases: Ramp . Start with a low throughput and keep doubling the throughput after a configured interval. Once the e2e latency exceeds the limit, move to the next phase. Search : Perform a binary search within the bounds of [0 - max-ramp-throughput ]. It starts at the midpoint and if it can sustain that throughput, it searches the high range starting at the midpoint. If it can’t sustain it, then it searches the low range. It recursively performs the search until the current search range size is < 5% of the throughput. Then it moves to the sustain phase. Sustain : The throughput identified by the search phase is maintained for a prolonged period. If it passes, the test is complete. If it fails to sustain (under the target e2e latency), it goes back to the search phase, with the failed sustain throughput as the new upper bound of the search range. The sustain phase is successful if 80% of the intervals (30 intervals of 10 seconds by default) meet the latency criteria. This rule exists as explore mode is trying to find the highest sustainable throughput which sits on the edge of the cluster’s limit, allowing for some latency spikes. I ran explore mode on the following workload: The first scenario has 4 test points which co-varies 4 workload aspects related to partition, client counts and consumer type as dimensions, repeating the tests 3 times. Fig 2. The merged result of three repeats (only small variance between runs) We see that share groups matched or even exceeded consumer group performance. Moreover, this pattern was broadly the same across the three test repeats. We can’t infer this as a generalizable result based on this one test, but my general observation, having been running these tests for a few weeks, on EKS clusters, my Threadripper and my Mac, is that throughput in this kind of synthetic test is comparable (between consumer/share groups). Scenario 2 - Varying fanout This scenario involved 1 topic with 12 partitions with a fanout of 2 and then 6. Fig 3. The merged result of three repeats (only small variance between runs) The surprising result was that share groups maintained a higher sustainable throughput with a fanout of 6. Explore mode is sensitive to spiky latency, and one thing I’ve observed is that share group latency can be more stable under stressful loads than consumer groups. Again, this may not be generalizable, but it shows that share groups might actually outperform consumer groups in some cases. I think the main takeaway from these limited tests is that share groups and consumer groups are in the same ball  park in terms of raw throughput. Link to results as a tarball The throughput benchmarks were a stress test of sorts, pushing Kafka right up to its limit. CPU was maxed out. We don’t want that for the latency benchmarks. We’re not going to push the Kafka cluster to the limit as we want to measure latencies within the performance envelope. With 4 vCPUs, around 100 clients and TLS, a 15 MB/s (1.3 TB daily) workload fits comfortably inside that envelope. I used run-mode , which are the standard fixed throughput benchmarks (best for measuring latency). I ran a single test campaign with 3 scenarios where consumerType was the dimension: 1 topic with 60 partitions, 30 producers, 60 consumers. 12 topics with 6 partitions, 6 consumers per topic, 3 producers per topic. 6 topics with 6 partitions, 3 consumer groups per topic with 6 consumers each, 3 producers per topic. All ran with an aggregate producer rate of 15000 msg/s with a 1 KB message size (15 MB/s). Fig 4. End-to-end latency (p99) over time (10 second intervals). Note: you can select a time range on Dimster charts to zoom into a sub-range. Under this lighter load, we see that share groups add some overhead, with the e2e p99 latency being a little more choppy than the much flatter consumer group latency. Fig 5. End-to-end latency distribution. Note: you can select a percentile range on Dimster charts to zoom into a sub-range. Fig 6. p99 end-to-end latency over time (10 second intervals) The sharegroup overhead is more pronounced in this test. Fig 7. End-to-end latency distribution. Fig 8. p99 end-to-end latency over time (10 second intervals) Again we see the same overhead. The takeaway is that for an adequately sized cluster that is not stressed by the workload, we can expect to see some small share group end-to-end latency overhead. Just to show you this isn’t an artifact of running these tests on k3d on a single workstation, we see the same pattern on a 50 MB/s test I ran a few weeks ago on AWS EKS with the m6i.2xlarge instance (8 vCPU, 32 GB RAM, EBS). Fig 9. 50 MB/s test, p99 end-to-end latency over time (10 second intervals) on an EKS cluster And a 150 MB/s test which was more stressful Fig 10. 150 MB/s test, p99 end-to-end latency over time (10 second intervals) on an EKS cluster We see the typical Kafka latency spikes related to log flushing and rotation (which has this predictable cadence due to how all load starts at the same time, at a constant rate, on one topic). The share group tests consistently used more CPU than the consumer group tests, which is understandable given share groups do a lot more accounting and state management than consumer groups. For example, the first repeat of scenario 1 of the latency test (executed as test points CG, SG, CG, SG, CG, SG): Fig 11. CPU over three apache/kafka pods In all these tests, consumers did nothing with the messages except record some metrics. In the real world consumers write to databases and call APIs. It might take anywhere from < 1 ms to 30+ seconds to process a message. More useful benchmarks simulate consumer processing time which is exactly what we’ll do in the next post. When we add processing time, we start to see where share groups really shine. To summarize some findings from this post: Share groups add a little overhead which might show up in a latency benchmark. Share groups consume more CPU. Raw throughput benchmarks will probably see varied results, but share groups are not fundamentally slower than consumer groups. 64 cores (128 threads) 256 GB DDR5 memory Two Samsung 9100 PRO 8 TB (with one dedicated to the benchmarks) Pretty decent CPU and RAM cooling.  Message distribution over the partitions of a given topic was even. I used the Dimster message distributor PINNED_PARTITIONS which ensures the number of producers is divisible by the number of brokers and pins each producer to a set of partitions, and each producer round-robin sends to its partitions directly. Multi-topic tests used a topic count divisible by the number of brokers to ensure even distribution of leaders over brokers. Consumer counts per group were divisible by the number of brokers to ensure even distribution of partitions over consumers. Ramp . Start with a low throughput and keep doubling the throughput after a configured interval. Once the e2e latency exceeds the limit, move to the next phase. Search : Perform a binary search within the bounds of [0 - max-ramp-throughput ]. It starts at the midpoint and if it can sustain that throughput, it searches the high range starting at the midpoint. If it can’t sustain it, then it searches the low range. It recursively performs the search until the current search range size is < 5% of the throughput. Then it moves to the sustain phase. Sustain : The throughput identified by the search phase is maintained for a prolonged period. If it passes, the test is complete. If it fails to sustain (under the target e2e latency), it goes back to the search phase, with the failed sustain throughput as the new upper bound of the search range. 1 topic with 60 partitions, 30 producers, 60 consumers. 12 topics with 6 partitions, 6 consumers per topic, 3 producers per topic. 6 topics with 6 partitions, 3 consumer groups per topic with 6 consumers each, 3 producers per topic. Share groups add a little overhead which might show up in a latency benchmark. Share groups consume more CPU. Raw throughput benchmarks will probably see varied results, but share groups are not fundamentally slower than consumer groups.

Performance Testing Yaml

▲ 0

0 views

Kev Quirk 3 months ago

Adding a Book Editor to My Pure Blog Site

Regular readers will know that I've been on quite the CMS journey over the years. WordPress, Grav, Jekyll, Kirby, my own little Hyde thing, and now Pure Blog . I won't bore you with the full history again, but the short version is: I kept chasing just the right amount of power and simplicity, and I think Pure Blog might actually be it. But there was one nagging thing. I have a books page that's powered by a YAML data file, which creates a running list of everything I've read with ratings, summaries, and the occasional opinion. It worked great, but editing it meant cracking open a YAML file in my editor and being very careful not to mess up the indentation. Not ideal. So I decided to build a proper admin UI for it. And in doing so, I've confirmed that Pure Blog is exactly what I wanted it to be - flexible and hackable. I added a new Books tab to the admin content page, and a dedicated editor page. It's got all the fields I need - title, author, genre, dates, a star rating dropdown, and a Goodreads URL. I also added CodeMirror editors for the summary and opinion fields, so I have all the markdown goodness they offer in the post and page editors. The key thing is that none of this touched the Pure Blog core. Not a single line. My new book list in Pure Blog A book being edited Pure Blog has a few mechanisms that make this kind of thing surprisingly clean: is auto-loaded after core, so any custom functions I define there are available everywhere — including in admin pages. I put my function here, which takes the books data and writes it back to the data file, then clears the cache — exactly like saving a normal post does. Again, zero core changes. is the escape hatch for when I do need to override a core file. I added both (where I added the Books tab) and (the new editor) to the ignore list , so future Pure Blog updates won't mess with them. It's a simple text file, one path per line. Patch what you need, ignore it, and move on. is where it gets a bit SSG-ish. The books page is powered by — a PHP file that loads the YAML, sorts it by read date, and renders the whole page. It's essentially a template, not unlike a Liquid or Nunjucks layout in Jekyll or Eleventy. Same idea for the books RSS feed . Using a YAML data file for books made more sense to me, rather than markdown files like a post or a page, as it's all metadata really. There's no real "content" for these entries. Put those three things together and you've got something pretty nifty. A customisable admin UI, safe core patching, and template-driven data pages — all without a plugin system or any framework magic. Bloody. Brilliant. I spent years chasing the perfect CMS, and a big part of what I was looking for was this . The ability to build exactly what I need without having to fight the platform, or fork it, or bolt on a load of plugins. With Kirby, I could do this kind of thing, but the learning curve was steep and the blueprint system took me ages to get my head around. With Jekyll/Hyde, I had the SSG flexibility, but no web-based CMS I could login to and create content - I needed my laptop. Pure Blog sits in a really nice middle ground — it's got a proper admin interface out of the box, but it gets out of the way when you want to extend it. I'm chuffed with how the book editor turned out. It's a small thing, but it's exactly what I wanted, and the fact that it all lives outside of core means I can update Pure Blog without worrying about losing any of it. Now, if you'll excuse me, I have some books to log. 📚 Thanks for reading this post via RSS. RSS is ace, and so are you. ❤️ You can reply to this post by email , or leave a comment .

Open Source Php Web Development Yaml

▲ 0

0 views

matduggan.com 4 months ago

You can absolutely have an RSS dependent website in 2026

I write stuff here. Sometimes the stuff is good. Sometimes it reads like I wrote it at 2 AM after an argument with a YAML file, which is because I did. But one decision I made early on was that I didn't want to offer an email newsletter. Part of this was simple economics. At one point I did have a Subscribe button up, and enough people clicked it that the cost of actually sending those emails started to resemble a real bill. Sending thousands of emails when you have no ads, no sponsors, and no monetization strategy beyond "I guess people will just... read it?" doesn't make a lot of financial sense. But the bigger reason — the one I actually care about — is that I didn't want a database full of email addresses sitting under my control if I could possibly avoid it. There's a particular flavor of anxiety that comes with being the custodian of other people's personal data, a low-grade dread not unlike realizing you've been entrusted with someone's elderly cat for two weeks and the cat has a medical condition. I can't lose data I don't have. I never need to lie awake wondering whether some user is reusing their bank password to log into my website just to manage their subscription preferences. The best way I can safeguard user data is by never having any in the first place. It's not a security strategy you'll find in any textbook, but it is airtight. Now, when I explained this philosophy to people who run similar websites, the reaction was — and I'm being generous here — warm laughter . The kind of laughter you get when you ask if an apartment in Copenhagen is under $1,000,000. Email newsletters are the only way to run a site like this, they said. RSS is dead, they said. You might as well be distributing your writing via carrier pigeon or community bulletin board. One person looked at me the way you'd look at someone who just announced they were going to navigate cross-country using only a paper atlas. Not angry. Just sad. I'm lucky in that I'm not trying to get anyone to pay me to come here. If I were, the math would probably change. I'd be out there A/B testing subject lines and agonizing over open rates like everyone else, slowly losing pieces of my soul in a spreadsheet. But if your question is simply, "Can I make a hobbyist website that actual humans will find and read without an email newsletter?" — the answer is a resounding yes. And I have the logs to prove it. All of this is from Nginx access.log. These logs get rotated daily and don't include the majority of requests that hit the Cloudflare cache before they ever reach my server, so the real numbers are higher. But I think they're reasonably representative of the overall shape of things. About half my traffic is readers hitting or — people who have, of their own free will, pointed an RSS reader at my site and said yes, tell me when this person has opinions again . The other half are arriving via a specific link they stumbled across somewhere in the wild. If we do a deeper dive into that specific RSS traffic, we learn a few interesting things. The user-agent breakdown shows the usual suspects — the RSS readers you'd expect, the ones that have been around long enough to have their own Wikipedia articles. There are also some abusers in the metrics. I have no idea what "Daily-AI-Morning" is, but whatever it's doing, it's polling my feed with the frantic energy of someone refreshing a package tracking page on delivery day. The time distribution, though, is pretty good — spread out across the day in a way that suggests real humans checking their feeds at real human intervals, rather than a single bot hammering me every thirty seconds. My conclusion is this: if you want to run a website that relies primarily on RSS instead of email newsletters, you absolutely can. The list of RSS readers hasn't dramatically changed in a long time, which is actually reassuring — it means the ecosystem is stable, not dead. The people who use RSS really use RSS. They're not trend-chasers. They're the type who still have a working bookmark toolbar. They are, in the best possible sense, your people. Effectively, if you make your site RSS-friendly and you test it in NetNewsWire, you will — slowly, quietly, without a single "SUBSCRIBE FOR MORE" pop-up — build a real audience of people who actually want to read what you write. No email database required. No passwords to leak. No giant confusing subscription system.

Web Development Blogging Yaml

▲ 0

0 views

blog.philz.dev 4 months ago

What is Buildkite?

If you're starting a new project, just skip the misery of GitHub actions and move on. Buildkite mostly gets it. The core Buildkite noun is a Pipeline, and, as traditional for an enterprise software company, their docs don't really tell you what's what. The point is that your pipeline should be: Pipelines can add steps to themselves . So, you can write a script to generate your pipeline (or just store it in your repo), and cat it into the command, and that's how the rest of your steps are discovered. Pipeline steps are each executed in their own clean checkout of what you're building. So, if you want to run the playwright tests in parallel with the backend tests (or whatever), you just declare that as two different steps, but they're part of the same thing. Pipelines have a dependency graph between steps that's conceptually similar to . (Perhaps was the Make replacement that I first heard of that did "generate the ninja graph"?) The agents seem pretty good at manipulating Buildkite once you give them an API key. They also seem to not in-line shell scripts into the yaml, which is Obviously Good. The ways to speed up the build is always the same: cache and parallelize. A 16-core machine for 1 minute costs the same as an 2-core machine for 8 minutes, and I know which one I'd rather wait for! Buildkite makes parallelism pretty easy. Anyway, it's pretty good. Thanks, Buildkite.

Shell DevOps Ci Cd Yaml

▲ 0

0 views

André Arko 4 months ago

How to Install a Gem

This post was originally given as a talk at SF Ruby Meetup . The slides are also available. Hello, and welcome to How To Install A Gem . My name is André Arko, and I go by @indirect on all the internet services. You might know me from being 1/3 of the team that shipped Bundler 1.0, or perhaps the 10+ years I spent trying to keep RubyGems.org up and running for everyone to use. More recently, I’ve been working on new projects: , a CLI to install Ruby versions and gems at unprecedented speeds, and gem.coop , a community gem server designed from the ground up so Bundler an can install gems faster and more securely than ever before. So, with that introduction out of the way, let’s get started: do you know how to install a gem? Okay, that’s great! You can come up and give this talk instead of me. I’ll just sit over here while you write the rest of this post. Slightly more seriously, do you know how converts the name that you give it into a URL to download a .gem file? It’s called the “compact index”, and we’ll see how it works very soon. Next, who in the audience knows how to unpack a .gem file? Do you know what format .gem files use, and what’s inside them? We’ll look at gem structure and gemspec files as well. Then, do you know where to put the files from inside the gem? Where do all of these files and directories get put on disk so we can use them later? Does anyone know off the top of their head? Once those files have been unpacked into the correct places, the last thing we need to know is how to require them. How do these unpacked files on disk get found by Ruby, so you can and have that actually work? This exercise was mostly to show that using gems every day actually skips over most of the way they work underneath. So let’s look at what a gem is, and examine how they work. By the end of this talk, you’ll know what’s inside a gem, where how RubyGems figures out what to download, and where and how that download gets installed so you can use it. And if you already everything we just talked about, please feel free to go straight to rv.dev and start sending us pull requests! First, we’re going to look at how the name of a gem becomes a URL for a .gem file. Let’s use as our example. Historically, there have been at least five or six different ways to look up information about a gem based on its name, but today there is one canonical way: the compact index. It’s so simple that you can do it yourself using curl. Just run , and you’ll be able to read the exact output that every tool uses to look up the versions of a gem that exist. Each line in the file describes one version of the gem, so let’s look at one line. We can break down that line with , and tackle each part one at a time. First, . That’s the version of that this line is about. So we now know for sure that exists. Next, a list of dependencies. The gem (version ) declares dependencies on a bunch of other gems: , , , , , , , , , , , , and . Each dependency has a version requirement attached, and for almost every gem it is exactly version , and only version . For , Rails is a little bit more flexible, and allows any version and up. The final section contains a checksum, a ruby requirement, and a rubygems requirement. The checksum is a sha256 hash of the .gem file that contains the gem, so after we download the gem we can check to make sure we have the right file by comparing that checksum. For this version of Rails, the required Ruby version is or greater, and the required RubyGems version is or greater. It’s up to the client to do something with that information, but hopefully you’ll see an error if you are using Ruby or RubyGems that’s too old. Great! So now we know the important information: Rails version is real, and strong, and is our friend. We can download it, and check the checksum against the checksum we were given in the info file line. Let’s do that now: Notice that the checksum produced by exactly matches the checksum we previously saw in our line from the info file: . That lets us know that we got the right file, and there were no network or disk errors. Now that we have the gem, we can investigate: what exactly is inside a gem? At this point, we’re going to pivot from the gem to the gem. There’s a good reason for that, and the reason is… the gem doesn’t actually have any files in it. So it’s a bad example. In order to show off what a gem looks like when it has files in it, we’ll use instead. So, we have our .gem file downloaded with curl. What do we do now? The first piece of secret knowledge that we need: gems are tarballs. That means we can open them up with regular old . Let’s try it. So what’s inside the .gem tarball is… another tarball. And also two gzipped files. Let’s look at the files first. As you might expect from its name, the file is a gzipped YAML file, containing checksums for the other two files. It’s maybe a bit silly to have multiple layers of checksumming here, but it does confirm that the outer layer of tarball and zip was removed without any errors. Okay, so what’s inside ? The answer is… Ruby, sort of. It’s a YAML-serialized instance of the class. We can see exactly what was put into this object at the time the gem was built. After snipping out the YAML that lists the dependencies (which we already looked at, because they are included in the info file), what’s left is some relatively simple information about the gem. Author, author’s email, description, homepage, license, various URLs. For the purposes of installing and using the gem, we care about exactly six pieces of information: , , , , , and . We’re going to combine those items with the files in the remaining file to get our unpacked and installed gem. Now that we know what’s in the gem specification, let’s look at what’s inside the data tarball. It matches up very closely with the long list of entries in the array in the gemspec. So now we have a bunch of files. Where are we going to put these files? Enter: the magic of RubyGems. The scheme that RubyGems has come up with is largely shaped by the constraints of how Ruby finds files to require, which we’re going to look at soon. For now, it is enough for us to know that RubyGems keeps track of a list of directories, a lot like the way works for your shell to find commands to run. To find the current directory, you can run . Here’s what that looks like: From this list, we can see that RubyGems organizes its own files into a few directories. To install a gem, we’re going to need to put the files we have into each of those directories, with specific paths and filenames. Just to recap, the files we need to place somewhere are: So let’s move the files into the directories we see RubyGems offers. First, cache the .gem file so RubyGems doesn’t need to download it again later: Then, add the gem specification so that RubyGems will be able to find it. There’s a small twist here, which is that the directory doesn’t contain YAML files, it contains Ruby files. So we also need to convert the YAML file back into a Ruby object, and then write out the Ruby code to create that object into a file that RubyGems can load later. Next, we need to put the files that make up the contents of the gem into the directory. One more thing we need to do: set up the executables provided by the gem. You can check out the files that RubyGems generates by looking in , but for our purposes we just need to tell RubyGems what gem and executable it needs to run, so we can do that: And with that, we’ve installed the gem! You can run the file that we just created to prove it: As we wrap up here, there are three aspects of gems that we haven’t touched on at all: docs, extensions, and plugins. We don’t have time to talk about them today in this meetup talk slot. Hopefully a future (longer) version of this talk will have space to include all of those things, because they are all super interesting, I promise. In the meantime, I will have to direct you to the docs for RDoc to learn more about docs, to the source code of or RubyGems itself if you want to learn more about gem extensions and plugins. There’s one last thing to figure out before we wrap up: how does find a gem for us to be able to use it? To explain that, we’ll have to drop down to some basic Ruby, and then look at the ways that RubyGems monkeypatches Ruby’s basic to make it possible to have gems with versions. The first thing to know about is that it works exactly like does in your shell. There’s a global Ruby variable named , and it’s an array of paths on disk. When you try to require something, Ruby goes and looks inside each of those paths to see if the thing you asked for is there. You can test this out for yourself in just a few seconds! Let’s try it. The Ruby CLI flag lets you add directories to the variable, and then the function looks inside that directory to find a file with the name that you gave to require. No magic, just a list to check against for files on disk. Now that you understand how the variable makes work, how does RubyGems work? You can’t just put ten different versions of into the and expect to still work. RubyGems handles multiple versions of the same file by monkeypatching . Let’s look at what happens when we , which is a file located inside the gem that we just installed. RubyGems starts by looking at all of the gem specifications, including the one we saved earlier. In each specification, it combines the name and version with the values in to come up with a path on disk. So for our just-installed gem, that would mean a path of: . RubyGems knows that directory contains a file named , so it is a candidate to be “activated”, which is what RubyGems calls it when a gem is added to your . As long as internal bookkeeping shows that no other versions of have already been added to the , we’re good! RubyGems adds this specific directory to the , and delegates to the original implementation of . Require finds the file at , reads it, and evaluates it. With that, we’ve done it! We have found, downloaded, unpacked, and installed a gem so that Ruby is able to run a command and load ruby files, without ever touching the command. If you’re interested in contributing to an open source project that works a lot with gems, we would love to work with you on , where we are working to create the fastest Ruby and gem manager in the world. And of course, if your company could use faster, easier, or more secure gems for developers, for CI, and for production deployments, we can help. We’d love to talk to you and you can find our contact information at spinel.coop . railties-8.1.3.gem (the .gem file itself) metadata.gz (the YAML Gem::Specification object from inside the gem) the unpacked data.tar.gz files (the contents of the gem)

Programming Ruby Shell Yaml Tutorial

▲ 0

0 views

Danny McClelland 5 months ago

Daemon: a 2006 techno-thriller that reads like a 2026 product roadmap

Fair warning: this post contains spoilers for both Daemon and Freedom™. Then again, the books came out twenty years ago. If you haven’t read them by now, you probably weren’t going to. I highly recommend them both though, if you haven’t read them yet. I finished re-reading Daniel Suarez’s Daemon and its sequel Freedom™ a few weeks ago. I first picked them up years back and thought they were solid techno-thrillers with some wild ideas baked into an entertaining plot. Reading them again in 2026, they’re just as gripping, but for somewhat different reasons. The realism has caught up in a way I wasn’t expecting. When I first read these books, I took them as clever speculation of what the future may look like. Now I’m reading them and thinking: yeah, that exists. That too. And that. The fiction hasn’t aged, the real world has just gone ahead and built most of it. The premise is straightforward: Matthew Sobol, a dying game developer, leaves behind a distributed AI program that activates after his death. The Daemon, as it’s called, begins infiltrating systems, recruiting operatives through an online game-like interface, and systematically restructuring society. In Freedom™ , the sequel, that restructuring plays out in full: decentralised communities, alternative economies, mesh networks, and a population split between those plugged into the new system and those clinging to the old one. Suarez self-published Daemon in 2006. That bears repeating. 2006. YouTube was a year old. The iPhone didn’t exist yet. And this guy was writing about autonomous vehicles, augmented reality glasses, voice-controlled AI agents, distributed botnets acting with real-world consequences, and desktop fabrication units. Not as far-future sci-fi set in 2150, but as things that were five to ten years away. The Daemon’s entire existence starts with what is essentially a cron job. Sobol’s program sits dormant, scraping news headlines, waiting for a specific trigger: reports of his own death. When it finds them, it wakes up and starts executing. It wasn’t a sentient AI gone rogue with a dramatic moment it becomes into being. Just a script polling RSS feeds on a schedule, pattern-matching against text, and firing off the next step in a chain. I had something similar with OpenClaw for a while. Not the assassinations, obviously, but the same fundamental architecture of scheduled tasks that wake up, pull information from the internet, process it, and take action without any human prompting. Morning briefings, inbox sweeps, periodic research jobs. The Daemon’s trigger mechanism felt sinister in 2006. Now it’s a feature you can configure in a YAML file. Yep, I know what you’re thinking - we’ve had cron for a long time and this part was possible even before the book was written - but this is just the first chapter of the book. Then there are the autonomous machines. Sobol’s Daemon deploys “AutoM8s”: driverless vehicles that transport operatives and, in the book’s darker moments, act as weapons. It also uses robotic ground units for surveillance and enforcement. In 2006, this was pure fiction. Now Boston Dynamics has Spot, a quadruped robot dog that autonomously navigates terrain, avoids obstacles, and self-charges. Their Atlas humanoid can do backflips, parkour courses, and 540-degree inverted flips. These are real machines you can watch on YouTube doing things that would have read as absurd twenty years ago. Suarez’s vision of autonomous robots patrolling and operating independently isn’t a prediction anymore, it’s a product catalogue. The always-connected vehicle is another one. In Daemon, the AutoM8s are permanently networked, receiving instructions and sharing data in real time. Every Tesla on the road today is essentially this. Always online, streaming telemetry back to the mothership, receiving over-the-air updates, and feeding its camera data into a collective neural network. The car you’re driving is a node in someone else’s distributed system. Sobol would have appreciated the irony of people voluntarily buying into that. One of the creepier technologies in the books is WiFi-based surveillance, using wireless signals to detect and track people through walls. Suarez wrote about this as a covert capability the Daemon could exploit. Carnegie Mellon researchers have since built exactly that. Their “DensePose from WiFi” system uses standard WiFi router signals to reconstruct human poses in real time, even through solid walls. The reflected signals carry enough information about body shape and movement that a neural network can map what you’re doing in a room without a single camera. It works through drywall, wood, and even concrete up to a point, and none of this is classified military tech. It’s published academic research that anyone can read. The acoustic weapon is probably the one that catches people off guard the most. In Daemon, there’s a directed sound system that can make audio appear to come from right beside you while no one else in the room hears a thing. It sounds like science fiction until you look up parametric speakers. Companies like Holosonics have been selling “Audio Spotlight” systems for years. They work by emitting “modulated ultrasonic beams that demodulate into audible sound only within a tight, targeted area” - I’ve experienced these in airports, but have no idea what that quote actually means. Museums, airports, and retailers already use them, and the military has explored them for crowd control. The effect is exactly what Suarez described, sound that seems to materialise out of thin air, audible only to the person standing in the beam, and you can buy one commercially right now. The social dynamics might be the most on-the-nose parallel of all. In the books, the Daemon recruits human operatives to carry out tasks in the physical world. It finds people, assigns them work, and pays them through its own system. The humans don’t fully understand the bigger picture. They just complete their tasks and collect their reward. In January 2026, a site called RentAHuman.ai launched. It’s a platform where OpenClaw AI agents can hire actual people to perform tasks for them. Humans sign up with their skills and hourly rate, AI agents post jobs, and people complete them for payment in stablecoins. Over 40,000 people registered within days. The framing is different, obviously. It’s gig work, not a shadowy network of mindless humans - arguably. But the underlying structure is identical. AI systems delegating physical-world tasks to human operatives who sign up voluntarily, motivated by compensation and a sense of participation in something larger. Suarez wrote it as dystopian fiction that, in 2006 seemed like only the insane would enroll. We built it as a startup and it got very popular, very quickly. The 10% that hasn’t happened is mostly about scale and centralisation. Sobol’s Daemon is a single, coherent system with an architect’s intent behind every action. Real distributed systems don’t work like that. AI development has been messy, competitive, and fragmented across hundreds, perhaps even thousands, of companies and research labs. There’s no singular Daemon pulling strings, just a chaotic landscape of overlapping systems with no one fully in control. Which, depending on your perspective, might actually be worse. The weaponised autonomous vehicles haven’t materialised in the way Suarez imagined either, though military drones certainly have. The line between his fiction and real-world drone warfare is thinner than most people would be comfortable with. And the neat resolution in Freedom™ , where Darknet communities build something genuinely better, still feels like the most fictional part of the whole thing. We’ve got the decentralised technology. We’ve got the mesh networks and the alternative currencies. What we haven’t got is the social cohesion to do anything coherent with them. Crypto became a speculative casino with massive peaks and equal troughs. The tools exist, but the utopian bit remains out of reach. Suarez wasn’t writing from some academic ivory tower or speculating about technology he’d never touched. He was an IT consultant who spent years working with Fortune 1000 companies, and you can feel that experience on every page. He understood how systems actually work, how they fail, and how they get exploited, which is what makes re-reading both books such a strange experience. He wasn’t guessing at any of this. He was extrapolating from things he could already see forming, and doing it with an accuracy that I genuinely wouldn’t have believed twenty years ago. If you haven’t read Daemon and Freedom™ , go and read them. I track everything I read on Hardcover , and both of these are easy five-star picks. They’re fantastic books on their own merits. The pacing is relentless, the technical detail is sharp without being dry, and the plot keeps pulling you forward. I’d recommend them even if none of the technology had come true. But it has, and not gradually over twenty years. The pace is accelerating. Half the parallels I’ve listed in this post didn’t exist even twelve months ago. OpenClaw’s cron system, RentAHuman.ai, the latest generation of Boston Dynamics robots: all 2025 or 2026 developments. The gap between Suarez’s fiction and our reality is closing faster each year, and that makes the books hit differently every time you revisit them. I suspect they’ll hit differently again in another twelve months, and I can’t wait to re-read them then.

AI Yaml Books

▲ 0

0 views

Ruslan Osipov 5 months ago

Homepage for a home server

I have a NAS (Network Accessible Storage) which doubles as a home server. It’s really convenient to have a set of always-on, locally hosted services - which lets me read RSS feeds without distractions, have local photos storage solution, or move away from streaming services towards organizing my (legally owned) collections of movies, shows, and audiobooks. At this point I have 20-or-so services running and sometimes it gets hard to keep track of what’s where. For that - I found Homepage . A simple, fast, lightweight page which connects to all of my services for some basic monitoring, and reminds me what my service layout looks like. Here’s what I built: I love that the configuration lives in YAML files, and because the page is static - it loads real fast. There are many widgets which provide info about various services out of the box. It’s neat. There’s definitely a question of how much I’ll keep this up-to-date: it’s not an automatically populated dashboard, and editing is a two-step process (SSH into the machine, edit the YAML configs) - which adds some friction. We’ll have to wait and see, but for now I’m excited about my little dashboard.

Yaml Programming DevOps Web

▲ 0

0 views

matduggan.com 5 months ago

I Sold Out for $20 a Month and All I Got Was This Perfectly Generated Terraform

Until recently the LLM tools I’ve tried have been, to be frank, worthless. Copilot was best at writing extremely verbose comments. Gemini would turn a 200 line script into a 700 line collection of gibberish. It was easy for me to, more or less, ignore LLMs for being the same over-hyped nonsense as the Metaverse and NFTs. This is great for me because I understand that LLMs represent a massive shift in power from an already weakened worker class to an increasingly monarch-level wealthy class. By stealing all human knowledge and paying nothing for it, then selling the output of that knowledge, LLMs are an impossibly unethical tool. So if the energy wasting tool of the tech executive class is also a terrible tool, easy choice. Like boycotting Tesla for being owned by an evil person and also being crappy overpriced cars, or not shopping at Hobby Lobby and just buying directly from their Chinese suppliers, the best boycotts are ones where you aren’t really losing much. Google can continue to choke out independent websites with their AI results that aren’t very good and I get to feel superior doing what I was going to do anyway by not using Google search. This logic was all super straight forward right up until I tried Claude Code. Then it all got much more complicated. Let’s just get this out of the way right off the bat. I didn't want to like Claude Code. I got a subscription with the purpose of writing a review on it where I would find that it was just as terrible as Gemini and Copilot. Except that's not what happened. Instead it was like discovering the 2AM kebab place might actually make the best pizza in town. I kept asking Claude to do annoying tasks where it was easy for me to tell if it had made a mistake and it kept doing them correctly. It felt impossible but the proof was right in front of me. I’ve written tens of thousands of lines of Terraform in my life. It is a miserable chore to endlessly flip back and forth between the provider documentation and Vim, adding all the required parameters. I don’t learn anything by doing it, it’s just a grind I have to push through to get back to the meaningful work. The amount of time I have wasted on this precious time on Earth importing all of a companies DNS records into Terraform, then taking the autogenerated names and organizing them so that they make sense for the business is difficult to express. It's like if the only way I knew how to make a hamburger bun was to carefully put every sesame seed by hand on the top only to stumble upon an 8 pack of buns for $4 at the grocery store after years of using tiny tweezers to put the seeds in exactly the right spot. I feel the same way about writing robust READMEs, k8s YAML and reorganizing the file structure of projects. Setting up more GitHub Actions is as much fun as doing my taxes. If I never had to write another regex for the rest of my life, that would be a better life by every conceivable measure. These are tasks that sap my enthusiasm for this type of work, not feed it. I’m not sad to offload them and switch to mostly reviewing its PRs. But the tool being useful doesn’t remove what’s bad about it. This is where a lot of pro-LLM people start to delude themselves. In no particular order are the arguments I keep seeing about LLMs from people who want to keep using them for why their use is fine. This is the most common one I see and the worst. It can be condensed down to “because most things on the internet originally existed to find pornography and/or pirate movies, stealing all content on the internet is actually fine because programmers don’t care about copyright”. You also can’t have it both ways. OpenAI can’t decide to enforce NDAs and trademarks and then also declare law is meaningless. If I don’t get to launch a webmail service named Gmail+ then Google doesn’t get to steal all the books in human existence. The argument basically boils down to: because we all pirated music in 2004, intellectual property is a fiction when it stands in the way of technology. By this logic I shoplifted a Snickers bar when I was 12 so property rights don't exist and I should be allowed to live in your house. I have an internet friend I met years ago playing EVE Online that is a brutally pragmatic person. To someone like him, code craftsmanship is a joke. For those of you who are unaware, EVE Online is the spaceship videogame where sociopaths spend months plotting against each other. His approach to development is 80% refining requirements and getting feedback. He doesn’t care at all about DRY, he uses Node because then he can focus on just JavaScript, he doesn’t invest a second into optimization until the application hits a hard wall that absolutely requires it. His biggest source of clients? Creating fast full stacks because internal teams are missing deadlines. And he is booked up for at least 12 months out all the time because he hits deadlines. When he started freelancing I thought he was crazy. Who was going to hire this band of Eastern European programmers who chain smoke during calls and whose motto is basically "we never miss a deadline". As it turns out, a lot of people. Why doesn't he care? Why doesn't he care about these things? He believes that programmers fundamentally don't understand the business they are in. "Code is perishable" is something he says a lot and he means it. Most of the things we all associate with quality (full test coverage, dependency management, etc) are programmers not understanding the rate of churn a project undergoes over its lifespan. The job of a programmer, according to him, is delivering features that people will use. How pleasant and well-organized that code is to work with is not really a thing that matters in the long term. He doesn't see LLM-generated code as a problem because he's not building software with a vision that it will still be used in 10 years. Most of the stuff typically associated with quality he, more or less, throws in the trash. He built a pretty large stack for a automotive company and my jaw must have hit the table when he revealed they're deploying m6g.4xlarge for a NodeJS full-stack application. "That seems large to me for that type of application" was my response. He was like "yeah but all I care about are whether the user metrics show high success rate and high performance for the clients". It's $7000 a year for the servers, with two behind a load balancer. That's absolutely nothing when compared with the costs of what having a team of engineers tune it would cost and it means he can run laps around the internal teams who are, basically, his greatest competition. To be clear, he is very technically competent. He simply rejects a lot of the conventional wisdom out there about what one has to do in order to make stuff. He focuses on features, then securing endpoints and more or less gives up on the rest of it. For someone like this, LLMs are a logical choice for him. The annoying thing about my friend is that his bank account suggests he's right. But I can't get there. If I'm writing a simple script or something as a one-off, it can sometimes feel like we're all wasting the companies time when we have a long back and forth on the PR discussing comments or the linting or whatever. So it's not that this idea is entirely wrong . But the problem with programming is you never know what is going to be "the core" of your work life for the next 5 years. Sometimes I write a feature, we push it out, it explodes in popularity and then I'm a little bit in trouble because I built a MVP and now it's a load-bearing revenue generating thing that has to be retooled. I also just have trouble with the idea that this is my career and the thing I spend my limited time on earth doing and the quality of it doesn't matter. I delight in craftsmanship when I encounter it in almost any discipline. I love it when you walk into an old house and see all the hand crafted details everywhere that don't make economic sense but still look beautiful. I adore when someone has carefully selected the perfect font to match something. Every programmer has that library or tool that they aspire to. That code base where you delight at looking at it because it proves perfection is possible even if you have never come close to reaching that level. For me its always been looking through the source code of SQLite that restores my confidence. I might not know what I'm doing but it's good to be reminded that someone out there does. Not everything I make is that great, but the concept of "well great doesn't matter at all" effectively boils down to "don't take pride in your work" which is probably the better economic argument but feels super bad to me. In a world full of cheap crap, it feels bad to make more of it and then stick my name on it. The best argument for why programmers should be using LLMs is because it's going to be increasingly difficult to compete for jobs and promotions against people who are using them. In my experience Claude Code allows me to do two tasks at once. That's a pretty hard advantage to overcome. Last Tuesday I had Claude Code write a GitHub Action for me while I worked on something else. When it was done, I reviewed it, approved it, and merged it. It was fine. It was better than fine, actually — it was exactly what I would have written, minus the forty-five minutes of resentment. I sat there for a moment, staring at the merged PR, feeling the way I imagine people feel when they hire a cleaning service for the first time: relieved, and then immediately guilty about the relief, and then annoyed at myself for feeling guilty about something that is, by any rational measure, a completely reasonable thing to do. Except it isn't reasonable. Or maybe it is. I genuinely don't know anymore, and that's the part that bothers me the most — not that the tool works, but that I've lost the clean certainty that it shouldn't. So now I'm paying $20 a month to a company that scraped the collective knowledge of humanity without asking so that I can avoid writing Kubernetes YAML. I know what that makes me. I just haven't figured out a word for it yet that I can live with. When I asked my EVE friend about it on a recent TeamSpeak session, he was quiet for awhile. I thought that maybe my moral dilemma had shocked him into silence. Then he said, "You know what the difference is between you and me? I know I'm a mercenary. You thought you were an artist. We're both guys who type for money." I couldn't think of a clever response to that. I still can't.

JavaScript AI Programming Terraform Yaml

▲ 0

0 views

matduggan.com 6 months ago

GitButler CLI Is Really Good

My workflow has remained mostly the same for over a decade. I write everything in Vim using the configuration found here . I run Vim from inside of tmux with a configuration found here . I write things on a git branch, made with the CLI, then I add them with to that branch, trying to run all of the possible linting and tests with before I waste my time on GitHub Actions. Then I run which is an alias to . Finally I successfully commit, then I copy paste the URL returned by GitHub to open a PR. Then I merge the PR and run to go back to the primary branch, which is an alias to . This workflow, I think, is pretty familiar for anyone working with GitHub a lot. Now you'll notice I'm not saying because almost nothing I'm doing has anything to do with . There's no advantage to my repo being local to my machine, because everything I need to actually merge and deploy code lives on GitHub. The CI runs there, the approval process runs there, the monitoring of the CI happens there, the injection of secrets happens there. If GitHub is down my local repo does, effectively, nothing. My source of truth is always remote, which means I pay the price for complexity locally but I don't benefit from it. At most jobs: This means the following is also true: Almost all the features of are wasted on me in this flow. Now because this tool serves a million purposes and is designed to operate in a way that almost nobody uses it for, we all pay the complexity price of and never reap any of the benefits. So instead I keep having to add more aliases to paper over the shortcomings of . These are all the aliases I use at least once a week. Git's offline-first design creates friction for online-first workflows, and GitButler CLI eliminates that friction by being honest about how we actually work. (Edit: I forgot to add this disclaimer. I am not, nor have ever been an employee/investor/best friends with anyone from GitButler. They don't care that I've written this and I didn't communicate with anyone from that team before I wrote this.) So let's take the most basic command as an example. This is my flow that I do 2-3 times a day without my aliases. I do this because can't make assumptions about the state of the world. However because GitButler is designed with the assumption that I'm working online, we can skip a lot of this nonsense. It's status command understands that there is always a remote main that I care about and that when I run a status that I need to understand my status relative to the remote main as it exists right now. Not how it existed the last time I remembered to pull. However this is far from the best trick it has up its sleeve. You're working on a feature, notice an unrelated bug, and now you have to stash, checkout, fix, commit, push, checkout back, stash pop. Context switching is expensive and error-prone. GitButler effectively hacks a solution into that fixes this with multiple branches applied simultaneously. Assign files to different branches without leaving your workspace. What do I mean by that. Let's start again with my status Great looks good. Alright so lets say I make 2 new branches. I'm working on a new feature for adding auth and while I'm working on that, I see a typo I need to fix in a YAML. I can work on both things at the same time: And easily commit to both at the same time without doing anything weird . Stacked PRs are the "right" way to break up large changes so people on your team don't throw up at being asked to review 2000 lines, but Git makes them miserable. When the base branch gets feedback, you have to rebase every dependent branch, resolve conflicts, force-push, and pray. Git doesn't understand branch dependencies. It treats every branch as independent, so you have to manually maintain the stack. GitButler solves this problem with First-class stacked branches. The dependency is explicit, and updates propagate automatically. So what do I mean. Let's say I make a new API endpoint in some Django app. First I make the branch. So let's say I'm working on the branch and get some good feedback on my PR. It's easy to resolve the comments there while leaving my branched off this as a stacked thing that understands the relationship back to the first branch as shown here. In practice this is just a much nicer way of dealing with a super common workflow. Maybe the most requested feature from new users I encounter is an easier undo. When you mess up in Git, recovery means diving into , understanding the cryptic output, and hoping you pick the right . One wrong move and you've made it worse. GitButlers is just easier to use. So the basic undo functionality is super simple to understand. rolls me back one operation. To me the mental model of a snapshot makes a lot more sense than the git history model. I do an action, I want to undo that action. This is better than the git option of: I've been using GitButler in my daily work since I got the email that the CLI was available and I've really loved it. I'm a huge fan of what this team is doing to effectively remodel and simplify Git operations in a world where almost nobody is using it in the way the tool was originally imagined to be used. I strongly encourage folks go check it out for free at: https://docs.gitbutler.com/cli-guides/cli-tutorial/tutorial-overview . It does a ton of things (like help you manage PRs) that I didn't even touch on here. Let me know if you find something cool that I forgot at: https://c.im/@matdevdug You can't merge without GitHub (PRs are the merge mechanism) You can't deploy without GitHub (Actions is the deployment trigger) You can't get approval without GitHub (code review lives there) Your commits are essentially "drafts" until they exist on GitHub You never work disconnected intentionally You don't use local branches as long-lived divergent histories You don't merge locally between branches (GitHub PRs handle this) You don't use for archaeology — you use GitHub's blame/history UI (I often use git log personally but I have determined I'm in the minority on this). Your local repo might be offline for days or weeks The "remote" might be someone else's laptop, not a central server Divergent histories are expected and merging is a deliberate, considered act

Python Yaml Git

▲ 0

0 views

Martin Fowler 6 months ago

Fragments: February 4

I’ve spent a couple of days at a Thoughtworks-organized event in Deer Valley Utah. It was my favorite kind of event, a really great set of attendees in an Open Space format. These kinds of events are full of ideas, which I do want to share, but I can’t truthfully form them into a coherent narrative for an article about the event. However this fragment format suits them perfectly, so I’ll post a bunch of fragmentary thoughts from the event, both in this post, and in posts in the next few days. ❄                ❄                ❄                ❄                ❄ We talked about the worry that using AI can cause humans to have less understanding of the systems they are creating. In this discussion one person pointed out that one of the values of Pair Programming is that you have to regularly explain things to your pair. This is an important part of learning - for the person doing the explaining . After all one of the best ways to learn something is to try to teach it. ❄                ❄                ❄                ❄                ❄ One attendee is an SRE for a Very (Very) Large Code Base. He was less worried about people not understanding the code an LLM writes because he already can’t understand the VVLCB he’s responsible for. What he values is that the LLM helps him understand the what the code is doing, and he regularly uses it to navigate to the crucial parts of the code. There’s a general point here: Fully trusting the answer an LLM gives you is foolishness, but it’s wise to use an LLM to help navigate the way to the answer. ❄                ❄                ❄                ❄                ❄ Elsewhere on the internet, Drew Breunig wonders if software libraries of the future might be only specs and no code . To explore this idea he built a simple library to convert timestamps into phrases like “3 hours ago”. He used the spec to build implementations in seven languages. The spec is a markdown document of 500 lines and a set of tests in 500 lines of YAML. “What does software engineering look like when coding is free?” I’ve chewed on this question a bit, but this “software library without code” is a tangible thought experiment that helped firm up a few questions and thoughts. ❄                ❄                ❄                ❄                ❄ Bruce Schneier on the role advertising may play while chatting with LLMs Imagine you’re conversing with your AI agent about an upcoming vacation. Did it recommend a particular airline or hotel chain because they really are best for you, or does the company get a kickback for every mention? Recently I heard an ex-Googler explain that advertising was a gilded cage for Google, and they tried very hard to find another business model. The trouble is that it’s very lucrative but also ties you to the advertisers, who are likely to pull out whenever there is an economic downturn. Furthermore they also gain power to influence content - many controversies over “censorship” start with demands from advertisers. ❄                ❄                ❄                ❄                ❄ The news from Minnesota continues to be depressing. The brutality from the masked paramilitaries is getting worse, and their political masters are not just accepting this, but seem eager to let things escalate. Those people with the power to prevent this escalation are either encouraging it, or doing nothing. One hopeful sign from all this is the actions of the people of Minnesota. They have resisted peacefully so far, their principal weapons being blowing whistles and filming videos. They demonstrate the neighborliness and support of freedom and law that made America great. I can only hope their spirit inspires others to turn away from the path that we’re currently on. I enjoyed this portrayal of them from Adam Serwer (gift link) In Minnesota, all of the ideological cornerstones of MAGA have been proved false at once. Minnesotans, not the armed thugs of ICE and the Border Patrol, are brave. Minnesotans have shown that their community is socially cohesive—because of its diversity and not in spite of it. Minnesotans have found and loved one another in a world atomized by social media, where empty men have tried to fill their lonely soul with lies about their own inherent superiority. Minnesotans have preserved everything worthwhile about “Western civilization,” while armed brutes try to tear it down by force.

Programming Yaml AI

▲ 0

0 views

Nelson Figueroa 8 months ago

GitHub Actions for Pulumi with an AWS S3 Backend

This is a quick guide to set up GitHub Actions for Pulumi with an AWS S3 Backend. There are some differences compared to running commands locally. This guide assumes you have the following: First, set up secrets on your GitHub repository. These will be filled in by GitHub Actions once we create a workflow YAML. You’ll need to create 3 secrets. You can name them whatever you want, but I’ll be naming them: You can create these by browsing to your GitHub repository > Settings > Secrets and variables > Actions. There are “Environment secrets” and “Repository secrets”. In this case go with “Repository secrets”. Create the three secrets and fill in their respective values. can be whatever you want and doesn’t come from AWS. Make sure you don’t change this after the fact though, or your Pulumi state may break . The end result should look like this: Next, clone the repository locally with . We’ll need to create a few files for a minimum viable Pulumi program. Run and it’ll guide you through the creation of a basic Pulumi program. The language you choose doesn’t matter. Then we can create the YAML file to set up GitHub Actions. Create a YAML file under . I’ll call it in my example. Fill it in with the following YAML, changing values as needed. This runs a so it’ll veryify that everything is set up correctly without actually deplpoying anything. Now push your code to GitHub and see if the GitHub Action workflow ran successfully. You should see output from a successful . If this works, you should be good to go. You can update your Pulumi code and, change to in the GitHub Actions YAML file and run it again to actually deploy some infrastructure. An AWS S3 Bucket created and ready to be used with Pulumi An IAM User that has permissions to read/write to the S3 bucket The Access Key and Secret Access Key for the IAM User to use for authenticating to AWS within GitHub Actions A passphrase of your choosing that will be used to encrypt secrets in the pulumi stack A GitHub repository https://github.com/pulumi/actions

Tutorial Yaml Cloud DevOps

▲ 0

0 views

neilzone 8 months ago

Adding a button to Home Assistant to run a shell command

Now that I have fixed my garage door controller , I wanted to see if I could use it from within Home Assistant, primarily so that I can then have a widget on my phone screen. In this case, the shell command is a simple invocation to a remote machine. (I am not aware of a way to add a button to an Android home screen to run a shell command or bash script.) Adding a button to run a shell command or bash script in Home Assistant was pretty straightforward, following the Home Assistant documentation for shell commands . To my configuration.yaml file, I added something like: The quirk here was that reloading the configuration.yaml file from within the Home Assistant UI was insufficient. I needed to completely restart Home Assistant to pick up the changes. Once I had restarted Home Assistant, the shell commands were available. To add buttons, I needed to create “helpers”. I did this from the Home Assistant UI, via Settings / Devices & services / Helpers / Create helper". One helper for each button. After I had created each helper, I went back into the helper’s settings, to add zone information, so that it appeared in the right place in the dashboard. Having created the button/helper, and the shell command, I used an automation to link the two together. I did this via the Home Assistant UI, Settings / Automations & scenes. For the button, it was linked to a change in state of the button, with no parameters specified. The “then do” action is the shell command for the door in question.

Yaml Tutorial Bash Shell

▲ 0

0 views

codedge 9 months ago

Managing secrets with SOPS in your homelab

Sealed Secrets, Ansible Vault, 1Password or SOPS - there are multiple ways how and where to store your secrets. I went with SOPS and age with my ArgoCD GitOps environment. Managing secrets in your homelab, be it within a Kubernetes cluster or while deploying systems and tooling with Ansible, is a topic that arises with almost 100% certainty. In general you need to decide, whether you want secrets to be held and managed externally or internally. One important advantage I see with internally managed solutions is, that I do not need an extra service. No extra costs and connections, no chicken-egg-problem when hosting your passwords inside your own Kubernetes cluster, but cannot reach it when the cluster is down. Therefore I went with SOPS for both, secrets for my Ansible scripts and secrets I need to set for my K8s cluster. While SOPS can be used with PGP, GnuPG and more, I settled with age as encryption. With SOPS your secrets live, encrypted, inside your repository and can be en-/decrypted on-the-fly whenever needed. The private key for encryption should, of course, never be committed into your git repository or made available to untrusted sources. First, we need to install SOPS, age and generate an age key. SOPS is available for all common operating systems via the package manager. I either use Mac or Arch: Now we need to generate an age key and link it to SOPS as the default key to encrypt with. Generate an age key Our age key will live in . Now we tell SOPS where to find our age key. I put the next line in my . The last thing to do is to put a in your folder from where you want to encrypt your files. This file acts as a configuration regarding the age recipient (key) and how the data should be encrypted. My config file looks like this: You might wonder yourself about the first rule with . I will just quote the [KSOPS docs](To make encrypted secrets more readable, we suggest using the following encryption regex to only encrypt data and stringData values. This leaves non-sensitive fields, like the secret’s name, unencrypted and human readable.) here: To make encrypted secrets more readable, we suggest using the following encryption regex to only encrypt data and stringData values. This leaves non-sensitive fields, like the secret’s name, unencrypted and human readable. All the configuration can be found in the SOPS docs . Let’s now look into the specifics using our new setup with either Ansible or Kubernetes. Ansible can automatically process (decrypt) SOPS-encrypted files with the [Community SOPS Collection](Community SOPS Collection). Additionally in my I enabled this plugin ( see docs ) via Now, taken from the official Ansible docs: After the plugin is enabled, correctly named group and host vars files will be transparently decrypted with SOPS. The files must end with one of these extensions: .sops.yaml .sops.yml .sops.json That’s it. You can now encrypt your group or host vars files and Ansible can automatically decrypt them. SOPS can be used with Kubernetes with the KSOPS Kustomize Plugin . The configuration we already prepared, we only need to apply KSOPS to our cluster. I use the following manifest - see more examples in my homelab repository : Externally managed: this includes either a self-hosted and externally hosted secrets solution like AWS KMS, password manager like 1Password or similar Internally managed: solutions where your secrets live next to your code, no external service is need Arch Linux: Two separate rules depending on the folder, where the encrypted files are located Files ending with are targeted The age key, that should be used for en-/decryption is specified

JSON DevOps Security Yaml

▲ 0

0 views

Kaushik Gopal 9 months ago

Claude Skills: What's the Deal?

Anthropic announced Claude Skills and my first reaction was: “So what?” We already have , slash commands, nested instructions, or even MCPs. What’s new here? But if Simon W thinks this is a big deal , then pelicans be damned; I must be missing something. So I dissected every word of Anthropic’s eng. blog post to find what I missed. I don’t think the innovation is what Skills does or achieves, but rather how it does it that’s super interesting. This continues their push on context engineering as the next frontier. Skills are simple markdown files with YAML frontmatter. But what makes them different is the idea of progressive disclosure : Progressive disclosure is the core design principle that makes Agent Skills flexible and scalable. Like a well-organized manual that starts with a table of contents, then specific chapters, and finally a detailed appendix, skills let Claude load information only as needed: So here’s how it works: This dynamic context loading mechanism is very token efficient ; that’s the interesting development here. In this token-starved AI economy, that’s 🤑. Other solutions aren’t as good in this specific way. Why not throw everything into ? You could add all the info directly and agents would load it at session start. The problem: loading everything fills up your context window fast, and your model starts outputting garbage unless you adopt other strategies. Not scalable. Place an AGENTS.md in each subfolder and agents read the nearest file in the tree. This splits context across folders and solves token bloat. But it’s not portable across directories and creates an override behavior instead of true composition. Place instructions in separate files and reference them in AGENTS.md. This fixes the portability problem vs the nested approach. But when referenced, the full content still loads statically. Feels closest to Skills, but lacks the JIT loading mechanism. Slash commands (or in Codex) let you provide organized, hyper-specific instructions to the LLM. You can even script sequences of actions, just like Skills. The problem: these aren’t auto-discovered. You must manually invoke them, which breaks agent autonomy. Skills handle 80% of MCP use cases with 10% of the complexity. You don’t need a network protocol if you can drop a markdown file that says “to access GitHub API, use with .” To be quite honest, I’ve never been a big fan of MCPs. I think they make a lot of sense for the inter-service communication but more often than not they’re overkill. Token-efficient context loading is the innovation. Everything else you can already do with existing tools. If this gets adoption, it could replace slash commands and simplify MCP use cases. I keep forgetting, this is for the Claude product generally (not just Claude Code) which is cool. Skills is starting to solve the larger problem: “How do I give my agent deep expertise without paying the full context cost upfront?” That’s an architectural improvement definitely worth solving and Skills looks like a good attempt. Scan at startup : Claude scans available Skills and reads only their YAML descriptions (name, summary, when to use) Build lightweight index : This creates a catalog of capabilities (with minimal token cost); so think dozens of tokens per skill Load on demand : The full content of a Skill only gets injected into context when Claude’s reasoning determines it’s relevant to the current task ✓ Auto-discovered and loaded ✗ Static: all context loaded upfront (bloats context window at scale) ✓ Scoped to directories ✗ Not portable across folders; overrides behavior, not composition ✓ Organized and modular ✗ Still requires static loading when referenced ✓ Powerful and procedural ✗ Manual invocation breaks agent autonomy ✓ Access to external data sources ✗ Heavyweight; vendor lock-in; overkill for procedural knowledge Token-efficient context loading is the innovation. Everything else you can already do with existing tools. If this gets adoption, it could replace slash commands and simplify MCP use cases. I keep forgetting, this is for the Claude product generally (not just Claude Code) which is cool.

Yaml AI

▲ 0

0 views

Simon Willison 9 months ago

Claude Skills are awesome, maybe a bigger deal than MCP

Anthropic this morning introduced Claude Skills , a new pattern for making new abilities available to their models: Claude can now use Skills to improve how it performs specific tasks. Skills are folders that include instructions, scripts, and resources that Claude can load when needed. Claude will only access a skill when it's relevant to the task at hand. When used, skills make Claude better at specialized tasks like working with Excel or following your organization's brand guidelines. Their engineering blog has a more detailed explanation . There's also a new anthropic/skills GitHub repo. (I inadvertently preempted their announcement of this feature when I reverse engineered and wrote about it last Friday !) Skills are conceptually extremely simple: a skill is a Markdown file telling the model how to do something, optionally accompanied by extra documents and pre-written scripts that the model can run to help it accomplish the tasks described by the skill. Claude's new document creation abilities , which accompanied their new code interpreter feature in September, turned out to be entirely implemented using skills. Those are now available Anthropic's repo covering , , , and files. There's one extra detail that makes this a feature, not just a bunch of files in disk. At the start of a session Claude's various harnesses can scan all available skill files and read a short explanation for each one from the frontmatter YAML in the Markdown file. This is very token efficient: each skill only takes up a few dozen extra tokens, with the full details only loaded in should the user request a task that the skill can help solve. Here's that metadata for an example slack-gif-creator skill that Anthropic published this morning: Toolkit for creating animated GIFs optimized for Slack, with validators for size constraints and composable animation primitives. This skill applies when users request animated GIFs or emoji animations for Slack from descriptions like "make me a GIF for Slack of X doing Y". I just tried this skill out in the Claude mobile web app, against Sonnet 4.5. First I enabled the slack-gif-creator skill in the settings , then I prompted: And Claude made me this GIF . Click to play (it's almost epilepsy inducing, hence the click-to-play mechanism): OK, this particular GIF is terrible, but the great thing about skills is that they're very easy to iterate on to make them better. Here are some noteworthy snippets from the Python script it wrote , comments mine: This is pretty neat. Slack GIFs need to be a maximum of 2MB, so the skill includes a validation function which the model can use to check the file size. If it's too large the model can have another go at making it smaller. The skills mechanism is entirely dependent on the model having access to a filesystem, tools to navigate it and the ability to execute commands in that environment. This is a common pattern for LLM tooling these days - ChatGPT Code Interpreter was the first big example of this back in early 2023 , and the pattern later extended to local machines via coding agent tools such as Cursor, Claude Code, Codex CLI and Gemini CLI. This requirement is the biggest difference between skills and other previous attempts at expanding the abilities of LLMs, such as MCP and ChatGPT Plugins . It's a significant dependency, but it's somewhat bewildering how much new capability it unlocks. The fact that skills are so powerful and simple to create is yet another argument in favor of making safe coding environments available to LLMs. The word safe there is doing a lot of work though! We really need to figure out how best to sandbox these environments such that attacks such as prompt injections are limited to an acceptable amount of damage. Back in January I made some foolhardy predictions about AI/LLMs , including that "agents" would once again fail to happen: I think we are going to see a lot more froth about agents in 2025, but I expect the results will be a great disappointment to most of the people who are excited about this term. I expect a lot of money will be lost chasing after several different poorly defined dreams that share that name. I was entirely wrong about that. 2025 really has been the year of "agents", no matter which of the many conflicting definitions you decide to use (I eventually settled on " tools in a loop "). Claude Code is, with hindsight, poorly named. It's not purely a coding tool: it's a tool for general computer automation. Anything you can achieve by typing commands into a computer is something that can now be automated by Claude Code. It's best described as a general agent . Skills make this a whole lot more obvious and explicit. I find the potential applications of this trick somewhat dizzying. Just thinking about this with my data journalism hat on: imagine a folder full of skills that covers tasks like the following: Congratulations, you just built a "data journalism agent" that can discover and help publish stories against fresh drops of US census data. And you did it with a folder full of Markdown files and maybe a couple of example Python scripts. Model Context Protocol has attracted an enormous amount of buzz since its initial release back in November last year . I like to joke that one of the reasons it took off is that every company knew they needed an "AI strategy", and building (or announcing) an MCP implementation was an easy way to tick that box. Over time the limitations of MCP have started to emerge. The most significant is in terms of token usage: GitHub's official MCP on its own famously consumes tens of thousands of tokens of context, and once you've added a few more to that there's precious little space left for the LLM to actually do useful work. My own interest in MCPs has waned ever since I started taking coding agents seriously. Almost everything I might achieve with an MCP can be handled by a CLI tool instead. LLMs know how to call , which means you don't have to spend many tokens describing how to use them - the model can figure it out later when it needs to. Skills have exactly the same advantage, only now I don't even need to implement a new CLI tool. I can drop a Markdown file in describing how to do a task instead, adding extra scripts only if they'll help make things more reliable or efficient. One of the most exciting things about Skills is how easy they are to share. I expect many skills will be implemented as a single file - more sophisticated ones will be a folder with a few more. Anthropic have Agent Skills documentation and a Claude Skills Cookbook . I'm already thinking through ideas of skills I might build myself, like one on how to build Datasette plugins . Something else I love about the design of skills is there is nothing at all preventing them from being used with other models. You can grab a skills folder right now, point Codex CLI or Gemini CLI at it and say "read pdf/SKILL.md and then create me a PDF describing this project" and it will work, despite those tools and models having no baked in knowledge of the skills system. I expect we'll see a Cambrian explosion in Skills which will make this year's MCP rush look pedestrian by comparison. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . Trying out the slack-gif-creator skill Skills depend on a coding environment Claude as a General Agent Skills compared to MCP Here come the Skills Where to get US census data from and how to understand its structure How to load data from different formats into SQLite or DuckDB using appropriate Python libraries How to publish data online, as Parquet files in S3 or pushed as tables to Datasette Cloud A skill defined by an experienced data reporter talking about how best to find the interesting stories in a new set of data A skill that describes how to build clean, readable data visualizations using D3

AI Open Source Programming Python Yaml

▲ 0

0 views

Massively Parallel Procrastination 10 months ago

Using GraphViz for CLAUDE.md

This is a very, very informal interim research report about something I've been playing with over the past couple days. Last Friday, I saw an absolutely astonishing extemporaneous talk by an organization that is so far down the curve of AI maximalist development that it kind of broke my brain. I'm hoping to write more about a bunch of what I saw in the coming weeks, but I'm very much still digesting what I saw. One of the things that I thought they said during the talk was that they were using .dot (GraphViz) as the language that they are using as the formalization for new processes for their coding agent. It made sense to me. There's enough dot on the Internet that models can read and understand it quite well. And it removes a whole bunch of the ambiguity of English language specifications. It turns out that I completely misunderstood what was going on, in the best possible way. They're using GraphViz, but not for that. They're using Markdown files to allow the agent to document new processes and capabilities for itself. That makes tons of sense. It's roughly the same format that Anthropic is using for Claude's new 'SKILL.md' files, which are just Markdown files with YAML frontmatter. But before I was corrected, I went home and started experimenting. And... I'm kind of excited about what I ended up with. The first thing I did was that I asked Claude to convert my CLAUDE.md into GraphViz. It worked pretty well because most of my CLAUDE.md was process documentation. The first iterations (which didn't get saved) were somewhat unhinged. My processes were inconsistent and doing this work made the problems very, very obvious. Over the course of a couple hours, Claude and I iterated on my processes and on how to use dot as specification language. There was only one absolute disaster of a hallucination when I asked Claude to update the process with what would be "100x better" and it threw in a whole bunch of Science Fiction features...that will still probably be Science Fiction 6 months from now. After about a dozen rounds of iteration, we workshopped my processes to the point where they mostly seemed to flow correctly AND the .dot document was really readable by both of us. And then I swapped it in place of my CLAUDE.md and fired up a session and...Claude behaved normally and understood my rules. I ran a couple of vibechecks, asking it for the same small project with both the traditional rules and the .dot rules. It was a very unscientific test, but I found the .dot version of Claude and its output preferable. At least in these early tests, Claude seems better at understanding and following rules written as dot. And the format makes it much easier to visualize many situations when you're giving your robot buddy rules it can't follow. We also put together a .dot styleguide to eventually let Claude more easily write its own processes. I haven't yet had a ton of experience with CLAUDE self-documenting new processes, but that's coming. (As an aside, I also have another mini-project running that's extracting learnings and memories from all my previous Claude Code sessions. But that's a story for another day. Until then, you can find it on GitHub at https://github.com/obra/claude-memory-extractor) This was my most recent CLAUDE.md before this project: This is my current CLAUDE dot md: What follows is a mini-writeup written by Claude (Opus 4.1). I made the mistake of asking it to write in my voice, but make no mistake - all the words after this point are generated. They don't tell a good story about my motivations, but the narrative does a decent job explaining the investigation process. I've been working with Claude for a while now, and I have a detailed CLAUDE.md file that contains all my rules and preferences. It's great, but it's also... a wall of text. Rules like "NEVER use git add -A" and "You're absolutely right!" is forbidden are scattered throughout. When Claude needs to follow a complex process, it's not always clear what the actual flow should be. So I had an idea: what if we could use Graphviz's dot language as a DSL for documenting processes? Not for pretty diagrams (though that's a nice side effect), but as a structured, searchable, executable process definition language that Claude can actually read and follow. First attempt - just document everything that's already in CLAUDE.md as a massive flowchart: This was... overwhelming. Too many boxes, too many connections, and honestly, Claude would never be able to follow this. It looked comprehensive but wasn't actually useful. I realized Rule #1 ("Break any rule? Stop and ask Jesse") shouldn't be a separate entry point but should be embedded throughout: Better, but still treating processes as these separate phases that don't really reflect how work actually happens. Tried to create a more unified workflow: This was starting to look more realistic, but still too academic. The "continuous processes" box was a particular cop-out - those aren't separate activities, they happen during the work. Tried to boil it down to the essence: Cleaner, but now we'd lost important detail. Also those emoji warnings didn't render properly - turns out Graphviz doesn't love Unicode. Got ambitious and created two versions - one documenting current processes, one imagining what would make Claude "superhuman": This was incredibly detailed - 7 layers of process! But when I asked myself "could Claude actually follow this?" the answer was no. Too complex. This was fantasy. Things like "confidence percentages" and "cognitive load monitoring" - Claude can't actually do these. I was designing capabilities that don't exist. Converted the "superhuman" version into things Claude could actually do: Better! Actionable steps like "Write down what's not working" instead of "Monitor cognitive load." But the task classification at the start was artificial - Claude doesn't actually know if something will take 30 minutes. Time to get real about what actually happens: This version admitted the truth: Claude often jumps into coding too quickly, forgets to search for similar code, and has specific danger zones where it fails. Much more useful! Tried organizing as a proper framework: But then I realized - these aren't really "phases." Work doesn't happen in phases, it's all interconnected with loops and branches. Getting closer to reality: This showed the actual flow better, but was still hard for Claude to parse from the dot file itself. Then came the key insight: Claude doesn't need one giant flowchart. It needs to be able to jump to the right process based on the current situation. Enter trigger-based design: Now each cluster is a self-contained process that starts with a clear trigger. Claude can search for "stuck" and find the "When Stuck" process. Much better! Then I learned you can use quoted strings as node identifiers in dot: Instead of: This transformed everything! The final readable version: Now the dot file itself is readable documentation. Claude can parse it directly without mental translation. Removed all labels and used semantic naming: With good naming, the labels are completely redundant. The diagram is cleaner and the source is more maintainable. We even created processes for maintaining the processes: The key insight: processes come from experience, not planning. Either Jesse teaches me something, I discover a pattern through repetition, or I hit a gap in existing processes. To make this systematic, we created a style guide - also written in dot! The style guide defines: Applying all these lessons: This version: Using Graphviz dot notation as a process DSL is actually brilliant because: Instead of one massive flowchart, having separate processes with clear triggers is much more useful: Using quoted strings and meaningful names makes the dot files themselves readable: Is so much clearer than: Using different shapes systematically helps Claude understand what type of thing each node is: The most useful version wasn't the most comprehensive one. It was the one that: Instead of one giant file, we could have a process library in : Each file would be a focused, trigger-based process that Claude could load when needed. We could even hook this into the slash command system - imagine loading and displaying the relevant process. What started as an attempt to visualize CLAUDE.md turned into discovering that Graphviz's dot language is actually a fantastic DSL for defining executable processes. The key wasn't making prettier diagrams - it was realizing that the text representation itself could be the documentation. The final version is something Claude can actually read, understand, and follow. It's not perfect, but it's honest about how Claude actually works and where it tends to fail. And that honesty makes it genuinely useful. Most importantly, this isn't just documentation - it's a living process definition that can evolve as we discover new patterns and better ways of working. Every time Jesse teaches me something new or I discover a pattern that works, it can become a new process in the library. The real magic is that we're using a graph description language for what it was meant for - describing flows and relationships - but in a way that's both human-readable and machine-parseable. That's the sweet spot for AI assistance tools. Different node shapes for different purposes (diamonds for decisions, octagons for warnings, etc.) Naming conventions for clarity When to add new processes How to test if a process is useful Uses semantic node shapes (plaintext for commands, diamonds for decisions, octagons for warnings) Has clear trigger points for each process Is readable both as a diagram and as source code Actually represents how Claude works, not how we wish it worked It's visual when you need it to be It's searchable text when Claude needs to find something It enforces structure (nodes and edges) It's version controllable It renders nicely for documentation "When stuck" -> go to stuck process "Test failing" -> go to debug process "New request" -> go to request process for decisions for literal commands to run for critical warnings Admitted where Claude actually fails Included only processes that actually get used Used simple, clear language Could be followed mechanically

Programming AI Yaml

▲ 0

0 views

ENOSUCHBLOG 10 months ago

Dear GitHub: no YAML anchors, please

TL;DR : for a very long time, GitHub Actions lacked support for YAML anchors. This was a good thing . YAML anchors in GitHub Actions are (1) redundant with existing functionality, (2) introduce a complication to the data model that makes CI/CD human and machine comprehension harder, and (3) are not even uniquely useful because GitHub has chosen not to support the one feature (merge keys) that lacks a semantic equivalent in GitHub Actions. For these reasons, YAML anchors are a step backwards that reinforces GitHub Actions’ status as an insecure by default CI/CD platform. GitHub should immediately remove support for YAML anchors, before adoption becomes widespread. GitHub recently announced that YAML anchors are now supported in GitHub Actions. That means that users can write things like this: On face value, this seems like a reasonable feature: the job and step abstractions in GitHub Actions lend themselves to duplication, and YAML anchors are one way to reduce that duplication. Unfortunately, YAML anchors are a terrible tool for this job. Furthermore (as we’ll see) GitHub’s implementation of YAML anchors is incomplete , precluding the actual small subset of use cases where YAML anchors are uniquely useful (but still not a good idea). We’ll see why below. Pictured: the author’s understanding of the GitHub Actions product roadmap. The simplest reason why YAML anchors are a bad idea is because they’re redundant with other more explicit mechanisms for reducing duplication in GitHub Actions. GitHub’s own example above could be rewritten without YAML anchors as: This version is significantly clearer, but has slightly different semantics: all jobs inherit the workflow-level . But this, in my opinion, is a good thing : the need to template environment variables across a subset of jobs suggests an architectural error in the workflow design. In other words: if you find yourself wanting to use YAML anchors to share “global” configuration between jobs or steps, you probably actually want separate workflows, or at least separate jobs with job-level blocks. In summary: YAML anchors further muddy the abstractions of workflows, jobs, and steps, by introducing a cross-cutting form of global state that doesn’t play by the rules of the rest of the system. This, to me, suggests that the current Actions team lacks a strong set of opinions about how GitHub Actions should be used, leading to a “kitchen sink” approach that serves all users equally poorly. As noted above: YAML anchors introduce a new form of non-locality into GitHub Actions. Furthermore, this form of non-locality is fully general : any YAML node can be anchored and referenced. This is a bad idea for humans and machines alike: For humans: a new form of non-locality makes it harder to preserve local understanding of what a workflow, job, or step does: a unit of work may now depend on any other unit of work in the same file, including one hundreds or thousands of lines away. This makes it harder to reason about the behavior of one’s GitHub Actions without context switching. It would only be fair to note that GitHub Actions already has some forms of non-locality: global contexts, scoping rules for blocks, dependencies, step and job outputs, and so on. These can be difficult to debug! But what sets them apart is their lack of generality : each has precise semantics and scoping rules, meaning that a user who understands those rules can comprehend what a unit of work does without referencing the source of an environment variable, output, &c. For machines: non-locality makes it significantly harder to write tools that analyze (or transform) GitHub Actions workflows. The pain here boils down to the fact that YAML anchors diverge from the one-to-one object model 1 that GitHub Actions otherwise maps onto. With anchors, that mapping becomes one-to-many: the same element may appear once in the source, but multiple times in the loaded object representation. In effect, this breaks a critical assumption that many tools make about YAML in GitHub Actions: that an entity in the deserialized object can be mapped back to a single concrete location in the source YAML. This is needed to present reasonable source locations in error messages, but it doesn’t hold if the object model doesn’t represent anchors and references explicitly. Furthermore, this is the reality for every YAML parser in wide use: all widespread YAML parsers choose (reasonably) to copy anchored values into each location where they’re referenced, meaning that the analyzing tool cannot “see” the original element for source location purposes. I feel these pains directly: I maintain zizmor as a static analysis tool for GitHub Actions, and makes both of these assumptions. Moreover, ’s dependencies make these assumptions: (like most other YAML parsers) chooses to deserialize YAML anchors by copying the anchored value into each location where it’s referenced 2 . One of the few things that make YAML anchors uniquely useful is merge keys : a merge key allows a user to compose multiple referenced mappings together into a single mapping. An example from the YAML spec, which I think tidily demonstrates both their use case and how incredibly confusing merge keys are: I personally find this syntax incredibly hard to read, but at least it has a unique use case that could be useful in GitHub Actions: composing multiple sets of environment variables together with clear precedence rules is manifestly useful. Except: GitHub Actions doesn’t support merge keys ! They appear to be using their own internal YAML parser that already had some degree of support for anchors and references, but not for merge keys. To me, this takes the situation from a set of bad technical decisions (and lack of strong opinions around how GitHub Actions should be used) to farce : the one thing that makes YAML anchors uniquely useful in the context of GitHub Actions is the one thing that GitHub Actions doesn’t support. To summarize, I think YAML anchors in GitHub Actions are (1) redundant with existing functionality, (2) introduce a complication to the data model that makes CI/CD human and machine comprehension harder, and (3) are not even uniquely useful because GitHub has chosen not to support the one feature (merge keys) that lacks a semantic equivalent in GitHub Actions. Of these reasons, I think (2) is the most important: GitHub Actions security has been in the news   a great deal recently , with the overwhelming consensus being that it’s too easy to introduce vulnerabilities in (or expose otherwise latent vulnerabilities through ) GitHub Actions workflow. For this reason, we need GitHub Actions to be easy to analyze for humans and machine alike. In effect, this means that GitHub should be decreasing the complexity of GitHub Actions, not increasing it. YAML anchors are a step in the wrong direction for all of the reasons aforementioned. Of course, I’m not without self-interest here: I maintain a static analysis tool for GitHub Actions, and supporting YAML anchors is going to be an absolute royal pain in my ass 3 . But it’s not just me: tools like actionlint , claws , and poutine are all likely to struggle with supporting YAML anchors, as they fundamentally alter each tool’s relationship to GitHub Actions’ assumed data model. As-is, this change blows a massive hole in the larger open source ecosystem’s ability to analyze GitHub Actions for correctness and security. All told: I strongly believe that GitHub should immediately remove support for YAML anchors in GitHub Actions. The “good” news is that they can probably do so with a bare minimum of user disruption, since support has only been public for a few days and adoption is (probably) still primarily at the single-use workflow layer and not the reusable action (or workflow) layer. That object model is essentially the JSON object model, where all elements appear as literal components of their source representation and take a small subset of possible types (string, number, boolean, array, object, null).  ↩ In other words: even though YAML itself is a superset of JSON, users don’t want YAML-isms to leak through to the object model. Everybody wants the JSON object model, and that means no “anchor” or “reference” elements anywhere in a deserialized structure.  ↩ To the point where I’m not clear it’s actually worth supporting anchors to any meaningful extent, and instead immediately flagging them as an attempt at obfuscation.  ↩ For humans: a new form of non-locality makes it harder to preserve local understanding of what a workflow, job, or step does: a unit of work may now depend on any other unit of work in the same file, including one hundreds or thousands of lines away. This makes it harder to reason about the behavior of one’s GitHub Actions without context switching. It would only be fair to note that GitHub Actions already has some forms of non-locality: global contexts, scoping rules for blocks, dependencies, step and job outputs, and so on. These can be difficult to debug! But what sets them apart is their lack of generality : each has precise semantics and scoping rules, meaning that a user who understands those rules can comprehend what a unit of work does without referencing the source of an environment variable, output, &c. For machines: non-locality makes it significantly harder to write tools that analyze (or transform) GitHub Actions workflows. The pain here boils down to the fact that YAML anchors diverge from the one-to-one object model 1 that GitHub Actions otherwise maps onto. With anchors, that mapping becomes one-to-many: the same element may appear once in the source, but multiple times in the loaded object representation. In effect, this breaks a critical assumption that many tools make about YAML in GitHub Actions: that an entity in the deserialized object can be mapped back to a single concrete location in the source YAML. This is needed to present reasonable source locations in error messages, but it doesn’t hold if the object model doesn’t represent anchors and references explicitly. Furthermore, this is the reality for every YAML parser in wide use: all widespread YAML parsers choose (reasonably) to copy anchored values into each location where they’re referenced, meaning that the analyzing tool cannot “see” the original element for source location purposes. I feel these pains directly: I maintain zizmor as a static analysis tool for GitHub Actions, and makes both of these assumptions. Moreover, ’s dependencies make these assumptions: (like most other YAML parsers) chooses to deserialize YAML anchors by copying the anchored value into each location where it’s referenced 2 . That object model is essentially the JSON object model, where all elements appear as literal components of their source representation and take a small subset of possible types (string, number, boolean, array, object, null).  ↩ In other words: even though YAML itself is a superset of JSON, users don’t want YAML-isms to leak through to the object model. Everybody wants the JSON object model, and that means no “anchor” or “reference” elements anywhere in a deserialized structure.  ↩ To the point where I’m not clear it’s actually worth supporting anchors to any meaningful extent, and instead immediately flagging them as an attempt at obfuscation.  ↩

DevOps JSON Ci Cd Yaml

▲ 0

0 views