I am so tired of this bullshit
some strong feels as I try to navigate this.
some strong feels as I try to navigate this.
I’m going to try to avoid falling into the trap of making a bad pun about the season…damnit! ;-) Welcome to the September edition of Interesting Links in the Data and AI World . It’s a bumper edition (aren’t they always), with lots and lots about Kafka and related technologies in particular this month. Oh, and AI. Obvs.
Having clankers churn out content to feed the clankers is a futile game to play. It’s an arms-race to the bottom of a cesspit.
June has been a busy month—113 links below for your enjoyment and delectation. I’m going to share one extra link up here with you though, but it’s not my fault if it wrecks your productivity! My friend Kris Jenkins has written this devishly simple but addictive browser-based game: Escape the Moon .
Welcome to May’s Interesting Links ! This month saw the Current conference in London with the usual 5k run , lots of familiar faces and friendly conversations—and plenty of excellent breakout sessions too. It seems live-tweeting conferences isn’t a thing any more, with only myself and Thomas Cooper seeming to post anything, but if you want you can go review the hashtag feed on BlueSky for some highlights of the conference.
Like a young child coming home from kindergarten with their latest crayon scrawls, the internet is currently awash with people sharing their AI-generated work. And just like the young child’s drawings, much of that work should be proudly put up on the walls within the artist’s house—and no further.
Another year, another Current—another 5k run/walk for anyone who’d like to join! Did I mostly copy-and-paste this from last year’s post ? You bet I did!
It’s now a joy to simply read any blog post that’s not AI-generated.
A bit of a streamlined edition, this month. Lots of interesting links still, but less commentary. You can put that down to me prevaricating on getting my previous blog about Materialized Tables in Apache Flink finished, and leaving myself little time to work on this one :) Not including the detailed narration actually knocks a bunch of time off the preparation—I’d be interested in your feedback as to how much the absence of narration impacts (if at all) your enjoyment of reading it. Let me know in the comments below! Something that I’m slowly changing is how I categorise links to do with AI. A few months back anything "AI" got its own section. It wasn’t much more than a novelty really; certainly not something worth distracting the regular link sections with. But now AI is just part-and-parcel of many people’s workflows, a regular component in their toolbox. So where an article is about credibly using AI as part of an existing topic (such as data engineering), I’ll file it in that section. (And if this news makes you cross because you abhor anything AI, well, I’ve got news for you ) .
Flink added support for what it calls Materialized Tables in 1.20 , released in 2024. You can read about the design and motivations in FLIP-435 . In a nutshell, Materialized Tables provide a way to include the SQL to populate and refresh a table as part of its definition.
As a non-Java coder, for the last ten years I’ve stumbled my way through the JVM-centric world of "big data" (as it was called then), relying on my wits with SQL and config files to just about muddle through. One of the things that drew me to Kafka Connect was that I could build integrations between Kafka and other systems without needing to write Java, and the same again for ksqlDB and Flink SQL—now stream processing was available to mere RDBMS mortals and not just the Java adonises. One thing defeated me though; if a connector didn’t exist for Kafka Connect, then I was stuck. I’d resort to cobbled-together pipelines leaning heavily on kafkacat kcat, such as I did in this blog post . I built some cool analytics on top of maritime AIS data about ships' locations, but the foundations were shaky at best: No failure logic, no schema handling, no bueno. What I really needed was a connector for Kafka Connect. However for that, you need Java. I don’t write Java. But Claude can write Java.
I’ve had a huge amount of fun this month exploring quite what AI (in the form of Claude Code) can do for a data engineer. Rather than just hack around at a prompt, I took a bit more of a considered approach to it, building a harness to test out different prompts and skills. You can read my write-up here, the headline of which is that literally Claude Code isn’t going to replace data engineers (yet) . I’ve also written up an AI Disclosure for my blog which I’ll keep up to date as my use of AI evolves, along with a sweary rant about why you basically have to get on board with AI if you value your career.
I wanted to explore the extent to which Claude Code could build a data pipeline using dbt without iterative prompting. What difference did skills, models, and the prompt itself make? I’ve written in a separate post about what I found ( yes it’s good; no it’s not going to replace data engineers, yet ). In this post I’m going to show how I ran these tests (with Claude) and analysed the results (using Claude), including a pretty dashboard (created by Claude):
I use AI heavily on this blog. I don’t use AI to write any content. As any followers of my blog will have seen recently, I am a big fan of the productivity —and enjoyment—that AI can bring to one’s work. (In fact, I firmly believe that to opt out of using AI is a somewhat negative step to take in terms of one’s career.) Here’s how I don’t use AI, and never will : I use AI heavily on this blog. I don’t use AI to write any content.
Ten years late (but hopefully not a dollar short ) I recently figured out what all the fuss about dbt is about . No it’s not (at least, not yet). In fact, used incorrectly, it’ll do a worse job than you. But used right, it’s a kick-ass tool that any data engineer should be adding to their toolbox today * . In this article I’ll show you why.
This is an addendum to the main post about using Claude Code with dbt . It shows an excerpt of a Claude session log so you can see exactly what goes on "under the covers" . For full details of the prompt, commentary, and conclusions, see Claude Code isn’t going to replace data engineers (yet) . Here we can see the steps that Claude Code takes as it figures out for itself anomalies in the data and adapts the dbt model to handle them.
AI slop is ruining the internet . Given half a chance AI will delete your inbox or worse (even if you work in Safety and Alignment at Meta): Nothing humbles you like telling your OpenClaw “confirm before acting” and watching it speedrun deleting your inbox. I couldn’t stop it from my phone. I had to RUN to my Mac mini like I was defusing a bomb. pic.twitter.com/XAxyRwPJ5R AI slop is ruining the internet .
Phew, what a month! February may be shorter but that’s not diminished the wealth of truly interesting posts I’ve found to share with you this month.
Funnily enough, Charles Dickens was talking about late 18th century Europe rather than the state of AI and LLMs in 2026, but here goes: It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of light, it was the season of darkness, it was the spring of hope, it was the winter of despair. For the last few weeks I’ve been coming back to this quotation, again and again. It is the best of times (so far) for AI—you can literally describe an idea for a program or website, and it’s generated for you . Hallucinations are becoming fewer. This is so much more than simply guessing the next word. Honestly, it’s a sufficiently advanced technology that really is indistinguishable from magic (with apologies to Arthur C. Clarke). Whether I’d call this the age of wisdom…I’m not sure yet ;) But at the same time… it is the worst of times, the age of foolishness, season of darkness. Bot-farms spewing divisive nonsense all over social media no longer need to copy and paste their false statements in a way that’s easily spotted; instead they can write custom text at scale whilst still giving the illusion of a real person behind the fake accounts. Combine human greed with the speed at which LLMs can generate content and you have an infinite flow of slop spurting all over the internet like a farmer’s muck spreader gone awry at scale. AI voice agents are becoming better and used for scamming people with realistic and targeted calls that would previously have been uneconomical to do at the scale necessary to reap a reward. AI-generated pictures are being used to create hoaxes and flood social media with dangerous rage-baiting. It might be the best & worst of times, but that doesn’t mean you have to pick sides. Having lived through the advent of cloud computing to where it is now, I can see real parallels in how developers in the tech industry are approaching it. Some, particularly vendors & VCs, are "all in". Others believe it’s a fad or a straight-up con and will give you a list of reasons why that is. Both extremes are utterly and completely wrong. If you’re the kind of Bare Metalsson character who believed the cloud was nonsense ( ) and took pride in racking your own servers (each of which has its own cute name), you’re probably also burying your head in the sand when it comes to using LLMs with cries of . And, just as running a homelab with servers and containers named after Star Wars characters is fun but you wouldn’t use the same approach at work, refusing to acknowledge that AI today has the potential to make you more productive as a developer starts to look somewhat childish or irresponsible. Just because AI makes shit up sometimes , it doesn’t mean that AI is not therefore ever a useful tool for the right job . Strikingly, what’s happened in the last month or two is that the list of jobs for which you can use it has suddenly grown drastically. The online chatter has moved from " omg you wouldn’t let an LLM code for you " to " omg how do we review all these PRs ", because guess what: all of a sudden people are letting an LLM generate code for them. AI, and specifically LLMs, are a valuable tool for developers, and it’s one that we need to recognise if we’re not to get left behind . Picture a Capuchin monkey sat on its haunches using a stone to crack open a nut . Rudimentary, but effective. Would we as developers use a stone when we needed a hammer to bang in a nail? No, that would be stupid—we use the right tool for the job, of course. Hammers are an evolution of the tool from a crude stone, and we use that because it’s the best tool for the job. But once the hammer drill came along, do we cling to our manual hammer when we’ve got a nail to bang into a brick wall? Again, no, that would be stupid. We want to use the best tool for the job . It’s the same evolution of tooling happening in AI. LLMs are a tool. Magical, bamboozling, hilariously-wrong at times tools; but ones that are evolving not over centuries or longer, but weeks and months. Some people fundamentally object to LLMs on principle, citing their use of resources, or threat to mankind. Personally, I believe that cat is out of the bag, the horse has bolted the stables…we’re way past that. Pandora’s box is open, and you and I are not shutting it. What I would observe is that if you’re working in IT, and you’re not already adopting AI and understanding what it can (and can’t) do for you, you might find yourself with a lot more time to discuss these opinions alongside the hansom cab drivers who figured that the motor engine was a fad and stuck with their horses. Put somewhat more confrontationally: you may as well be against the internet, or the combustion engine, or atomic energy. All have awful uses and implications; all also serve a role that cannot be overstated. What LLMs are enabling is truly of seismic impact, and I cannot fathom a path forward in which they do not continue to be central to how we do things with computers. Not convinced by my reasoning above? How about these folk: You can't let the slop and cringe deny you the wonder of AI. This is the most exciting thing we've made computers do since we connected them to the internet. If you spent 2025 being pessimistic or skeptical on AI, why not give the start of 2026 a try with optimism and curiosity? Not a fan of DHH? How about Charity Majors : this year was for AI what 2010 was for the cloud: the year when AI stopped being satellite, experimental tech and started being the mainstream, foundational technology. At least in the world of developer tools. It doesn’t mean there isn’t a bubble. Of COURSE there’s a fucking bubble. Cloud was a bubble. The internet was a bubble. Every massive new driver of innovation has come with its own frothy hype wave. But the existence of froth doesn’t disprove the existence of value. Or Sam Newman : To those of you who are deeply pessimistic around the use of AI in software delivery, the old quote from John Maynard Keynes comes to mind: "The market can remain irrational longer than you can remain solvent". For a considered look at the uses of LLMs, Bryan Cantrill wrote an excellent RFD: Using LLMs at Oxide Read the above linked articles, and also check out Scott Werner’s post "The Only Skill That Matters Now" which puts it even more clearly into focus, with a nice analogy about how "skating to the puck" is no longer a viable strategy. The long and short of it is that the rate of change in AI means you have no idea where the puck will even be. I read an article a while back that I found again here , in which a hospital consultant described their view of LLMs thus: "Think of it as the most brilliant, talented, often drunk intern you could imagine," This was in May 2023 (eons ago, in LLM years). As an end user of LLMs, I think this mental model really does work. If you, as a senior+ developer, think of an LLM as a very eager junior developer working for you. They’re fresh-eyed and bushy-tailed, and goddamnit they talk too much, don’t listen enough, and make stupid mistakes. But…you give them a job to do, point them in the right direction, and iterate with them under close supervision …and suddenly you’re finding yourself a lot more productive. Tutored well, a junior developer becomes a force-multiplier, a mini-me. A common instinct amongst inexperienced senior+ developers tasked with looking after a junior can unfortunately be "I’ve not got time to show them this, I’ll do it myself". As any decent developer knows, that’s a short-sighted and flawed way of developing others (as well as oneself). Mentoring and teaching and nurturing juniors is one step back, two steps forward. And…the same goes for an LLM. Do you have to keep telling them the same thing more than once? Yes. Do they write code that drives you into fits of rage with its idiocy and overcomplexity? Yes. Do they improve each time and ultimately give you more time on your plate to think about the bigger picture of system design and implementation ? Yes. I’m not intending to imply—as some may take from this—that in drawing the analogy I am actually suggesting we replace junior developers with AI. After all, junior developers learn, and in time become the senior developers who know when Claude is talking bollocks—that pipeline matters. Rather, I’m trying to characterise how one may look at the tool and one’s interactions with it. I am also leaving wide open the issue of what the impact of AI on junior developers themselves actually could be. The consequences for the software industry are likely to be vast. Commenting on this is beyond my experience—and there is also plenty being written elsewhere about it. Working with Claude Code over the past few weeks really has got me convinced that we’ve now taken a step forward where time invested in learning how to use it (because there is a learning curve) is time that’s well spent. Previously, using an LLM was not much more than typing (or various cargo-culting "prompt engineering" techniques). Now you have to learn about context windows and the magical file called and prompting to get the most out of it for coding, and that’s ok. Some tools are simple (pick up a hammer and hit something) and others require more understanding (I’m not using a chainsaw anytime soon without training on it first). Junior developers are humans. They get tired, they need rest breaks, they need feeding, and at some point they want to go home. LLMs, on the other hand, will keep on going so long as you keep feeding them tokens. The impact of this on you as their boss is substantial. You might task your junior developer with a piece of work and they’ll return to you later that day, perhaps with a few interruptions to clarify a point. Claude Code, on the other hand, is like an eager puppy, bounding back and forth demanding your attention often every minute or so. I’m still trying to work out how to balance the dopamine hit of each interaction bringing another astounding chunk of functionality delivered, with the impact the rapid context switching has on my brain. Interacting with Claude Code feels a bit like the hit we get from scrolling short video feeds. One more prompt…one more video… Because the feedback loop is so fast, it’s also very easy to get drawn down a rabbit hole of changes and either end up on a side-quest from one’s intended task, or lose sight of the big picture and end up meandering aimlessly through some Frankenstein-like development path that feels fruitful because of the near-instantaneous results but which is ultimately flawed. That’s it. Go fuck around, and find out. Exciting things are happening. Yes the hype and BS is real and nauseating; but that doesn’t stop it being true. If you’re interested in the F’ing around and what I Found Out, have a look at the companion post to this one: Cosplaying as a webdev with Claude Code in January 2026 .
In which Claude and [A]I play at being webdevs. For some reflections on the bigger picture of AI as a productivity tool for developers, have a look at the companion post to this one . I used to speak at a lot of conferences and meetups, and published my talks on a site called . It’s free to use, but you could pay for bells and whistles including a custom domain, which I duly did: . My background is databases and SQL; I can spell HTML (see, I just did) and am aware of CSS and can fsck about in the Chrome devtools to fiddle with the odd detail…but basically frontend webdev is completely beyond me. That meant I was more than happy to pay someone else to host my talks for me on an excellent platform. This was a few years ago, and the annual renewal of the plan was starting to bite—over £100 for what was basically static content that I barely ever changed (I’ve only done three talks since 2021). So I decided to see what Claude Code/Opus 4.5 could do, and signed up for the £18/month "Pro" plan. The way Claude Code works is nothing short of amazing. You use natural language to tell it what to do…and it does it. I started off by saying to ( prompting ) it with something like this: Claude Code then poked around the two sites and probably asked me some questions (did I want to import all content, what kind of style, etc), and then spat out a Python script to do a one-time ingest of all the content from noti.st. After seeking permission it then ran the Python script, debugged the errors that were thrown, until it was happy it had a verbatim copy of the data. Along the way it’d report in on what it was doing and I could steer it—much the same way you would a junior developer. For example, on noti.st a slide deck’s PDF is exploded out into individual images so that a user can browse it online. This meant a crap-ton of images which I didn’t care about, but Claude Code assumed I would so started grabbing them. Claude then proceeded to build and populate a site to run locally. There were plenty of mistakes, as well as plenty of yak-shaving ("hmm can you move this bit to there, and change the shade of that link there"). This can be part of the danger with Claude. It will never roll its eyes and sigh at you when you ask for the hundredth amendment to your original spec, so it’s easy to get sucked into endless fiddling and tweaking. I found I quickly burnt through my Pro token allowance, which actually served well as a gatekeeper on my time, forcing me to step back until the tokens were refreshed. After four early morning/late nights around my regular work, I cut over my DNS and you can see the results at https://talks.rmoff.net/ . The key things that Claude Code did that I’d not been able to get ad hoc chat sessions (or even Cursor) to do last year include: Planning out a full project like this one, from the overview down to every detail Talking the talk (writing the code) and walking the walk (independently running the code, fixing errors, evaluating logic problems, etc) Rapidly iterating over design ideas, including discussing them and not just responding one-way to instructions Discussing deployment options, including working through challenges given the cumulative size of the PDFs Explaining and building and executing and testing the deployment framework Before the sceptics jump in with their , my point is not that I couldn’t theoretically have done this without Claude. It’s that it took, cumulatively, perhaps eight hours—and half of that will have been learning how to effectively interact with Claude. It’s that it’s a single terminal into which one types, that’s it. No explosion of tabs. No rabbit-holes of threads trying to figure this stuff out. One place. That fixes its own errors. That writes code that you could never have done without a serious investment of time. Would I apply for a frontend engineering job? Heck no! Does my new site stand up to scrutiny? Probably not. Will real frontend devs look at the code and be slightly sick in their mouths? Perhaps. Does this weaken my point? Not in the slightest! £18-worth of Claude Code (less, if you pro-rata it over the month) and I’ve saved myself an ongoing annual bill of £100, built a custom website that looks exactly as I want it, has exactly the functionality that I want—oh, and was a fuck-ton of fun to build too :) Not whilst I have access to Claude ;) I realise that in reading this the choler will be rising in some seasoned software engineers. After all, who is this data engineer poncing around pretending to build websites? And that’s perhaps the crux of it: I’m a data engineer, branching out into something I couldn’t do before, courtesy of Claude. I would definitely use Claude to help me write SQL queries and generate DDL, but I’d be damned if I’d put my name to a pull request with a single byte of code that I couldn’t explain—because that’s my job . I like Oxide’s words here: However powerful they may be, LLMs are but a tool, ultimately acting at the behest of a human. Oxide employees bear responsibility for the artifacts we create, whatever automation we might employ to create them . So I can have fun building a website that’s just my personal site and only on me if it fails. But if I’m writing code as a professional for my job, it’s on me to make sure that it’s code I can put my name to. There is a lot written about Claude Code. Some of it cargo-culting, some of it frothy-hype nonsense. Below I’ve listed out some of my 'top tips' that I’d be passing onto a colleague getting into Claude Code from scratch tomorrow. If you’re doing any kind of webdev work, follow Kris Jenkins' tip and use Playwright so that Claude can "see" as it develops. You can manually take screenshots and paste those into Claude too if you want (including ones you’ve annotated with observations and instructions) but in general and particularly for regression testing, Playwright is an excellent addition. Because this is Claude, you don’t need to actually know how to configure Playwright or run its tests, or anything like that. You just tell Claude: "Use Playwright to test the changes". And it does. Oh, and it’ll install it for you if you don’t have it already. Claude will sometimes ask for permission to do something, or tell you it’s finished its current task. If you’ve got it sat in a terminal window behind your other work you may not realise this, so adding a sound prompt can be useful. In your include: Obviously, you can waste a lot of time customising it to use just the right sound effect from your favourite 1980s arcade game. Depending on how you pay for Claude (fixed plans, or per API calls) you’ll discover sooner or later that it can be quite expensive. You can include the cost of the current session in the status line by adding this to the same config file as above, : It’ll look something like this: Also check out which uses the Claude log data to calculate usage and break it down in different ways which can help you optimise your use of it. Different Claude models (Opus, Sonnet, Haiku) cost different amounts, and you can optimise your spend by learning a bit about their relative strengths. I found that asking Claude itself was useful; using Opus (the most capable model) you can describe what you’re going to want it to do, and which model it would recommend. Like all of this LLM malarky, none of it is absolute, but I found its recommendations useful (i.e. the models it recommended were cheaper and did achieve what I needed them to). Think of it as having different pairs of running shoes in your closet—different ones are going to be suited to different tasks. You’re not going to wear your $200 carbon-plate running shoes to kick the ball around the park, are you? Go read up on things like: Context windows—what the LLM knows about what you’re doing Context rot—the more that’s in the LLM’s context window, the less effective it can sometimes become —where Claude makes a note of what it is you’re building and core principles, toolsets, etc You can get a lot of value by spending some time on this so that you can restart your session when you need to (e.g. to clear the context window) without having Claude 'forget' too much of the basics of what you’ve told it Work with Claude on this file—literally say, look at your CLAUDE.md, I have to keep telling you to do x , how can you remember it better. If you give it permission, it’ll then go and update its own file Use plan mode and accept-change (shift-tab) judiciously. If you just YOLO it and accept changes without seeing the plan you’ll often end up with a very busy fool going in the wrong direction. Claude is your servant (for now) and it’s up to you to boss it around firmly as needs be. Watch out for Claude spinning its wheels—if you see it trying to repeatedly fix something and getting stuck you might be burning a ton of tokens on something that it’s misunderstood or doesn’t actually matter I’ve been experimenting with a few non-coding examples, both pairing Claude with basic-memory and an Obsidian vault. Proofreading my blog ( here’s the prompt , if you’re interested; PRs welcome 😉). I also have a Raycast AI Preset to do this, but am finding myself more and more reaching for Claude’s terminal window. It works well because I write my blog posts in Asciidoc, which Claude can read and edit directly (if I ask it to). Claude can also help you write the prompt/skill—I gave it verbatim some feedback I got from a real human being on the initial version of this post, and it distilled that into what it ought to check for next time and updated its skill . Neat. Planning a holiday. Iteratively building up with Claude a spec that captures the requirements of the holiday, it can then help with itineraries, checklists, discuss areas, etc etc. As with the coding project above, it being one window with which to interact is really powerful. Acting as a running coach. Plugging in Garmin and Strava data via MCP I can capture all of my running and health info, and discuss with Claude planned workouts, even weaving in notes from past physio appointments. Obviously I am not following it blindly but as an exercise (geddit?!) in integration and LLMs, it’s pretty fun . This post may well have you spitting coffee into your cornflakes, I realise that. For some reflections on the bigger picture of AI as a productivity tool for developers, have a look at the companion post to this one . Planning out a full project like this one, from the overview down to every detail Talking the talk (writing the code) and walking the walk (independently running the code, fixing errors, evaluating logic problems, etc) Rapidly iterating over design ideas, including discussing them and not just responding one-way to instructions Discussing deployment options, including working through challenges given the cumulative size of the PDFs Explaining and building and executing and testing the deployment framework Context windows—what the LLM knows about what you’re doing Context rot—the more that’s in the LLM’s context window, the less effective it can sometimes become —where Claude makes a note of what it is you’re building and core principles, toolsets, etc You can get a lot of value by spending some time on this so that you can restart your session when you need to (e.g. to clear the context window) without having Claude 'forget' too much of the basics of what you’ve told it Work with Claude on this file—literally say, look at your CLAUDE.md, I have to keep telling you to do x , how can you remember it better. If you give it permission, it’ll then go and update its own file Use plan mode and accept-change (shift-tab) judiciously. If you just YOLO it and accept changes without seeing the plan you’ll often end up with a very busy fool going in the wrong direction. Claude is your servant (for now) and it’s up to you to boss it around firmly as needs be. Watch out for Claude spinning its wheels—if you see it trying to repeatedly fix something and getting stuck you might be burning a ton of tokens on something that it’s misunderstood or doesn’t actually matter Proofreading my blog ( here’s the prompt , if you’re interested; PRs welcome 😉). I also have a Raycast AI Preset to do this, but am finding myself more and more reaching for Claude’s terminal window. It works well because I write my blog posts in Asciidoc, which Claude can read and edit directly (if I ask it to). Claude can also help you write the prompt/skill—I gave it verbatim some feedback I got from a real human being on the initial version of this post, and it distilled that into what it ought to check for next time and updated its skill . Neat. Planning a holiday. Iteratively building up with Claude a spec that captures the requirements of the holiday, it can then help with itineraries, checklists, discuss areas, etc etc. As with the coding project above, it being one window with which to interact is really powerful. Acting as a running coach. Plugging in Garmin and Strava data via MCP I can capture all of my running and health info, and discuss with Claude planned workouts, even weaving in notes from past physio appointments. Obviously I am not following it blindly but as an exercise (geddit?!) in integration and LLMs, it’s pretty fun .