I was down in Sesto a few days ago for Apparat ’s concert. The new album is great, attending the event with family and a few friends was a very enjoyable experience, and the atmosphere was very blue! Thank you for keeping RSS alive. You're awesome.
The batter for verifiable data is increasingly defining the AI race, from Meta to Grok to the frontier labs.
For a while I've been planning to put together a separate machine for local LLM training. Until now, I've been using my desktop PC, . I have an RTX 3090 installed, and can get useful training runs done (most recently, a 163M-parameter GPT-2 small style LLM in JAX ), but there are a couple of problems. And relatedly to all of those: the two-day limit to the training runs I've been doing is something I set because that's the maximum amount of time I'm willing to have tied up. It would be really interesting to try longer training runs! I also have longer-term plans; a multi-GPU box would be interesting to put together -- not just to have more power locally, but so that I could test larger-scale cloud multi-GPU training runs before starting to pay for expensive machines. US$15.92 an hour to rent a machine isn't a lot of money, but it adds up, especially if you're spending it while debugging parallelism issues. And finally, I've always been interested in putting together a custom water-cooling loop in a PC. I've been building my own machines since 1995 or so, but never got round to that side of things. It sounds fun! But despite all of those future plans, this is a fairly normal machine-building post -- how I repurposed an old PC, plugged in a second-hand RTX 3090 from eBay, tested it all, accidentally trained an LLM for 11 days, and almost cooked a CPU. Over time, I expect to be posting more -- and more interesting -- build details. Let's think of this as establishing the baseline. Back before I moved to Lisbon, we had a holiday home here. When we came over, I'd bring my laptop, but that was always somewhat unsatisfactory -- limited CPU power for work, limited GPU for my occasional gaming. During Covid, we started staying in the holiday home for longer periods -- and this became too big of an annoyance to ignore. So in 2020 I put together a small form-factor PC, which I named . The constraints were: The build was a bit fiddly, like all SFF PCs. You can see the component list and build notes here on PCPartPicker , but in short she had: She looked like this: (Gosh, I'd forgotten how... vivid our wallpaper was in that dining room.) For scale -- that case is slightly taller than two cans of coke stacked on top of each other. So, pretty small. When we moved to Lisbon full-time, I brought with me from London, and while he's been upgraded several times since (including adding an RTX 3090 in late 2023 ), he's been my daily driver since. So sat in the corner of my study, sad and unused :-( It was time to bring her out again. Initial plan: get her up and running in a new, larger case, with a PSU that could potentially handle three graphics cards. Initially, I found that she wouldn't switch on: a quick check suggested that the problem was the PSU. I'd had problems with SFF PSUs in the past, and given that the plan was to give her a new one, I just got one, along with a new, larger case -- specifically: A few days later, the parts arrived. Here's a family photo: is to the left, centre, sitting on top of her new case, and Cornélia (wearing her Flower of Shame) is to the right. For scale, Cornélia is quite a large cat. (I appreciate that that is not immensely helpful.) Time to put the old motherboard and the new PSU into the new case. Here's what it looked like: The Mini-ITX motherboard in a case designed for full ATX looks comically like a postage stamp. I switched her on, and luckily enough, everything worked! Must have been a PSU issue. The OS that she had was a more than three-year-old version of Arch, so I wiped the drives and installed the most recent version with my normal config, and it was time for a quick test. One of the nice things about having done all of this LLM training stuff recently is that you have a ready-made burn-in test for new hardware :-) I didn't have my JAX training code yet, but I did have the PyTorch one . Now, with her GTX 1660 Super GPU, was clearly not going to be able to train an LLM of the size I could with 's RTX 3090. I did some fiddling around with the model and training run parameters, and found that I could fit in a cut-down version of GPT-2 small with this setup: I trained it with a microbatch size of 4, gradient accumulation over 16 steps, and all other hyperparameters the same as my normal training runs on . The number of training tokens went down -- the model had 76,933,120 parameters, so I needed to train for just over 20x that -- about 1.5B instead of the 3.2B I've been training my other models on. I kicked that off, and out of interest, I kicked off another training run on with the same setup to see what happened. The training run went normally -- GPU running at full blast, 368W, and it completed in about 9 hours. That's less than 1/4 of the time my normal training runs take, which makes sense because time taken for this kind of thing scales roughly linearly with both the size of the model and the number of tokens, and both of those were about half the normal size. was a bit more interesting. In , the GPU usage showed up as 100%, but with an "effective" utilisation of 53%. The power draw matched the latter, being 67W out of a total possible 125W. I'm not quite sure what was causing that -- clearly there was a bottleneck somewhere. Not really worth digging into, though, given that I was going to replace the card shortly. Anyway, that took 963,257 seconds to run. That's 267.57 hours, or just over 11 days. What's kind of interesting there is that this training run not only took much longer (which is only to be expected), but that it used more electricity. 67W over 267.57 hours is just short of 18kWh, whereas 368W over 9 hours is about 3.3kWh. Buy an RTX 3090, save the planet! I decided to run my normal evals to confirm that what had come out the other end was sane. When asked to complete "Every effort moves you", 's model said: And 's said: Those were actually rather good, I thought! And looking at my normal loss test confirmed that the models really weren't that bad; 's got 3.855702, and 's 3.855981. That was actually better than the 3.943522 I'd got on before I went down my rabbit hole of optimising hyperparameters . So, that was an interesting test -- I was talking to ChatGPT about it at the time and it called it "maybe an art project", which I thought was amusing if a bit arch. Time to do something a bit more useful. Finding an RTX 3090 for a decent price from a trustworthy-seeming vendor is kind of hard right now. But it's still the sweet spot for price-performance if you're looking to train models locally, so I set up an alert on eBay, and eventually one popped up in Bulgaria. I bought it, and a few days later, this turned up: It's actually not as ugly as it looks in that photo -- it's considerably uglier. The stuff that looks a bit like crinkled aluminium foil is really white plastic with a kind of crystalline texture. Made me glad that I'd gone for the mesh-sided case rather than the glass one. Well, I hadn't bought it for the looks. I removed the old GTX 1660, and put in the new card, switched it on, and: Wow, a disco in my PC. Lovely. It was time to kick off another training run to see if it worked. This time, I did my normal GPT-2 small sized train with optimised hyperparameters. It ran for about ten minutes, and then switched herself off. That didn't look good. I spent some time digging around trying to work out why my new graphics card was broken, and then happened to be sending the video above to a friend, and spotted something. Check out the Noctua fan -- the beige and brown one you can see behind the cooler mount, above the graphics card. It wasn't spinning. That's the CPU cooler fan and should always be spinning, even if slowly, when the machine is on. I log basic metrics for all of my PCs to a central InfluxDB instance, so I checked that out and: A CPU temperature spike up to about 115°C! Not good. Clearly an emergency thermal shutdown from the CPU. I initially thought that I must have knocked the fan cable loose while plugging in the new GPU -- plausible, though they were quite far apart -- but unplugging then reseating it, then powering up the machine still didn't start the fan spinning. And it was not visible in the BIOS. I then zoomed out a bit in Grafana; I only keep 30 days' worth of metrics, and it had been more than a month since I did my original burn-in test, so I didn't have anything for that. But I did have this: had been idle for all of that time, and was averaging CPU temps of over 70°C. The dropoff prior to running the test was because she'd had a chance to cool down while I installed the GPU. Having spent ages setting up my InfluxDB monitoring stuff so that I have metrics for everything, I should probably actually look at them every now and then, because the fan had obviously not been doing anything for a month or so. Well, thank goodness for Amazon next-day delivery. I bought a new Noctua NF-A9x14 PWM (praying that the problem was the fan and not the header on the motherboard), and when it arrived, I put it in. This time, when I powered her on, the fan was spinning. Phew. I left her running for an hour, and the CPU temperature stabilised at 35.5°C. Next, I kicked off a version of my standard LLM training run with the number of tokens reduced so that it would run for an hour. During that, the CPU temperature went up to a moderately-toasty 76°C -- not ideal, but remember that with the broken fan, she was running that hot at idle. It seemed a bit odd that it was that hot at 10% CPU usage, but given that one core was running at 100%, it didn't seem totally off. The heatsink and fan are designed for SFF PCs anyway, and those tend to run somewhat hot. The GPU temperature also went up to 70°C and stabilised there, while power draw was stably about 368W out of 370W, and GPU utilisation at 100%. That was particularly pleasing because Nvidia cards throttle at 83°C or so by default, so if I was getting a lower temperature at full power, the fans clearly had some headroom for cooling. Once that was completed, it was time for another full training run for a burn-in. I kicked off my normal run. CPU and GPU temperatures stabilised at the same level as they had with the one-hour test, which was promising, so it was just a question of waiting... ...until I got this: About 40 hours, which is pretty much standard -- certainly the same as I'd expect from . The smoke test: Don't you just love it when your LLM tries to sell you something? 1 But anyway, loss on the test set was 3.548880, which is essentially the same as the same training run on too. So, now is a properly-configured training machine -- one RTX 3090, a CPU that runs a bit hot but at least doesn't do emergency shutdowns, and a case and a PSU with enough space for more GPUs. I think that the next step will be to move on to water cooling. In order to support more than one GPU, I'll need a new motherboard and probably a new CPU, so I don't think there's any point in watercooling the latter, despite its toastiness -- I'd just be buying a waterblock for it that I'd throw away in the not-too-distant future. Instead, I'll get the block for the GPU, and set up a loop to cool just that. Who knows -- maybe I can get rid of that horrendous RGB stuff at the same time! We live in hope. Also, that "expertise and expertise" tiny model smell. ↩ is my daily driver. If he's doing a training run, then everything is just a little bit sluggish as CPU and GPU alike are busy. Although I don't play games often, it's annoying to have the option ruled out for days at a time. While the GPU is busy with a training run, I can't do other experiments in parallel -- for example, to scope out what the next step might be. Small enough to fit in a carry-on bag. I was building the machine in London, and wanted to be able to bring her to Portugal easily, and to be able to bring her back if I wanted to. Portable enough to quickly move around the flat. In the holiday home, the dining room was my study, so I wanted to be able to keep there normally, but move her when we had guests for dinner. Powerful enough to be able to run the games I was playing -- at the time I was a big fan of Assassin's Creed Odyssey , which didn't need a flagship card, but wasn't lightweight either. An AMD Ryzen 5 3600 3.6GHz 6-Core CPU A Noctua NH-L9a-AM4 CPU cooler A Gigabyte X570 I AORUS PRO WIFI Mini ITX Motherboard 32 GiB Corsair Vengeance DDR4 RAM 2x Samsung 970 Evo 500 GB NVMe SSDs A Zotac GTX 1660 Super 6 GiB GPU A Lian Li PC-TU100 Mini ITX case A Corsair SF450 450W SFF PSU An ASRock Phantom Gaming PG-1600G 1600W , which would have power in spades -- an RTX 3090 goes up to about 370W at full draw, so that should hopefully handle three of them plus a CPU without problems even if one or two of the GPUs had power spikes. A Fractal Design North XL . was already in a North (not the XL variant) and I love the case; the XL one looked like a good option if I was going to be cramming more GPUs in there, and had plenty of space for water-cooling. Vocab size: 50257 -- this was fixed because I was using the GPT-2 tokeniser. Context length: down from 1024 to 512 Embedding dimensions: down from 768 to 512 Number of heads: down from 12 to 8 Number of layers: down from 12 to 8 QKV bias: no (different to GPT-2, but the same as my own best local model). Also, that "expertise and expertise" tiny model smell. ↩
Previously I opined that Valve was about to win the console generation . I couldn't have possibly predicted that both Microsoft and Sony would just self-sabotage so hard that they're both going to lose. Between Microsoft's decimation of the Xbox division , slaughtering off the IdTech team , and continued increases of Xbox hardware prices ; there's nothing to really be excited about with the Xbox. Sure their most recent presentation showed off a bunch of exclusives, but none of them really made me think "wow, I should go get an Xbox to play that". Hell, few of them made me think "wow I should go play that" beyond the Halo remake coming out next month (and really I just want to see how much of a trainwreck that is going to be). Microsoft is also starting to double-down on their in-house games being Xbox exclusives, which really doesn't give me much reason to want to play them because I simply can't buy them without buying an Xbox. Sony also has discontinued porting their games to PC because they're not hitting the (probably impossible) revenue targets that they need to make up for big-ticket failures like Concord . I do have a PS5 that has mostly been relegated to gathering dust when it's not playing YouTube and Twitch duty in the living room, it's likely going to be replaced in favour of my Steam Machine whenever that comes in next year. However nothing that's come out in terms of Playstation exclusives is really compelling, and what is compelling enough just isn't that compelling to want to buy it on Playstation as opposed to just getting it on Steam where I can run it on my tower or on the home theatre PC. Sony also has been raising prices and recently announced that they're killing physical media next generation . It's starting to make me wonder if I should even bother getting the next generation of Playstation. If I can't give people physical games as gifts anymore, why should I bother buying the new console? My husband and I both can't remember why we even got a PS5 in the first place, maybe it so that we could do couch gaming without hearing the fan noise or so that the video streaming experience from the NAS could support HDR. We have a Switch 2 at home, it's mostly there to play Nintendo exclusives like Mario Kart World and the Xenoblade series. If those exclusives were available on Steam, we wouldn't buy them on the Switch 2. Otherwise, everything is via Steam or other PC storefronts anyways. Man, Valve really does win by doing absolutely nothing while the rest of the industry shoots itself in the head. I fear for what happens when Gabe Newell retires and the MBA cancer fully infects Valve.
Read on the website: So I was considering starting a wiki. But it seems there are some unsurmountable value mismatches with me and wikis. So here’s a small reflection piece engaging Wittgenstein, literary theory, and a straw man idea of wikis.
“We asked 100 people: What are the top three companies on earth best positioned to make a world-class Mac-assed Mac app ?” Survey says: Yes! Apple at the number one spot. Makes sense. Who better to make the very definition of a great Mac app than the people who make the Mac? No brainer, I suppose. Granted, they’ve had some misses , but nobody bats 1000. Ok, let’s keep going. “We asked 100 people: What are the top three companies on earth best positioned to make a world-class Mac-assed Mac app?” “Anthropic!” Survey says: Wow, that’s odd huh? You’d think Anthropic would be right there at number two. Not only do they have billions of dollars, but they also develop, maintain, and control the super intelligence we’ll all soon be subservient too, right? Surely if anyone (besides Apple) is well positioned to make a world-class Mac app, it would have to be Anthropic — right? And yet, here we are with Claude Desktop as an Electron app . Ok, let’s keep going. “We asked 100 people: What are the top three companies on earth best positioned to make a world-class Mac-assed Mac app?” Maybe not . Not so much . I’m sorry, but that’s three strikes. Apparently it’s a mistake to assume that a big company with piles of cash is well poised to make a great Mac app — even if they are enabled by hyper-super-intelligence. “Well who cares? It just goes to show you don’t have to make a good Mac app to be obscenely successful in terms of revenue!” Well, maybe that’s true. Actually, come to think of it, it kinda does seem like the bigger you get and the more money you make, the more likely it is you’re making an Electron app. There seems to be a correlation between “Mac-assed Mac app-edness” and “Company size/revenue”. Why is that? I’ll leave that as an exercise for the reader (though my mind is leaning towards something to do with care ). Thank you for playing reading this game of family feud. Reply via: Email · Mastodon · Bluesky
This post is the capstone of the most long-running series on my blog . In December 2024 (!), I started reading Sebastian Raschka 's book " Build a Large Language Model (from Scratch) ", and worked through it carefully. Being who I am, despite trying to apply a strict "no side quests" policy, I found myself zooming off and digging into all kinds of things. It's time to wrap it up. I had decided that the endpoint would be to build and train an LLM from scratch just using my notes -- no reference to the book, no reference to the model code I'd written when following the book. After an X/Twitter poll, I decided to use JAX for that, just to make sure that I really was building it from scratch and not regurgitating bits of PyTorch code like a bad coding LLM spitting out half-digested lumps of Stack Overflow. In my last post , I showed how I built a JAX training script that mirrored what I had built for the original PyTorch version of the model. To test it as I went along, I used it to train a really dumb "LLM", which instead of trying to predict the next token for every token in an input sequence, instead predicted the input -- that is, if you fed it It would return the same thing. I called that an A-to-A model. In this post, I'll show you how I turned it into a GPT-2 model, and then trained it from scratch on my RTX 3090 (using the parameter counts for the original paper's "small" size). What turned out really well with this is that I found a route that meant that almost every component I added made the model better! That's not guaranteed -- sometimes different aspects of an AI model depend on each other, so adding A without also adding B makes things worse. But (admittedly with a bit of backtracking in places) I was able to find a route that shows a nice clear progression. The final training run took 37 hours 15 minutes -- compared to 40 hours, 38 minutes for an equivalent PyTorch model . That is despite it being full-fat 32-bit -- the PyTorch one was using Automatic Mixed Precision (AMP), which allowed it to use 16-bit calculations in places where it would be relatively harmless in terms of loss. When asked to continue "Every effort moves you", it came back with a decent response: The model got 3.418784 loss on my held-back test dataset, as compared to my PyTorch model's 3.538161, and even more impressively, it was better than the original GPT-2 small's result of 3.499677 on the same dataset! However, just as I found previously , the OpenAI weights still beat mine consistently in instruction fine-tuning challenges. Let's get started. At the end of the last post, we had a solid training loop, using all of the tricks I'd picked up with my PyTorch code. The A-to-A model we were training with it looked like this: That was based on my preferred model of how LLMs work , where at the top level for a model, we feed in a sequence of token IDs, then: The A-to-A model basically skipped the second step completely: it would project to embedding space, then immediately project back to vocab space -- and after training, it was pretty good at mapping a sequence to itself. One interesting question is, if we train the same code, but this time try to get it to make next-token predictions, how good will it be at that? Obviously it can't be as good as a full LLM. But there are correlations between tokens; full stops will generally be followed by spaces, adjectives will normally be followed by other adjectives or nouns (at least in English), and so on. It would be kind of like the predictive text systems on a phone, where (at least until recently) it would just use the last word you entered to generate a list of possible next words to select from. Old-school natural language processing has a name for this: bigrams. The idea is that you can work out statistically what the most common two-word pairs are, which allows you to make a guess at a next word from a single one. (There are also trigrams, where you look at the last two words when predicting the next, then 4-grams, 5-grams, and so on.) You'd build up a full probability table -- for every word in your vocab, you'd have the probability of every word coming next. So maybe even with that minimal model, we could get it to learn something similar to a set of token-level (rather than word-level) bigrams, which would then get the loss down. Obviously it wouldn't be as good as a full bigram table -- for our GPT-2 vocab size of 50,257, that would need 50 , 257 2 = 2 , 525 , 766 , 049 parameters -- but perhaps it could approximate one. (For comparison, the model we're using has just an embedding table and an output head, each mapping between 50,257 dimensions and 768, so that's 2 × 50 , 257 × 768 ≈ 77 million parameters -- about 3% of the full table.) An uninitialised model would (hopefully) have a loss of about 10.82, implying a perplexity equal to the vocab size. If we can train our dumb model to get better loss than that, then we'd have the beginnings of an LLM. That was a simple test to run. In my training code, I had a dataset class that looked like this: That is, the inputs, the , were the same as the targets, the . If we fed it ...then we'd be training it to output exactly the same thing. The modified version for a real LLM would involve feeding it something like this: ...and targeting this: That's a simple change -- that method became this: I did that, and kicked it off to train on the 92,209,152 tokens that I was (somewhat arbitrarily) using in the last post to test my training loop. The loss chart looked like this: That was pretty promising! Loss came down from roughly 10.82 down to a fairly stable 6 or so by global step 768, and seemed to flatten out there. It's possible that further training could have got it down a bit more, but I decided (again, somewhat arbitrarily) to use the average train loss in the checkpoint period ending at step 937 as my starting point. If we could make changes that reduced that, then we'd be moving forward. For this model, that value was 5.909. So, what were the changes we needed to make to change our bigram-style model to a real, if small, LLM? Adapting from my how LLMs work post, a GPT-2-style LLM looks like this. We receive our sequence of token IDs, and then: Inside the Transformers blocks, we: So that gave me the checklist; looking at it, the most tempting next step was layer normalisation (henceforth LayerNorm). It's used at the end of the core loop, and then twice in the Transformers blocks. What would happen if we coded it up, and then added it to the core only? The purpose of LayerNorm is to stabilise training. We constrain the values flowing through our model so that they have certain statistical properties that tend to make the whole thing more trainable. That would mean that if it did help with this model -- placed in between the embedding layer at the start, and the output head at the end -- then we'd hope for loss to go down faster, and ideally finish at a lower level. Time to code it up! NNX has its own LayerNorm implementation , of course (as does PyTorch ), but in the book, we implement it ourselves, and that felt like the correct path to take. Firstly, I implemented a dummy version: ...and updated the core to create and call one: And kicked off a training run for a few seconds just to make sure that it hadn't broken anything and that loss dropped -- being my first NNX module-inside-a-module, I worried that there might have been something non-intuitive that I had to do to get it to work. But everything seemed good -- loss was dropping, no errors. So, following the notes I made when I first learned about LayerNorm , I needed to make the values flowing through centred around zero by subtracting their mean, and then scale them to have a variance of one by dividing by the standard deviation (details in those notes). The shape of the I had coming into my class's was this: That was . So we needed to do those operations strictly on the last axis, manipulating each embedding independently. JAX has a function and a one , both of which take an parameter. The object repackaged those as methods, which was convenient, so I did a first cut test like this: That printed out these results: ...which looked plausible; one number for each embedding vector. Could we broadcast them across the array? This blew up: Fair enough. But and have a kwarg that looked like it would help: ...and it did! Excellent. So the next step was to see if that would work even slightly. Interestingly loss started off a bit higher at 11.29 after the first global step -- so adding in the LayerNorm had actually made the model worse than it was -- but it seemed to be falling rapidly. Things weren't totally broken, at least. But there was more to LayerNorm than just zeroing the mean and scaling to the variance; we also needed to scale them up by a learnable amount, and then shift/bias them by adding on a different trainable amount. More precisely, both of those trainable amounts were different for each of the (in this case) 768 embedding dimensions. We needed two learnable vectors of length . I hadn't noted it down at the time but I figured (as it turned out, correctly) that a sensible starting point for those values would be all-zero for the bias, and all-one for the scale. From this help page , the way you create a trainable array associated with an NNX module is this: That code created a random vector, rather than the zeros/ones we needed, and we'd need to get the dimensions right. Because of the "Incompatible shapes for broadcasting" error I'd just had, I was feeling a bit paranoid about the latter, so I chose a shape of , and wrote this: That looked pretty plausible, though in retrospect I think I was being overly cautious and didn't need the leading two axes for the scale and bias. The only thing I was unsure about was whether the wrappers I had put in were really making those arrays trainable. I put some code in to print them out and kicked off a run for a few minutes, and confirmed that they were changing in ways that seem plausible -- small non-zero bias, scale close to but not equal to one. That was all good! Next, I spotted one issue. What if one of the standard deviations was zero? That would lead to a divide-by-zero error here: Now, the standard deviation, if it's not zero, has to be positive -- so adding on a small value would fix that 1 : With that in place, I felt that it was ready to go. Time to do a full training run! I kicked that off, and it completed with this output: Loss looked like this: Let's look at the results for the previous run without LayerNorm for comparison: You can see that the new run, the first one, drops faster. It's harder to see from the chart, but it also finished up with a lower training loss at 937 (my relatively arbitrary metric): 5.734 rather than 5.909. That was interesting! The new model was basically doing the same thing -- predicting the next token based only on the "current" token, but loss was lower. My take is that if we had trained the non-LayerNorm model for longer, it might have managed to eventually grind out a better loss. But LayerNorm was doing its job -- it was stabilising training, and as a result we converged faster. That was a win! I decided to run it through my old smoke test from the PyTorch training runs, and see how it completed "Every effort moves you": It was kind of impressive that it managed to finish the first line before it got stuck in a loop -- but it was understandable that we couldn't expect anything good yet. Each predicted token was based entirely on the token before it. What next? Back to our checklist: Inside the Transformers blocks, we: So, at this stage, for each input token we were predicting the next one based on the input token only -- like I said earlier, we were doing a somewhat roundabout way of building an approximation of a table of bigram probabilities. What would happen if we started paying attention to the tokens to the left? And what would be the simplest, dumbest way to do that? The real LLM has multiple layers of multi-head attention, each one also having a feed-forward network, some LayerNorms, and some shortcut connections. Single-head attention is easier to code, but even on its own, you'd expect it to be able to add some value. Each token would get at least some information from the ones to the left. And one layer, likewise, you'd expect might help a bit. I suspected that it wouldn't work on its own -- I expected I'd need shortcut connections too -- but decided to start with attention on its own. I modified the main class to have a single "Transformers" layer: ...where that layer was actually just single-head attention: Next, it was time for the class. I'm not going to write yet another attention explainer -- I think my "How do LLMs work?" one does a decent job of that, and "The 'why' of attention, or: attention heads are dumb" works well too. So in the next bit I'll assume that you understand the basics. My first cut was basically just the maths (up to the causal mask) to get the attention scores: It did the projections into query, key and value space, worked out the attention scores with the array multiplication, normalised it by dividing by the square root of the number of dimensions in the Q-K embedding space, and then zeroed out the scores where a token was attending to tokens in its "future". There were a couple of problems, though. Firstly, that wouldn't work if we were working with batches, and secondly, zeroing out the non-causal scores wasn't quite correct. The batches first. Our incoming here would have the shape . After the projections to the Q-K embedding space, both and would also be shaped . Now, the property on the JAX array class just reverses the axes, so the code above would give us with the shape . That would break! Matrix multiplication in JAX expects all but the last two axes to represent batches, so we actually wanted to have the shape ` . That meant that what we actually wanted was to just transpose the last two axes. The JAX function takes an parameter that allows you to specify the specific re-ordering of the input axes that you want. So I could rewrite the code like this: As would have the shape , and the transposed version of would be , they'd be compatible for matrix multiplication and give us a result that was -- just what we wanted for attention scores. The next step was to fix the causal mask. The next step in this attention mechanism was going to be running the causal attention scores in through softmax over the last dimension, to convert them into attention weights. Now, our current code was zeroing out unwanted acausal scores, but a zero still contributes to softmax. If you want a particular value to come out of softmax guaranteed to be zero, you need to set it to minus infinity. I decided that the easiest way to do this was to create a causal mask -- a boolean array that matched the size of , but was full of s: Then I could zero out (well, "false out") the cells in the mask related to unwanted future-facing scores, just like I was previously doing on the scores: ...and then I could apply that mask to omega with , telling it to create a new array, taking the value from where the mask had , and in places where it had . That seemed solid, so I just needed to run the result through , specifying that the last dimension was the one where it should apply the function, and that would give me the attention weights: Finally, I just needed to use those attention weights to get the attention output by mixing in appropriate portions of the projection of the inputs into value space, : As was shaped , and (like and ) was shaped , the batch axes were at the start where they belonged, and the matrix multiplication would work and return something shaped . With that, we were done! The final single-head attention class looked like this: I kicked off a training run with that, and it did work, in that loss went down over the course of the run -- but at the end of the run, the loss at step 937 was 5.934 -- significantly above the 5.734 I got on the previous run, with no attention. But that made sense! As I'd said earlier, I suspected that this wouldn't help if we had no shortcut connection. Intuitively, if you want to work out what token should be at position n + 1 , on average the most important other token you need to know about is probably whichever one is at position n . Knowing about the tokens at n − 1 , n − 2 , and so on, could well be helpful -- maybe very helpful -- but not at the cost of not knowing about the one at n . Now, single attention heads are just simple pattern-matchers. They can't learn complex rules, it's only by working together -- "horizontally", in multi-head attention or "vertically" across multiple layers -- that they can do complex things. What we were asking this head to do was to learn some way of gathering information about previous tokens, and also to keep the knowledge about the "current" one. That's a tall order for a dumb attention head! In my mind, this is a large part of the benefit of shortcut connections. They are often presented as a way to make sure that during training, gradients flow smoothly from the output end of the model to the earlier layers. But I prefer to think of them as preserving the original embeddings, so that each layer doesn't completely replace what came into it, but instead does something closer to adding on its own notes -- like scholars adding commentary to a core text in the Talmud . In the training run above, the attention head was trying to learn how to preserve the meaning of the embedding it was working on, while also merging in information from earlier ones. If we added a shortcut connection, then it would only have to do the second of those two jobs. The code was simple: I updated the module to do a shortcut connection: I kicked off a training run, and at the end it printed this: The loss chart looked like this: And, importantly, that training loss at step 937 which I was using as a metric was 5.553 -- a decent improvement over the previous best of 5.734. Even a dumb single attention head was able to do something useful, if it had a shortcut connection. I decided to run another qualitative smoke test: I mean, it was repetitive, but it was actually getting noticeably closer to making sense! So that was excellent news. What next? Our checklist looked like this: Inside the Transformers blocks, we: Now, our single attention layer was lacking something. Without position embeddings, that layer has no idea what order the tokens before the one it's looking at come in. If it's considering the " cat" in ...it doesn't know if it's looking at "The fat cat" or "fat The cat". Position embeddings are simple, and might help, so that was the next step. These were trivial to add. We had this core code: So I just added a position encoding module in : ...and mixed it in with the token embeddings to create new, improved to be used in our "Transformers" layer: I kicked off a training run with that: Pretty hard to distinguish from the previous one, but the metric I was tracking, that loss at step 937, had improved again! We were down to 5.354 from 5.553 :-) A quick qualitative smoke test didn't show that improvement, though: Pretty much indistinguishable to the previous one. But still, Loss Number Went Down, and that's what was important at this stage. It was time to try the next step. From the checklist: Inside the Transformers blocks, we: We had only one attention head right now. Individually, attention heads are dumb , so switching to multi-head attention seemed like a good thread to pull. At this point, my single-head attention code looked like this: I decided to re-implement multi-head attention (which I'll call MHA from here onwards) from first principles rather than working strictly from my notes, and then to come back and check it. If you're looking at your browser's scrollbar with horror (" still only 50%?!") and really don't want to read a full derivation of MHA, you can skip straight to the first complete version of the code . The point of MHA is that we're running multiple copies of the calculation above in parallel -- let's pin down the name of the number of copies as . Now, we could naively implement it just by spinning off threads and running the existing code in each, but that wouldn't really take advantage of the GPU's inherent parallelism. I felt that we could rely on the fact that JAX's matrix multiplications treat all but the last two dimensions as "batches". For example, if you have two arrays with shapes: ...then you can multiply them. A m × n matrix multiplied by a n × p one will be m × p , so you'll get something that is The other dimensions (so long as they match) will essentially act as an a × b × c × . . . × l batch. Now, right now we were just using a single batch dimension. Let's look at the core multiplication in the attention mechanism, which works out , the attention scores. I had this: Breaking that apart into two steps: We got from this line: Let's look at the shapes here. is our input embeddings for this layer; its shape is . Projecting it through , which is shaped gives us a shape for of again. , being a projection of through , which is the same shape as , will have the same shape as . Now, that means that is , and the calculation ...is doing a batched matrix multiplication getting us the that we want, shaped . But as I said above, there's no need to stop with just one batch dimension. Let's say that we have heads, and that they each work with embeddings sized . Imagine that we've already somehow done multiple projections into the key and query spaces for each of our heads, and that the results have somehow been put into arrays such that and are shaped -- that is, we've gained an extra axis that keeps the projections for each head into its query-key space separate. We could use the fact that both of those two leading axes are basically just batch dimensions, and the existing single matrix multiplication will still work, with one tiny tweak: the current transpose is this: ...to swap around the last two axes of a three-axis array. With one extra batch dimension, we'll need to take account of that and do this instead: That will be a multiplication of , shaped , with , shaped , which gives us an of the right shape, . So, if we can start treating the heads as just another batch dimension, things seem simpler, at least for the attention score calculation. Let's continue down through the single-head code, and then come back later to how we might get the inputs into that double-batched shape. The next line after the calculation just scales the attention scores by a scalar: That looked fine, just a broadcast division-by-float. We'd need to change that to be in some manner, but that's all. The will give us an array that's full of s. That seems reasonable. The next step: What will that do? Well, per the documentation : When , operates batch-wise on the trailing axes. ...which sounded good. and would be treated as batch axes, which meant that the next line: ...would work. Likewise, with the next line: ...the axis to apply to is explicitly stated as the last one, which is what we wanted. So at the end of all of those steps, we'd have shaped , where the last axis had been softmaxed (softmaxxed?). The next line looked a little trickier: In the single-head version we had of shape , and V of shape , so multiplying them gives us In the new MHA code so far, we had our shaped . So in order for the matrix multiplication to work, we'd need to be shaped . That would give us a result shaped as . And conveniently, we'd already decided that the correct shape for and for was . If we could use the same "magic" to do the projection into value space -- that is, to get such that the heads formed a new batch-like axis like we had for and -- then we'd be all set. So, at that point, I'd worked out the core of MHA. If we could get all of the inputs into the shape , and somehow handle an output of the shape , then we could use MHA code something like this: The next question was, how do we get our inputs into that shape? We could run them all through separate per-head weights -- that is, have an array with one per head, like , and for the first one. But that, again, felt like it would be failing to take advantage of the GPU properly. The solution was to think of how matrix multiplications work. If you multiply two matrices, X · Y , the value in the result, in row r , and column c , is the dot product of row r in X and column c in Y . So, imagine if you wanted to multiply X by n different versions of Y , let's call them Y 0 , Y 1 , and so on up to Y n . If you imagine a new matrix, Y all , which is basically all the Y x s stacked side-by-side, then the dot-product understanding of multiplication makes it pretty clear that if you did X · Y all , you would get the results of all of those separate multiplications, also stacked side-by-side. I'll call that kind of matrix a "striped" one, for want of a better word. Now, when we project our inputs into the embedding spaces used for attention, we have code like this: We've initialised the weights, in this case, as an , so what is happening under the hood here is basically: That is, it is just a matrix multiplication. 2 So if we imagine that is one of those "striped" matrices, holding all of the separate matrices to do the projections for all of the heads in a single one shaped , then we could stick with the current code -- the Our input would be shaped , so the result would be , and would have the projections for each head in the same vertical stripes as the separate heads' projection weights. Now, like PyTorch, JAX allows you to reshape arrays. You can take one axis of length (say) m × n , and split it into two of lengths m and n respectively -- or, conversely, you can combine two axes of length m and n to one of m × n . If our data had the shape , we could reshape it like this: ...and that would split things up. So we'd have Q shaped as . That's almost what we wanted! We needed , and a simple transpose could sort that out: Likewise for and , and that was our inputs sorted. Moving on to the output; it came from this: ...and as we worked out above, it was shaped . I remembered that we wanted to run that through a single linear layer to combine all of the different heads' outputs into one. It felt like the best way to do that would be to get it back into a "striped" layout: . This would be something like the inverse of the input-wrangling. That would need a reshape, but before I could do that, I'd need to get the axes that needed to be merged next to each other. If the input to the linear layer was going to be , we'd need to convert it from to first: ... and then we could just reshape it to : Finally, we could run it through a linear layer, with set to , and set to . I put that all together, and decided to throw something extra into the mix. I remembered that Raschka's code had various checks to make sure that , which seemed a little artificial -- I'd read that this was true of GPT-2, but wasn't a necessary restriction for GPT-style models, which makes sense. There's no obvious reason per se why the heads' embedding dimensions should sum up to the higher-level embedding dimensions. So I decided initially to just pass in and to the constructor. In my training script I could force them to match the GPT-2 model, but if I wanted to use the code later for something different, I could vary them. Then I remembered that although the dimensionality of the embedding spaces for the query and the key vectors have to match (because otherwise you can't multiply them to work out attention scores with Ω = Q K T ), the value vector's dimensionality can in theory be different. So I decided to break into two separate and parameters. The result was this: Unusually for a case where I went off the reservation like this, the whole thing with the embedding space dimensionality didn't cause any problems at all! But there was one small bug in this code, which I didn't discover until later -- we'll come to it by the end of the post. At this point, I did another of my short training runs, and: ...with a loss chart that looked like this: The training loss at the 937th global step was 5.336, only a tiny bit better than the 5.354 with single-head attention. That was quite possibly within the noise. Even though (due to the restriction I was enforcing in my training script) the , , and arrays were the same size, I was creating that , which would consume randomness and make things vary. If I were doing a proper scientific experiment to see if a single layer of MHA beat a single layer of single-head attention, I think I would have run both for more steps to see if the difference became more pronounced later. But for the purposes of this post, I decided to move on. My checklist now looked like this: Inside the Transformers blocks, we: Adding that simple neural network -- the FFN -- seemed like a good next step. The feed forward network is simple; you take the output of the MHA block, run it through a biased linear layer to expand it from to , then run it through the GELU activation function, then shrink it back down to with another linear layer. I didn't really see any value in writing my own implementation of GELU, given that even in the book we were just given code for an approximation to type in. So, using , I wrote this: Note that I added in a shortcut connection around the FFN as well, so that it didn't overwrite what was there, but only "added on its notes". I kicked that off, and it ran for ten minutes or so, but then OOMed: Adding didn't help. I spent some time trying to dig into what might be causing it, but eventually noticed something interesting: in , the VRAM usage was consistently 75% throughout. Now I knew that JAX pre-allocates 75% of VRAM when it starts up, but I'd been assuming that it would try to grab more if it needed it. It turned out I was wrong with that assumption -- it grabs 75%, but that's all you ever get! The solution turned out to be the environment variable. If you set that to, say, , then JAX will pre-allocate 90% of the VRAM, and you can use all of that. (You can also make it allocate as-needed with , and there are various other settings you can control with other environment variables on that linked page). Anyway, setting it to to grab 90% of VRAM worked, and I was able to get a successful run: The loss chart was this: ...and the training loss at global step 937 was 5.295, compared to the 5.336 from MHA alone. Another tiny improvement, another one that could have been in the noise. Again, if I were doing a proper experiment, I'd do a longer run, but for now, I decided to move on. The checklist looked like this: Inside the Transformers blocks, we: Now, my gut instinct was that the layer normalisation inside the Transformers blocks was of most value as a way of stabilising training over deep networks. And with one layer, it didn't seem like the right time to add it. Instead, I decided to add on multiple layers. For GPT-2 small, you have 12 layers. That was already being passed in to my 's method as , so I just replaced this: ...with this: ...and then just renamed it where it was called; this: ...became this: I kicked it off, and it completed! However, the loss chart was telling: Ouch. Loss started dropping quite nicely, but then things got out of control and it settled down at a loss that was essentially that of a random model. At step 937, we were at 10.75, so just a hair less than the 10.82 that randomly guessing next tokens would give. Well, LayerNorm is specifically meant to stabilise training, and the checklist looked like this: Inside the Transformers blocks, we: ...and the only remaining step was that LayerNorm in the Transformers blocks, so it was time to add it in! As per the checklist, we do the LayerNorm after we've taken our copy for the shortcut connection, just before MHA, and then likewise after the second shortcut copy, before the FFN. As I understand it, this was a GPT-2 innovation -- previously, people had done normalisation after those steps, but this pre-norm setup turned out to work better. The code changes were simple. I added two modules to the class, and then called them in the appropriate places (taking the opportunity to tidy up the variable naming in the forward pass while I was there): I kicked it off and ran it, and got these results: That certainly looked much healthier! However, when I looked at the loss at step 937, it was 5.311 -- a tiny bit higher than the single-layer MHA example, which got 5.295. I'd been willing to play a bit fast and loose with this loss number and allow myself to accept a win when the loss went down a tiny bit, even if it was such a small amount that it could have been within the noise. But increasing loss -- even if it could also be within the noise -- was a step too far. I decided that in this specific case, I'd be strict and test the hypothesis that longer training runs would demonstrate an improvement between one single layer without pre-norm, and multiple layers with pre-norm. I had to remember that these training runs would not be comparable with the earlier ones. In the training script, I had a learning rate schedule like this : That straight-line warmup period and the following cosine decay were 5% and 95% of the training run respectively, which meant that (for example) global step 937 of the short runs we had been doing would be at a completely different point in the schedule than the same step would in these longer runs. However, they would be comparable to each other, and that was what mattered. After some humming and hawing, I decided that a full Chinchilla-optimal (for the full model) training run over 3,260,190,720 tokens, rounded up to fit into a round number of global steps, would be a nice experiment. I expected it to run comfortably overnight for the single-layer run, and take a bit less than two days for the multi-layer one. So I kicked off the first. Just over 11 hours later: Here's the loss chart: The last checkpointing period in that run ended at global step 33,164, and the training loss then was 4.165 -- indeed, it had been at around 4.17 for quite some time, though the trend still seemed to be a tiny bit downward. So then I kicked off a run of the full version -- multiple layers, with pre-norm in the Transformers blocks. Just over 37 hours later: The "Final train loss" line at the end said it all, really! But here's the loss chart: ...and the loss at step 33,164 was 3.399. Definitely quite an improvement over the 4.165 that a single layer got. Again, at some point I might do the equivalent tests for the earlier results where improvements appear to be pretty much in the noise. It would be good to be sure that the changes really did have the impact I think they did. But for now: our checklist was looking like this: Inside the Transformers blocks, we: Everything was checked off. So was this journey over? Well, there was one thing that the original PyTorch code had that my new code didn't: dropout. I'd found in my lengthy interventions experiments that dropout seemed to make models worse. It was, I felt, a smart idea back in the days when people had little data and did multiple epochs, each sweeping over everything, but it made less sense nowadays with single-epoch training runs over very large datasets. (Though I do have some intuitive ideas about why it could still help .) Still, it would be good to show that it harmed loss for this model as well. Checking my notes, I found that there were four places where dropout was applied: The changes are tiny and rather dotted around the code, so rather than showing you isolated bits of code, if you'd like to see it you can take a look at the code at this point and search for "dropout". When I started running that, I got an error when saving the first checkpoint: This was happening deep inside the bowels of Safetensors, but it made a lot of sense. The object needs to keep track of the state of the random number generator, and that meant that the function that I was using might return a structure that had something that contained that state, and was not compatible with Safetensors. I decided that I'd cheat a little bit here. If I skipped the dropout layers when I saved my checkpoints, like this: ...then I'd be able to save them. This would have a problem -- if I restarted from a checkpoint, the dropout pattern after the restart would mirror the dropout pattern from the start of the training run, because the random seed it started with would not have come from the checkpoint, but just the initialisation code. I felt that this would not have a serious impact, though, and given that I'd not had to restart from checkpoints so far, I (wrongly, as it turned out) decided it wouldn't matter. I kicked off the run, and... after four hours, it OOMed. I cursed, decided that I'd nurse this run through anyway (despite my dropout checkpointing concerns), and kicked it off again. Three hours later, it OOMed again. I happened to be away from home at the time, logging in to my machine remotely (thanks, Tailscale !), and on looking at , I realised that the X window system on my machine was using a gig or so of VRAM. I was running the training run in a session, which meant that I could kill X and not lose state, so I did that, and adjusted the environment variable I was using -- it had been 0.90, so I bumped it up to 0.95. I kicked it off again, and... Note that the tokens seen only relates to the period since the restart, which is why it was lower. One more loss chart: ...and the training loss at step 33,164 was 3.524, higher enough than the 3.399 I got without dropout that I was comfortable that it wasn't in the noise. That was very reassuring. Once again, if this was a proper scientific experiment I'd fix the issue with saving dropout, and run it completely from scratch -- or, at least, run it all the way through from scratch without restarts, even if I had to try several times to get it done. But I don't think that "replaying" dropout would make the loss any worse. And for this experiment, I felt this was enough. So: checklist complete. GPT-2 model coded up. It was time for some evals! I wanted to evaluate these models against the ones I got using the old PyTorch code: specifically, the last local training run that used exactly the same training hyperparameters, and only differed in that it was trained using AMP -- 32-bit floats in general, but using 16-bit where the framework thought it would not be harmful. In order to do exactly the same evals, I decided it would be easiest to write a conversion script to take the Safetensors files written to my JAX checkpoints, and write out new files that were compatible with the PyTorch model code -- then I'd be able to use the original PyTorch eval code. I put something together , converted my last two models -- the full runs with and without dropout -- and tried to load them up. Unfortunately there was an error: You might remember that back when I went through multi-head attention, I mentioned that I'd made a mistake. Somehow, I'd misremembered, and thought that the output projection -- the one that mixes together all of the different heads' outputs -- was a linear layer without bias, despite my original notes being perfectly clear that it did have bias. The good news was that if I disabled bias in the PyTorch code, I could load the safetensors files that I had. So the two models I'd trained so far were not useless, and could actually work as a kind of natural experiment into the benefits of having that bias there. But anyway, in order to do things properly, I was going to need to fix the bug and train yet another model. The fix was simple, I just replaced this (in ): ...with this: Then it was time to kick off yet another training run. After another 37 hours: ...with this loss chart: ...and the training loss at step 33,164 was 3.398 -- almost exactly the same as the 3.399 that I got in the no-dropout training run without MHA bias above! Well, now it really was time for the evals. I updated my conversion script to handle the bias on the MHA output projections, and used it to convert the three models -- the un-biased ones, with and without dropout, and the biased one, without -- to the PyTorch format, then ran the loss test that I had been using to compare the old models on each. Here are the results, compared to the previous models, and OpenAI's: That was a pretty amazing result -- I'd clearly proven that JAX trains much better models than PyTorch! 3.5% better in the best case. Well, OK, no. My guess is that the difference was probably something like better luck with the initial weights on the JAX side, plus the improvement from not using AMP . Anyway, the important thing was that the JAX models were in the same kind of loss range as the PyTorch ones -- and while a 3.5% improvement in loss was more variation than I'd been expecting, it was definitely the right ballpark. Now, one thing I had found in the past was that the OpenAI weights -- and some of my own models, like the Fineweb-Edu ones -- were consistently better at an instruction fine-tuning test than their test loss scores would indicate. Would that hold here? The IFT eval code fine-tuned each model on the Alpaca dataset until validation loss started rising, then used the model prior to the start of the rise to generate responses for a test set. These were saved, and then run past an OpenAI model so that they could be compared with each other: ...with the model order randomly changed for each query to avoid any position bias. The methodology seemed solid, but I was uncertain about the "train until loss starts rising", as it meant that different models had wildly different amounts of fine-tuning -- between two and seven epochs. On the one hand it felt "unfair" to certain models that they'd get less training than others. On the other hand, if the less-trained models had been trained past the point where their validation loss started rising, then assuming that loss would continue to rise, further training would actually be a disadvantage rather than an advantage. I decided to stick with the original plan, and train until validation loss started rising. I did, however, switch the judge model from the GPT 5.4 that I used in my last IFT test to GPT 5.5. Here are the results: More interesting datapoints! As before, you can see that low loss is not particularly well-correlated with a high score on this instruction fine-tuning test. The OpenAI weights continue to lead the pack, and while one of our new JAX models did quite well, it's still beaten by the Cloud FineWeb, 8x A100 40 GiB model. But what was important here, just as with the loss, was that the new JAX models landed in the same ballpark as the PyTorch ones. They did, and so I could be confident that they were doing essentially the same thing. And that meant that, after 18 months, I had reached the end of my LLM from scratch journey. It's been a long trek . I started reading "Build a Large Language Model (from Scratch)" on 22 December 2024. I was planning to breeze through over the Christmas break, but somehow it morphed into being a curriculum onto which I could hang projects to learn the fundamentals of LLMs, beyond what was in the book. In May 2025, I had my first real conceptual breakthrough when I realised that attention heads are (individually) dumb , and as I continued, the second big one came later on in the same month, when the concept of embeddings as being projections between vocab space and embedding space (and the converse projection in the other direction that happens in the LLM's output head) became clear. In August I had the first moment where I felt that the standard teaching approach to LLMs might not be the full story; shortcut connections are normally explained as a way to fix vanishing gradients, while I felt that a better way to see them was a way to allow attention and the FFN to "annotate" the existing information, similarly to how Jewish scholars have annotated the original text of the Talmud . (The results in this post seem to point in that direction, given how even a single layer of attention was massively helped by adding them.) By early December, I had essentially finished the book, and felt I wanted to try to train my first base model from scratch on my RTX 3090 . It worked, and wasn't far off the quality of the original GPT-2 small. I was really surprised that I could do that with consumer hardware, and became interested (perhaps obsessively so) with whether I could match OpenAI's weights. In January 2026, I trained a model using DDP on Lambda Labs , and then spent the following months training model after model, trying to work out which interventions -- learning rate scheduling, gradient clipping, etc -- would improve the loss. I wrapped that up in late April , with the interesting finding that although I'd been able to get the test loss pretty low, that didn't seem to map cleanly to performance in my instruction fine-tuning tests. In other words, Loss Number Goes Down is an interesting technical game to play, but doesn't cleanly map to real-world performance. The final step was this post, and the previous one -- could I, using my notes, implement GPT-2 completely from scratch in JAX without referencing the book? And as you've read, the answer was a definite yes! Of course, as with any long-running project, there are some loose ends -- from this post alone, there's the interesting fact that JAX trained faster than PyTorch (perhaps could close the gap?) and had a larger possible batch size for full-fat 32-bit. And the fact that fixing the multi-head attention bias bug didn't seem to help with the loss much was interesting too. But those are really details, and there's so much beyond them to learn. Longer-context LLMs: position embedding improvements like RoPE, efficiency tricks like flash attention and attention variants like DSA. Mixture of experts models. How do optimisers really work? ( Do they work? ) And plenty more. So it's time to draw a line under this series, and start thinking about what comes next. It's been a blast; if you've been reading along, I hope it's been as useful (and fun) to read as it was to write. And as always, comments, questions and corrections very welcome below. On looking back at Raschka's code, after having worked through all of this, there's a slight difference. I do this: ...whereas he does this: Now, the standard deviation is the square root of the variance, so if you ignore the small numbers -- in my case, and in his -- the calculations are the same. But there is a difference once those are taken account of. I don't think it's large enough to have any serious effect in these runs, though. ↩ In PyTorch, linear layers are stored as the transpose of the matrix that would allow you to do that, so it would be: Also, note that for simplicity (heh) I'm disregarding bias in this discussion. ↩ Firstly, we convert them into embeddings, so we get a sequence of vectors, one for each token. We do this by a lookup into a table, but we can see it conceptually as a projection via a matrix, from vocab space (where a particular token ID is a one-hot vector) to embedding space. Next, we do the magic with our Transformers layers, getting embeddings for the next token. After these layers, the embedding at position n in the output sequence is for the predicted token to come after the token at position n in the input sequence, considering that input token and all other tokens to its left. Finally, we project those back from embedding space to logits, this time actually using a real matrix (in the form of a linear layer), the output head. The logits (after being run through softmax) represent the probabilities for each token of it being the next one. Convert them into embeddings. ✔ ️done Add on position embeddings. Run these embeddings through multiple successive Transformers blocks. Layer normalisation Project them back from embedding space to vocab space. ✔ ️done Take a copy of the input sequence of embeddings Layer normalisation Run multi-head attention Add the copy back in so that the version that came out of MHA is something more like an "annotation" of the original Take a second copy of that one Layer normalisation again Run it through a simple neural network Add the results of that back in. Convert token IDs into embeddings. ✔ ️done Add on position embeddings. Run these embeddings through multiple successive Transformers blocks. Layer normalisation ✔ ️done Project them back from embedding space to vocab space. ✔ ️done Take a copy of the input sequence of embeddings Layer normalisation Run multi-head attention Add the copy back in so that the version that came out of MHA is something more like an "annotation" of the original Take a second copy of that one Layer normalisation again Run it through a simple neural network Add the results of that back in. Convert token IDs into embeddings. ✔ ️done Add on position embeddings. Run these embeddings through multiple successive Transformers blocks. part-done -- one layer only Layer normalisation ✔ ️done Project them back from embedding space to vocab space. ✔ ️done Take a copy of the input sequence of embeddings ✔ ️done Layer normalisation Run multi-head attention part-done -- single-head attention only Add the copy back in so that the version that came out of MHA is something more like an "annotation" of the original ✔ ️done Take a second copy of that one Layer normalisation again Run it through a simple neural network Add the results of that back in. Convert token IDs into embeddings. ✔ ️done Add on position embeddings. ✔ ️done Run these embeddings through multiple successive Transformers blocks. part-done -- one layer only Layer normalisation ✔ ️done Project them back from embedding space to vocab space. ✔ ️done Take a copy of the input sequence of embeddings ✔ ️done Layer normalisation Run multi-head attention part-done -- single-head attention only Add the copy back in so that the version that came out of MHA is something more like an "annotation" of the original ✔ ️done Take a second copy of that one Layer normalisation again Run it through a simple neural network Add the results of that back in. Convert token IDs into embeddings. ✔ ️done Add on position embeddings. ✔ ️done Run these embeddings through multiple successive Transformers blocks. part-done -- one layer only Layer normalisation ✔ ️done Project them back from embedding space to vocab space. ✔ ️done Take a copy of the input sequence of embeddings ✔ ️done Layer normalisation Run multi-head attention ✔ ️done Add the copy back in so that the version that came out of MHA is something more like an "annotation" of the original ✔ ️done Take a second copy of that one Layer normalisation again Run it through a simple neural network Add the results of that back in. Convert token IDs into embeddings. ✔ ️done Add on position embeddings. ✔ ️done Run these embeddings through multiple successive Transformers blocks. part-done -- one layer only Layer normalisation ✔ ️done Project them back from embedding space to vocab space. ✔ ️done Take a copy of the input sequence of embeddings ✔ ️done Layer normalisation Run multi-head attention ✔ ️done Add the copy back in so that the version that came out of MHA is something more like an "annotation" of the original ✔ ️done Take a second copy of that one ✔ ️done Layer normalisation again Run it through a simple neural network ✔ ️done Add the results of that back in. ✔ ️done Convert token IDs into embeddings. ✔ ️done Add on position embeddings. ✔ ️done Run these embeddings through multiple successive Transformers blocks. ✔ ️done Layer normalisation ✔ ️done Project them back from embedding space to vocab space. ✔ ️done Take a copy of the input sequence of embeddings ✔ ️done Layer normalisation Run multi-head attention ✔ ️done Add the copy back in so that the version that came out of MHA is something more like an "annotation" of the original ✔ ️done Take a second copy of that one ✔ ️done Layer normalisation again Run it through a simple neural network ✔ ️done Add the results of that back in. ✔ ️done Convert token IDs into embeddings. ✔ ️done Add on position embeddings. ✔ ️done Run these embeddings through multiple successive Transformers blocks. ✔ ️done Layer normalisation ✔ ️done Project them back from embedding space to vocab space. ✔ ️done Take a copy of the input sequence of embeddings ✔ ️done Layer normalisation ✔ ️done Run multi-head attention ✔ ️done Add the copy back in so that the version that came out of MHA is something more like an "annotation" of the original ✔ ️done Take a second copy of that one ✔ ️done Layer normalisation again ✔ ️done Run it through a simple neural network ✔ ️done Add the results of that back in. ✔ ️done Once in the main body, just after we've worked out the embeddings. Twice in the transformers block: once after attention (but before the shortcut is mixed back in), and once after the FFN (ditto) Inside multi-head attention, on the attention weights ( which surprised me ). On looking back at Raschka's code, after having worked through all of this, there's a slight difference. I do this: ...whereas he does this: Now, the standard deviation is the square root of the variance, so if you ignore the small numbers -- in my case, and in his -- the calculations are the same. But there is a difference once those are taken account of. I don't think it's large enough to have any serious effect in these runs, though. ↩ In PyTorch, linear layers are stored as the transpose of the matrix that would allow you to do that, so it would be: Q = xs × W q T Also, note that for simplicity (heh) I'm disregarding bias in this discussion. ↩
John Gruber at Daring Fireball penned a short eulogy for Maestral: As of today Maestral continues to work just fine. I don’t know when these certificates are expiring. And I don’t know what I’m going to do when they do. Dropbox enshittified its app – my friend joked once that Dropbox is a rare example of a company that pivoted away from a product-market fit – and it seems Apple’s API changes didn’t really help, either. Maestral stepped in to help restore the minimalistic, functional core of Dropbox – I believe Doctorow terms this “disenshittification” – but it was helmed by one person, Sam Schott, who has every right to move on to other things. #enshittification #software eulogies #third party fixes
Listening to modern baseball games through the static of the past, via a Bluetooth speaker in a laser-cut housing modeled from a vintage cathedral radio.
I have never been particularly fond of “shake to undo” on the iPhone. It’s not a pleasant gesture to perform, I feel like typically I don’t have strong enough of a grip on my iPhone to invoke it without fear, and the gesture often undertriggers, requiring an even harder and more cumbersome shake, etc. etc. (One thing I never want to undo is my screen’s pristine surface by having it meet the sidewalk.) I am aware that many years ago, iOS introduced an alternative: a three-finger swipe. But I feel like Apple flubbed that, also – three fingers are hard to plop onto a small screen, and while regular going back navigation means swiping from left to right, undo is inexplicably a three-finger right-to-left swipe. I mean, okay, it’s explicable – it’s the movement of the cursor before and after the typing is undone. But to my brain that feels less strong than the other association, and undo is not always about typing. I also see many people not knowing about this alternative and I must not be the only person struggling, since I see more and more apps throw in the towel and put undo and redo as on-screen actions: = 3x)" srcset="https://unsung.aresluna.org/_media/an-accident-inside-an-accident/1-framed.1600w.avif" type="image/avif"> = 3x)" srcset="https://unsung.aresluna.org/_media/an-accident-inside-an-accident/2-framed.1600w.avif" type="image/avif"> = 3x)" srcset="https://unsung.aresluna.org/_media/an-accident-inside-an-accident/3-framed.1600w.avif" type="image/avif"> Curiously, I even spotted Gmail on desktop doing that recently: = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/an-accident-inside-an-accident/4.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/an-accident-inside-an-accident/4.1600w.avif" type="image/avif"> It’s all a welcome improvement under the circumstances, but those are literally all over the place – imagine if on a laptop, each app had a different key shortcut for undo. (We’ve had that, in the 1980s. The 1980s Nostalgia Industrial Complex doesn’t want you to know about stuff like that.) Anyway, some time ago I promised more onboarding content , and here’s a little thing that happened to me recently. The inciting incident is that I accidentally shook my iPad, and then I saw this: = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/an-accident-inside-an-accident/5.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/an-accident-inside-an-accident/5.1600w.avif" type="image/avif"> Wait, does it mean there is yet another, third undo shortcut? I swiped through the carousel to see these: = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/an-accident-inside-an-accident/6.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/an-accident-inside-an-accident/6.1600w.avif" type="image/avif"> = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/an-accident-inside-an-accident/7.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/an-accident-inside-an-accident/7.1600w.avif" type="image/avif"> None of these feel particularly pleasant to use – although they are nicer on the iPad than on the iPhone – but I started playing with them, and I discovered a fourth entry point. Just a single three-finger tap shows a new-to-me onscreen editing menu, sort of the equivalent of the Edit menu on the desktop: This works on the iPhone and the iPad, and since then that’s the one thing I did remember and I find using. So, to summarize: Yeah, even this still doesn’t feel great. But it’s there in a (no pun intended) pinch. So, is this a success story for onboarding? I think not quite. It all started with an accidental iPad shake, after all, and the gesture I ended up using I also discovered accidentally. But to be fair, I also did learn something, and I think there are some bones of the right solution in here somewhere. Onboarding and in-product education generally feel so bad that even this rickety encounter can be counted as a small victory. #apple #onboarding #touch #undo shake to undo – unpleasant double tap with three fingers to undo – unpleasant a three-finger swipe to undo – unpleasant, confusing direction single tap with three fingers to show a menu, then tap to undo – less unpleasant, but stuck with me
Hello RSS reader! This post contains an interactive feature. Please visit the canonical web page for an optimal viewing experience :) At the risk of pissing on people’s chips I figured it’d be helpful to illustrate the three classes of AI user I’ve identified in the slopageddon. You might be thinking: “Hey, those personas are all negative!” — and you’re absolutely right! Believe me, I’d love nothing more than to shut up about “AI”. The thing is, not a week goes by without one of my peers crying out in abject despair. Until the grifters cease spitting in my face and threatening my career, please allow me to extend a middle finger their way. I’m working on more positive plans that I hope to announce soon(-ish). Makes sense to be more proactive and spend energy where it matters. Not that this post didn’t! I enjoyed a few technical challenges artworking the page. Images used with modifications: Chalk Outline by Simon Child from Noun Project (CC BY 3.0) Hand by Elisa Pintonello from Noun Project (CC BY 3.0) Zombie by Hamstring from Noun Project (CC BY 3.0) Previous Next Thanks for reading! Follow me on Mastodon and Bluesky . Subscribe to my Blog and Notes or Combined feeds. The Grifter Dabbles with free chatbots Chuckles at social media slop Forced to endure a work mandate Never consented to any of this Helpless to the human toll Bends the knee to Big Tech Lives by “AI is inevitable” mantra Anthropomorphises their chat box Ignores self-inflicted deskilling Gambles with house money Flogs AI and AI paraphernalia Will not take “no” for an answer Dehumanises the effect of AI Idolises the techno-fascists Revels in gaslighting
As powerful as Bazel is, sometimes it’s not featureful enough. When using this build system, it’s common practice to wrap it in a launcher script—and in fact, this is natively supported by Bazelisk , Bazel’s native dispatcher that stands for the binary in the user’s . Bazelisk will first download the version of Bazel requested by the project, and then, if exists, invoke it instead of the downloaded binary. is what’s known as a Bazel wrapper and is the point of today’s article. Well, not quite. The actual point of today’s article is to demonstrate a simple trick I learned from the GNU Autoconf and Automake days to implement full-blown conditionals in an ad-hoc template system. But because such trick is trivial once you see it, I have to present it in the context of a modern real-world scenario. So what I’m going to do is guide you through the creation of your very own Bazel wrapper to customize Bazel’s configuration file in ways that the native Bazel tool doesn’t support. Let’s get started. But wait! Take a moment to subscribe. I’m sure you’ll enjoy future posts, and it’s the only way for me to know that they are worth writing in the first place! Template systems are everywhere. Take any static blog generation system and you’ll find some. Take system management tools like Ansible and you’ll find others. Take a cloud orchestration service like Kubernetes and you’ll find Helm. Heck, even Go’s standard library provides a full blown text template system out of the box. There is clear benefit and appetite for these and, surely enough, it’s tempting to use any pre-existing such system in your own project… but if all you need are a bunch of variable replacements, some of which may be only conditionally applied, you can go a long way by not taking any dependencies. A call to or the substring function of your language of choice is all you need. To put this in context, let’s say you have Bazel’s configuration file, which is not very flexible, and that you need to set some arguments based on dynamic values that depend on the environment. E.g. something like this: If you know a little bit about , however, you may squint at that and say: “That’s silly! Make those flags conditional on a configuration and you’re set!” So you try something like this: And… this does not work. Gotcha! Startup flags cannot be placed behind a configuration so there is no way for you to parameterize the JVM’s max heap value passed in . And having to remember to pass from CI all the time is fragile, because you might forget and not get the desired configuration in place. Solving the above is not difficult if we could parameterize the configuration. We might want to write something like this instead: … and then have and be replaced dynamically depending on some runtime arbitrary logic. We can do that via the Bazel wrapper, and this sort of dynamic configuration is a common thing to do from it. So let’s do this. Let’s start with the template logic: Ugly(?) bash syntax but nothing too complicated: The global hashmap tracks variable names and their replacement values. The function inserts a new key/value pair into . (It’s important that the values given to don’t contain -special characters like , backslashes, or the separator we chose—but we control the generation of those values so we are good.) The function transforms the hashmap into a set of arguments of the form and then calls to process the given input file into the given output file. Then, we can plug everything together into a minimal Bazel wrapper: In this new snippet, the function instantiates the file from the contents of via and then calls the actual Bazel binary provided by Bazelisk in . The complexity here may seem overkill, but it’s necessary : while it’s pointless to invoke Bazel in parallel due to its global lock, users will run Bazel in parallel and you must make sure that the wrapper is reentrant. Otherwise, you’ll definitely run into races. The rest of the script in does the actual work to compute key/value pairs to substitute in your now-templated and then delegates to Bazel via . That’s it. This is a barebones implementation of a text template system using bash—and I had to use bash, not sh, to get the niceties of a hashmap —that serves as a launcher. Go try it. By the way, the and nomenclature are inherited from GNU Autoconf’s AC_SUBST primitive . “Great!” I hear you say in a sarcastic tone. “You have just applied string replacements! But what about conditionals, huh? You CaNnOt Do ThAt So EaSiLy!!11!one!” Ah, but you can , and showing you that trick is the whole point of this short article, remember? The necessary insight is that we can use string replacements to comment out lines in the original file. What if we did this: In here, we are defining different configurations for developer workstations and for CI, like we did earlier, but then we are auto-magically picking the default configuration depending on and . How? Well: will expand to the empty string when running on CI and will expand to , so the corresponding lines will be enabled and disabled. And the opposite replacement values will appear when not running on CI. Ta-da! Conditionals. We can make things nicer with a helper function and meta-programming: Don’t panic about that . Just as with the invocation above where we could have issues with special characters appearing in values, we control the arguments to so the is safe. And note that we can even nest conditionals arbitrarily. There is nothing preventing you from doing: Which corresponds to the conceptual equivalent of: Let’s do loops? Sorry no, can’t do! Well akshually… we could do loops. Not by using simple tricks like above, but we could definitely sketch something like this: However, this is starting to look a lot like a high-level parser, not scripting where you glue simpler components together. And if you are headed that way, you are better off transitioning to a proper programming language and a well-known template system. What do you think? Do you hate this already? You can, but note that the whole world runs on this stuff. All of that foundational code behind Linux systems ends up using GNU Automake and GNU Autoconf, and those packages are full of stuff like this in their and files. And you can get very far with just the above constructs if you treat the shell like a real language . The Bazel wrapper that I maintain at work these days grew to almost 1000 lines of code before I pruned a lot of features that had become unnecessary, but it’s still pretty large. We are now transitioning it to a Go-based wrapper for better readability and maintainability… but as we do this, I’m reminded that well-groomed shell scripts give you some flexibility that no other language can match in just a few lines. So, keep things simple. You can do a lot with just a few primitives. As powerful as Bazel is, sometimes it’s not featureful enough. When using this build system, it’s common practice to wrap it in a launcher script—and in fact, this is natively supported by Bazelisk , Bazel’s native dispatcher that stands for the binary in the user’s . Bazelisk will first download the version of Bazel requested by the project, and then, if exists, invoke it instead of the downloaded binary. is what’s known as a Bazel wrapper and is the point of today’s article. Well, not quite. The actual point of today’s article is to demonstrate a simple trick I learned from the GNU Autoconf and Automake days to implement full-blown conditionals in an ad-hoc template system. But because such trick is trivial once you see it, I have to present it in the context of a modern real-world scenario. So what I’m going to do is guide you through the creation of your very own Bazel wrapper to customize Bazel’s configuration file in ways that the native Bazel tool doesn’t support. Let’s get started. But wait! Take a moment to subscribe. I’m sure you’ll enjoy future posts, and it’s the only way for me to know that they are worth writing in the first place! The context Template systems are everywhere. Take any static blog generation system and you’ll find some. Take system management tools like Ansible and you’ll find others. Take a cloud orchestration service like Kubernetes and you’ll find Helm. Heck, even Go’s standard library provides a full blown text template system out of the box. There is clear benefit and appetite for these and, surely enough, it’s tempting to use any pre-existing such system in your own project… but if all you need are a bunch of variable replacements, some of which may be only conditionally applied, you can go a long way by not taking any dependencies. A call to or the substring function of your language of choice is all you need. To put this in context, let’s say you have Bazel’s configuration file, which is not very flexible, and that you need to set some arguments based on dynamic values that depend on the environment. E.g. something like this: If you know a little bit about , however, you may squint at that and say: “That’s silly! Make those flags conditional on a configuration and you’re set!” So you try something like this: And… this does not work. Gotcha! Startup flags cannot be placed behind a configuration so there is no way for you to parameterize the JVM’s max heap value passed in . And having to remember to pass from CI all the time is fragile, because you might forget and not get the desired configuration in place. Basic string replacements Solving the above is not difficult if we could parameterize the configuration. We might want to write something like this instead: … and then have and be replaced dynamically depending on some runtime arbitrary logic. We can do that via the Bazel wrapper, and this sort of dynamic configuration is a common thing to do from it. So let’s do this. Let’s start with the template logic: Ugly(?) bash syntax but nothing too complicated: The global hashmap tracks variable names and their replacement values. The function inserts a new key/value pair into . (It’s important that the values given to don’t contain -special characters like , backslashes, or the separator we chose—but we control the generation of those values so we are good.) The function transforms the hashmap into a set of arguments of the form and then calls to process the given input file into the given output file.
A cybersecurity startup dangling millions of dollars to acquire zero-day security vulnerabilities in popular software is run by a pair of far-right conspiracy theorists and convicted felons whose most recent ventures included fake intelligence companies and a now-defunct AI-based lobbying platform they operated under assumed names. The X/Twitter account IRIS C2 (@C2IRIS) has gained more than 4,000 followers since its creation in January 2025, posting frequently about security vulnerabilities, AI and software exploits. IRIS C2 says it is a company in McLean, Va. that sells offensive cybersecurity capabilities. The IRIS C2 website dangles the possibility of million-dollar payouts for exploits to attract talent. “Our business model is this,” reads a pinned post on top of the IRIS C2 account on X. “Attract the very best vulnerability researchers and exploit developers in the world to join our company. This mostly revolves around junior engineers with raw talent/extremely high IQ. We don’t care if they have a college degree/industry experience.” The website linked in that profile — irisc2[.]com — says the company is hiring for a number of open positions, and a recent post on its LinkedIn page enthuses about an overwhelming number of applications from potential employees. The website claims IRIS C2 is in the business of acquiring “zero-day exploits, individual primitives, partial chains, and full capabilities across all major platforms. Payouts range from $10,000 to $7 million depending on target, reliability, and operational value.” The government contracting portal g2exchange.com reports that irisc2[.]com is operated by a business based in Virginia called Calvexa Group LLC . The “contact” link on the website for Calvexa Group — calvexagroup[.]com — forwards visitors to irisc2[.]com. G2Exchange shows that while Calvexa Group LLC is registered as a federal contractor, it does not appear to be working on any direct government contracts. A search on the Arlington, Va. address listed in the incorporation records for Calvexa Group LLC finds the property is occupied by Jack Burkman , the 60-year-old founder and managing partner of the lobbying firm Burkman & Associates . When approached with questions about IRIS C2, Burkman referred further inquiries to his longtime associate, 28-year-old Jacob Wohl . Jack Burkman (left) and Jacob Wohl, at a press conference in August 2020. Image: Wikipedia. Burkman and Wohl have a storied history of creating fake intelligence companies and using them to spread false claims about and frame public figures, including fabricated sexual assault claims against then FBI director Robert Mueller , and Pete Buttigieg , then mayor of South Bend, Indiana and a Democratic candidate for the presidency. In 2019, Burkman and Wohl held press conferences falsely alleging extramarital affairs by Sen. Elizabeth Warren (D-Mass.) and then-2020 presidential candidate Kamala Harris . In the wake of the 2020 presidential election, Wohl and Burkman were prosecuted by multiple U.S. states for making thousands of robocalls to residents of battleground states and disseminating false claims about mail-in ballots. They were indicted in Cleveland on 15 felony counts of orchestrating a robocall scheme aimed at suppressing the black vote in Detroit, and were sentenced in late 2025 to probation after their appeals to dismiss the charges were rejected. In 2022, Wohl and Burkman both pleaded guilty to a single felony charge of telecommunications fraud in Ohio, and sentenced to a fine, probation, and community service. In March 2023, a judge in a New York civil case ruled that Wohl and Burkman had violated federal and state civil rights laws, and the two agreed to pay a $1 million settlement. In June 2023, the Federal Communications Commission (FCC) imposed a $5.1 million fine against Wohl and Burkman for their robocall campaigns, at the time the largest fine ever sought by the FCC under the Telephone Consumer Protection Act. Jacob “Jay” Wohl’s GitHub account. By the age of 17, Wohl had started multiple investment firms, and cultivated the nickname “Wohl of Wall Street” after appearing on Fox News in 2015 to discuss his new hedge funds. In 2017, the Arizona Corporation Commission charged Wohl and his investment funds with 14 counts of securities fraud, and ordered him to pay $35,000 in restitution. In 2019, Wohl pleaded guilty in California to four felony counts of selling unregistered securities and was sentenced to two years of probation. The market for previously unknown security vulnerabilities has always been populated by a colorful mix of researchers, academics, charlatans, clout-chasers and people actively involved in cybercrime communities. But the market for selling offensive security services to the U.S. government tends to be far more circumspect. Plenty of government contractors recruit vulnerability researchers and pay for the exclusive rights to novel software exploits, yet none of them do so quite as brazenly and openly as IRIS C2. Recent posts from the Twitter/X account IRISC2 (@c2iris). Indeed, KrebsOnSecurity was unaware of IRIS C2 until last month, when an attendee at a regional cybersecurity conference shared that Wohl and Calvexa Group were pestering people at the conference about selling their vulnerability research. In an interview with KrebsOnSecurity, Wohl said Mr. Burkman was not involved in the day-to-day operations of IRIS C2. Wohl shared that IRIS C2 originally began as a penetration testing company, but shifted its focus recently to selling phone-hacking services to the government. Several times throughout the interview, Mr. Wohl mentioned working on federal government contracts, but when pressed for specifics said he was not at liberty to speak publicly about them. Mr. Wohl said he does not have any formal education or training in computer science or information security, and that most of his knowledge on the matter is self-taught. “I know more about tech than anyone,” Wohl bragged. “My background has always been extremely technical, and I’ve always been deeply into tech. People know me as someone who is able to create spectacularly exquisite capabilities that would make your head spin.” Wohl said security researchers bring the company unique vulnerability findings “on a regular basis,” but that in many cases those findings are preliminary and not fully fleshed-out. “Let’s say someone finds a flaw in a media decoder on a phone,” Wohl said. “A lot of times what we receive is an exploit primitive, where the idea is there but the [execution] needs work. You need that exploit to be stable and reliable, and that’s what we do.” Wohl claims IRIS C2 has approximately 40 employees, although he said none of them are allowed to list their employment on LinkedIn for operational security reasons. In May, the author of the IRIS C2 account on X said that his girlfriend had no idea what he did for a living. But if IRIS C2 has any other employees, they may be similarly unaware of Mr. Wohl’s history of outright fabrications — or even his real name. In September 2024, Politico reported that Burkman and Wohl were bragging about big companies supposedly buying services from their now-defunct company LobbyMatic , which claimed to use artificial intelligence to assist in political lobbying efforts. However, Politico found the pair were running the company using pseudonyms, with Wohl reportedly adopting the name “Jay Klein” and Burkman using the moniker “Bill Sanders.” Politico reported that two of the former LobbyMatic employees resigned after learning of their true identities, while other employees only learned after they had left the company.
Before we begin, I want to say I don't care that much if you disagree with me. There's no sound precise mathematical definition of Object Oriented, no ISO standard, and no grand arbiter who decides what is and what is not OO (although I believe Casey Muratori may have applied for that position somewhere in his 12 day monologue about the subject). You are unlikely to change my mind, and I am unlikely to change yours, and that's perfectly fine. On with the post! In my heart of hearts, Rust feels like an object oriented language. Practically everything I write in Rust is a datum + set of behaviours intimately associated therewith. I hide my struct fields, give my totally-not-objects equality semantics and string representation. I utilise polymorphism via traits - and while I know that traits are not technically interfaces, that fact occupies the same region of my brain that knows that mandrills are not technically baboons. Almost all my functionality is written in methods, and those methods belong to the data they operate on, in a way that they definitely don't in something like C or OCaml. But what about inheritance? OO-haters often fixate on this as the defining thing about objects, which has always perplexed me. I came up in the JavaMania era and "composition over inheritance" was the accepted wisdom of every programmer who took OO seriously. I scarcely used it during my time in the C# mines. Historically speaking the case is weak as well; neither the first smalltalk, nor the first simula had inheritance. Self was hugely influential (traits originated in Self) and didn't have it either. I won't deny it wasn't a common feature, but it never took center stage in my mind outside the brief "Cat inherits Mammal" phrase we all go through, and it certainly wasn't encouraged in the OO design books I read. So why the disconnect? Why does talking about this not resonate with other Rust programmers? I think because in the Rust culture, "objects" are something very different. They're virtual destructors & inheritance trees. They're a nasty thing C++ had that Rust forwent because dynamic dispatch is slow (except in Zig of course where it's fast now). Of course Rust isn't OO! But for those of us who took object-oriented design seriously, everything you read just reinforced that they're neat little black boxes you call methods on. And of those, rust is full.
Birgitta Böckeler now reports on her recent experiences trying local models for coding. She compares them using two standard tasks, and tries out the most promising model for day-to-day use.
Microsoft's Xbox division is conducting big layoffs, as the company deals with abject failure of its Game Pass strategy.
Today, we're releasing the second season of the Now Go Build documentary series. Five episodes featuring technology leaders from around the world solving the hardest problems in healthcare and education.
Our Lego Duplo blocks are in short supply meaning we’ll have to get creative to assemble, build, and rebuild new things. I want a house on wheels! is by far the most popular request here. No problem, we can do that, only to do that, we’ll have to demolish that other structure over there. No? But little brother is playing with that! No he isn’t? But I want a house on wheels! How to defuse this situation? By building a less conventional house on wheels, of course. Or by building a row boat and claiming it’s also a house. Or by attaching a few farm tools on it and claiming it can also act as a tractor. That also works. Hey, no! No brother you can’t have my house! Wait since when is that your house? I’m working on your house give me a minute. Why don’t you build something yourself? No. I can’t. You must. Ok then. No! Brother can’t have my figures! That’s not allowed! Isn’t it dad? Sigh. We have twenty figures. Even if we would have two hundred, I think I’d hear the same complaints. Here’s your house slash thing. Happy now? A collage of six different weird constructions made with Lego Duplo. Daddy? Yes? I don’t want to play anymore . But I just built you the thing you really really wanted? _I don’t want it._Really? I mean I just… No. Sigh. Welcome to my summer holiday… Hey! Little brother, no, you can’t eat that! Wait! I’m sorry, I have to go rescue Peppa Pig’s skirt. Related topics: / lego / parenting / creativity / By Wouter Groeneveld on 8 July 2026. Reply via email .
How a four-minute video taken on a Scottish train destroyed multiple bad tech policy arguments at once.
Imagine being worth $2 billion. Would you give away $550 million? That's a quarter of your wealth, more money than I could spend in several lifetimes. Yet that's how much Evan Spiegel, the CEO of Snapchat, and his wife have donated to a charity in California. Specifically, they donated to Undue Medical Debt, an organization that buys Californians' medical debts and expunges them. A noble act. I'm not one to tell you that billionaires shouldn't exist, or what is or isn't fair in a system I don't control. But one thing I've come to see over the years is that public good deeds are rarely what they seem. Whether it's a feel-good story on the news, a TV show pimping your car, or another turning your house into a mansion, there's an underlying truth that often gets obscured by the appearance of a good deed. Bill Gates, who was once the richest man in the world, pledged to leave almost all his money to charity. Over the years this story has been repeated, and I'm the last person to tell you how to feel about it. For me, it's a good thing to help a charitable organization that's trying to help others. Last year, Bill Gates renewed this pledge, stating that he would give away around $200 billion and keep less than 1% for himself. I mean, $200 billion? That's crazy. Any charitable organization receiving even a fraction of that money will be able to do a lot of good with it, especially when Elon Musk said he could end world hunger with just $6.6 billion. But that's only part of his statement. He isn't leaving his money to any random organization that needs it. He's leaving it to the Bill and Melinda Gates Foundation. That's a different thing. Not a bad thing, but it changes the statement from "I'm donating my money to charity" to "I'm going to use my money to do charity." The money moves from one pocket to another. The point I'm trying to make is that good stories often come with a few asterisks. When I read the Evan Spiegel story, I was intrigued because, well, I'm Californian. I've been here for over 20 years, my kids were born and raised here, and so far I have no intention of going anywhere else. Oh, and I've had one of those surprising medical bills that nearly made me faint. (A story for another day) Reading a story where a billionaire pays off people's medical debt makes you feel good. It made me feel good. But when I looked at the details, things didn't add up. The language of billionaire philanthropy. First of all, the title of the LA Times article said: "Snap CEO Evan Spiegel and Miranda Kerr help erase $550 million in medical debt for Californians" . Note that it says "help erase." The charitable interpretation is that they paid $550 million of our debts. But that's not what it says, it says "help erase." Further into the article, it states: The couple made a multimillion-dollar donation to Undue Medical Debt, a nonprofit that provides debt relief to people in financial need. The organization acquires medical debt in bulk from hospitals, physician groups, collection agencies and other groups for a fraction of the cost. So rather than pay the debt directly, they made a donation to Undue Medical Debt, which had already acquired the debts. And the most interesting part of all this is that Undue Medical Debt acquires debts for "a fraction of the cost." In the article, they admit that the actual amount the couple donated was not disclosed; however, they do explain how much the organization pays to acquire debt: Every $10 donated to Undue Medical Debt relieves an average of $1,000 in medical debt. In other words, they acquire debt for a hundredth of its original value, a penny on the dollar (1/100). 10 years ago, to the date, John Oliver aired an episode of his show where he bought $15 million of medical debt “from Texas at a cost of less than half a cent on the dollar, which is less than 60 grand”. He then paid it off, relieving 9,000 people of their medical debt. It’s a ripe business for anyone looking for a quick PR win. Again, I'm not complaining about this, I'm just doing the math. It's a good thing that they're taking on people's debt and finding rich people to pay for it. That's a good thing for the person receiving the relief. "No one should go bankrupt because of a cancer diagnosis and no family should have to choose between insulin and groceries." It's a good thing: San Diego County residents benefited the most from the donation, with total medical debt relief through the couple's gift totaling roughly $99 million and affecting 40,369 people. In Los Angeles County, the gift provided $26.7 million in medical debt relief to 17,466 people, according to the nonprofit. That's close to 60,000 people benefiting from this relief. But the language keeps circling the drain instead of just telling us what they actually gave. It says "the donation with total medical debt relief through the couple's gift totaling roughly..." They never tell us how much they gave, just how much the medical debt is "roughly" worth. But we can do the math. Undue Medical Debt purchases debt for a penny on the dollar. Evan and Miranda paid off $550 million in debt. At a penny on the dollar, or 1/100, that puts the purchase price at roughly $5.5 million. That's a huge difference in value. Evan and Miranda donated $5.5 million to Undue Medical Debt, "roughly". Why was that so hard to say? Is the value too low? Not good enough? I'd argue it's still very generous. But it doesn't generate the same amount of PR, does it? It's one thing to say you've donated a quarter of your wealth, but a whole other thing to say you've donated 0.27% of your money. I'm not trying to shame them really, I think they should just be honest. Either donate silently, or tell the truth. Why inflate it? It also tells us something about how inflated medical bills really are. If the debt only costs a hundredth of its face value, then patients should be able to pay it off themselves. When my children were in the NICU, we were charged "roughly" $20,000 a night for our two-month stay in the hospital, per child. I'll spare you the math for now. (Again that's a story for another day) There's always an angle to these charitable stories. We celebrated Bill Gates' pledges without questioning that he was funding his own foundation, and that it did more than just charitable work (like oil, fast food, or pharma). We celebrated Warren Buffett's pledges, while he quietly changed his tune in his last annual letter. Evan and Miranda boasted $550 million, while they actually donated only a hundredth of that amount. I'm not saying the money they donated or actions these people took aren't commendable, it's just that it would have been better if they had been honest about it from the beginning. When the rich donate money or make a show of it, there is always more to the story.
Entertainment
Music
Programming
Data
AI
Hardware
DevOps
Go
Gaming
Web
Business
Tutorial
Open Source
Cloud
Design
iOS
Culture
Career
Shell
Security
Rust
C++
C#
Media