Posts in Ruby (20 found)
Maurycy 4 days ago

Notes on optimizing battery life:

Ok, so you have something with a battery, and you want it to run for a long time. I'll be using the classic CR2023 non-rechargeable lithium "coin cell" as an example, but everything here applies to other types of battery. (except the exact voltage and capacity numbers) First off, it helps to measure power draw in current and charge in well, charge. It is tempting to convert everything into power and energy, but don't. Most circuit's power draw is much closer to constant current than constant power: a single clock cycle on a microcontroller involves charging or discharging some number of MOSFET gates. That requires some number of coulombs, not some number of joules. Linear regulators turn any circuit into a perfect current sink: no matter what potential is supplied, the device sees a constant voltage and will always draw the same current. Even if you don't use any, most chips will use a few to generate internal voltages. This is the "typical" current draw of an AVR32DD32 microcontroller over voltage from the datasheet : Black: 25 °C. Yellow: 125 °C. Also, battery capacity is nearly-universally specified as charge, usually in milliamp hours: a 100 mAh battery can support 1 mA of current for 100 hours before it's "dead". (more on what this means later) Non-ideal batteries : This battery has 3 volts stamped right on it... but that's kinda of a lie: Measuring the battery with a meter, the voltage is actually 3.3 volts. However, checking the datasheet, getting the manufacturer's claimed 235 mAh capacity requires operating down to 2 volts: From the datasheet (yes, these have one) With these "CR" Li/MnO 2 cells, the discharge curve is fairly flat: a device that only works down to 85% of nominal (2.6 volts) can still use a good 90% of the capacity. However, an "Alkaline" Zn/MnO 2 1.5 volt cell falls below 80% of nominal with a quarter of it's charge remaining. The manufacturer considers them dead at 0.8 volts — around half the original voltage. In a typical circuit, two batteries will be connected in series to produce a 3 V-ish supply. To get the advertised capacity, the device must be able to run down to 1.6 volts: the same as a (fresh) single cell! Think of supply voltage like a budget : If your battery will drop down to 2 volts and the MCU needs 1.8 V, any other components involved in supplying power must not drop more than 200 mV. It's not that the same MCU won't work on two AA batteries, but it won't be able to use the last 10% or so of capacity because it requires at least 1.8 / 2 = 0.9 volt per cell. Ok, so design for half the nominal supply voltage ? Batteries have non-trivial internal resistance, which causes a voltage drop when any current is drawn: a coin cell is usually around 10 ohms, while large AA cells sit around 0.1 ohms. To understand what causes this, let's look at how a coin cell works: On the negative electrode, a piece of lithium metal looses it's electron and dissolves into the electrolyte. Li → Li + + e - The resulting ions travel over to positive electrode and steal oxygen from the manganese dioxide: 2 MnO 2 + 2 Li + + e - → Li 2 O + Mn 2 O 3 This reaction releases a lot of energy because lithium is an alkali metal the manganese doesn't really care. That released energy is actually what powers the connected circuit. Crucially, the whole thing depends on positive lithium ions reaching and reacting with the positive electrode: moving against the electric field produced by the battery. The open circuit voltage, 3.3 volts, is enough to completly stop the reaction. This is why batteries only discharge once a circuit drains some of the accumulated electrons... but for the reaction to proceed at a reasonable rate, the voltage must drop quite a bit below the measured open-circuit voltage. If you've done any chemistry, it should come as no surprise that this is affected by temperature : As a rule-of-thumb, to operate down to -40 C, plan for ten times the internal resistance at room temp. If you see the voltage rail dropping by 50 mV at 20 C, make sure there's still enough voltage to go around if it drops 500 mV. Another thing that impacts reaction rate is the amount of reagents present , or in other words, the charge left in the battery: resistance increases as the battery is drained. As a test, I discharged an Alkaline battery at 400 mA: Orange: open circuit, blue: under load With a fresh cell, pulling almost half an amp only results in 100 mV of drop, or 0.25 ohms. By the time the battery is half empty, the resistance doubled to around half an ohm. At 60% discharge, the under-load voltage has dropped below the 0.8 V "dead" threshold. Reducing the voltage requirement won't help here: shortly afterwards, the resistance increased so much my test rig needed to supply power to force those 400 mA through. The smaller CR2032 cells start at around 10 ohms, and reach several hundred ohms by the time the open-circuit voltage falls to 2 V. It follows that any circuit that draws a lot of current can not use the full rated capacity. For pulsed loads, large capacitors can help, but they have their own problems which I'll discuss later. Also, batteries get worse as they age . Electrolytes can evaporate/leak and side-reactions can form layers that impede current. There's a good chance you've experienced this: a battery that tests fine on a meter but refuses to actually power anything. What's happened is that it developed a huge internal resistance (many killohms). In series with a high-impedance multimeter, it doesn't create any noticeable voltage drop. When connected to an actual device, the voltage drops to almost nothing. This is why you should be skeptical of any claims of 20 year, 30 year, 50 year battery life. Sure, that might be what you get by dividing nominal capacity by average current draw, but there's no telling how well the battery will work after all that time: I doubt even the manufacture really knows what happens past a decade or two. There's also self discharge , where leakage currents drain the battery, even when it's sitting on a shelf: This is usually given by the manufacturer as percent of capacity per year. Because the cell's voltage doesn't change all that much during discharge, — and the current is quite small — it's a fraction of the original capacity, not of what's remaining. This alone is enough to kill a AA battery in only 5 years depending on temperature (hotter is worse)... but again, this is not the only mechanism at play: Just because self-discharge might suggest a hundred year shelf-life, doesn't mean it will actually work in a hundred years. Another "fun" effect is voltage droop : Drawing current can deplete the chemicals around the electrode, causing a temporary increase in resistance. Applying a 400 mA current pulse to a half-empty ZnMnO 2 500 mAh cell caused the internal resistance to triple over the course 40 seconds: Yellow: cell voltage. Blue: Current Eventually, the battery does recover, but it took a good minute or so: Actually a trace of a different pulse, so the starting voltage is higher. What's interesting is that even though no current is being drawn, the battery circuit voltage is still not back to where it should be. This is where the "resistance" model starts to break down. It's more accurate to say that the pulse temporarily pushed the cell down it's discharge curve: increasing the resistance and decreasing the open circuit voltage. This gets worse when the battery is nearly empty: I applied a similar 10 second pulse to an 80% drained cell, it took around 5 minutes minutes to for it's open circuit voltage rise back above 0.8 volts. This effect highly variable depending on temperature (colder is worse) and state of charge, so it's good to include a wide voltage margin when designing a circuit that will draw sustained current. In short , internal resistance increases when... ... it's cold ... the battery is close to being empty ... the battery is used ... you do nothing at all Plan for a much worse voltage drop than what you see on your workbench: it's possible to loose as much as a volt per each mA drawn with a mostly empty coin cell on a cold night. With that in mind , it's time to look at those capacity numbers. As already discussed, aiming for longer than a decade or so is largely pointless because of battery aging. These CR2023 batters have quoted shelf life of 10 years, so it's going to be my target: From a CR2032 (~230 mAh), a device can draw an average of 2.6 uA if it runs down to 2 volts. From a AA (~3000 mAh) AA battery, a device can draw 34 uA if it runs down to 0.8 volts per cell. ... so we have a voltage budget and a target current. Keep in mind that internal resistance will cut into the voltage if when draw pulses in excess of a few microamps. Measurement techniques: These small currents present a problem: most multimeters don't really do well below a microamp. Benchtop models that can measure down to the nanoamps exist but are quite expensive. On paper, measuring current is easy: Insert a known resistor into the circuit and measure the voltage drop across it... except this either requires adding a large resistance or measuring a tiny voltage. A better way is to use an op-amp to hide the voltage drop from the device under test: The amplifier tries to keep its two inputs at the same voltage, which requires it to exactly match the device's current through the feedback resistor. This results in exactly the same voltage as if it was used as a shunt, except with zero burden voltage. Since most chips have two opamps, I use the other to create a VDD/2 supply rail which is used as the ground. This allows the chip to have access to voltages both above and below it. Most modern chips are "rail-to-rail", meaning they are designed to operate close to one of the supply rails... but this doesn't work too well: Consider what happens when the input current drops to zero. The amplifier has to pull the output (with a non-trivial amount of capacitance) down to zero. If the best the amplifier could do is connect the output to the negative rail, the voltage would exponentially decay, approaching zero but never reaching it. Would this be a huge problem? Probably not. Is it a good idea to make the chip's job as easy as possible? Yes. As a bonus, this allows the device to measure currents in both directions. Using the 100 pA/mV range, the circuit has an offset of ~10 pA, so it's not quite a picoammeter, but it's close. This makes it good for testing the leakage of MOSFETs, diodes, capacitors and the such. However, this design has one huge snag: It's zero burden voltage up to a fairly modest point. Once the output maxes out (100 nA - 100 uA depending on the range), the device will can see the full shunt resistance. This is a non-issue for testing component leakage, but it becomes a problem when measuring the current drawn by a microcontroller. For measuring sleep current, it's best to build a firmware image that never wakes up, and short the meter's input or connect a second power source during startup. Another option is to use a tiny feedback resistor: connecting a 1 kohm resistor between the input and output yields a 1 uA/mV range with a maximum of 1 mA. Once the microcontroller boots, the resistor can be removed to measure it's sleep current. (and if you are drawing more than this, you probably shouldn't) This is also a good trick to avoid crashing MCUs when switching ranges, which can cause a momentary disconnection depending on the geometry of your selector switch. Shielding is not optional : 100 picoamps is a kind of current that floats around on the air. It's best to put the whole setup inside a metal box connected to the meter's ground. Running coax to a scope or meter is fine because the wire's sheath is connected to the rest of the shield: this isn't RF stuff. If you don't have a box, wrapping the whole thing in aluminum foil works almost as well. (make sure it's not touching anything!) Also, it's a little silly to carefully screen out interference only to reintroduce it with a power supply, so it's best to run everything with batteries: Two 1.5 volt alkaline cells provides 3 volts and four is close enough to 5 volts. Also, be careful with what's touching the meter or part under test: a post-it note can easily conduct a whole nanoamp at 5 volts. Wood and fabric are similarly problematic. If in doubt as to if something is a problem, test it. When measuring capacitors, there's a really annoying property to be aware of : The dielectric material can slowly absorb or release charge over multiple hours. This effect is mostly known for recharging high-voltage capacitors after they've been removed from circuit — with unpleasant results — but it can also result in a deceptively high leakage current that goes away if the capacitor is used in a real circuit. Unless you have fancy polypropylene capacitors, you'll have to leave them in the test rig for several hours before taking a reading. Circuit testing : Of course, it's not enough to test individual components. The whole system has to work correctly with an imperfect power supply: A device running on a coin cell should be able to tolerate the full 1k with a two volt supply. ... also, it's a good idea to simulate a dead battery: an empty battery shouldn't result in hardware damage or data loss. Temperature can greatly effect leakage currents. If you expect the components to get up to 80 C, grab a heat gun and see how it performs at those temperatures. Practical advice: Before considering any components, does to circuit board itself consume any power? There's lots of people on forums saying you shouldn't use a soldermask, or that flux on the board causes leakage... For testing, I used a nothing special JLCPCB, green, FR4, 2-layer board. It had two quarter millimeter traces 30 mm long and separated by 2.7 mm. For the measurements, I used a 9 volt bias, which should represent worst case results: Clean : Testing the board as it came from the factory Humid : Breathing on it for a few seconds (99% RH, no visible condensation) Fingers : Touching it to get skin oils on the board Rosin : Spread some RMA flux and burned it with a soldering iron. Board condition and soldermask Current Soldermask, clean < 5 pA Soldermask, fingers < 5 pA Soldermask, humid < 5 pA Soldermask, rosin < 5 pA No soldermask, clean < 5 pA No soldermask, fingers 10 pA No soldermask, humid 30,000 pA No soldermask, rosin 20 pA The main troublemaker is humidity. If you are designing a circuit that needs to work outside, underwater or underground, it would be a good idea to include some desiccants: most plastic will allow water vapor to permeate inside. The soldermask prevented any significant leakage between traces, but problems could still happen between component pins. Conformal coatings will protect against short exposures, but will suffer from the permation problem. Soldering residue or skin oils aren't a problem unless you are doing picoamp metrology. Capacitors : Electrolytic or tantalum capacitors can leak multiple microamps at just a few volts: A jellybean 100 uF 16V electrolytic pulled 26 uA at nine volts, which is ten times the entire current budget for a CR2032! That cap alone could discharge the battery just a year or two. Ceramic capacitors a lot better: I grabbed a random 1 uF capacitor from my parts bin initially pulled several hundred nanoamps, but it dropped down to 920 pA @9 volts after two hours. Even a hundred of these would only draw 92 nA, which is only 3% of the budget. TLDR ; Don't use electrolytic or tantalums. Ceramic capacitors are fine in reasonable quantities and when run well below their rated voltage. Diodes are very commonly used for reverse polarity protection, but there are two possible configurations: A series diode uses a forward biased diode to prevent reverse current from getting to the device. A parallel diode adds a reverse biased diode to clamp the reverse voltage before the device is damaged. In the series configuration, voltage drop is very important : Real diodes are quite different from the idealized model. The voltage drop of a 1N4148 is only 0.6 V at 1 mA of draw and at 25 C. The relationship between current and voltage drop is roughly exponential: For a silicon PN diode, passing 10 times the current requires an extra 100 mV. This also works in the other direction: A circuit that only needs 10 uA (peak) will only see around 0.4 volts of drop across that diode. Temperature affects this: The threshold will rise ~2 mV for each degree the diode is cooled. At -40, expect 130 mV of extra voltage drop compared at room temperature. A Schottky diode has a much lower threshold voltage: 1 mA of current only needs 0.25 V. This can be a huge improvement to your voltage budget, although it's still a non-trivial amount. In the parallel configuration, reverse leakage matters . Because it's highly dependent on voltage, I measured a few diodes at 5 volts, which is closer to normal operating conditions: 2N4148 [PN] @5V: 2.3 nA BAT46 [Schottky] @5V: 2.4 uA In this test, the schottky doesn't do so well: It's three orders of magnitude worse than a similar PN diode. So, use a PN diode right? Well, if the battery can supply 50 mA into a short (fresh coin cell), there might be around a volt across the device. That can be enough to cause damage. So, what's a good reverse polarity protection circuit? An n-channel low-side switching version is also possible A MOSFET can act as a near ideal diode: If the gate (connected to the negative rail) is in fact, the lowest voltage, it's switched on. If the battery is inserted backwards, the gate now has the highest voltage in the circuit and the transistor stays off. However, it's still important to consult the datasheet or conduct experiments: the battery voltage might not be enough to fully turn on the FET, and even a properly "on" MOSFET still has a voltage drop. The final option is nothing: Battery clips that physically prevent a user from inserting a battery backward exist. These have no electrical penalties except for the contact resistance (which is negligible when compared to the battery's). Schottky leakage also poses a problem for dual power supply circuits. A microamp of backfeed into the backup battery can actually be enough to damage it. In these cases, you may be forced to use a PN diode or use a variation of the MOSFET trick: connect the gate to the primary supply rail. This will, at a minimum, perform as well as a silicon diode because of the transistor's intrinsic body diode. Once the power rail drops down to zero, the MOSFET's gate will be negative and it will turn on. However, it's performance won't be perfect if the main rail takes more than a millisecond or so to loose voltage. It's best to plan for a PN diode drop and consider any extra voltage as be a nice bonus. Computers : In theory, CMOS logic doesn't draw any power when sitting idle. In practice, it absolutely does. An 8-bit AVR128DD28 microcontroller draws 1.5 uA during sleep mode. Connecting a 32KHz crystal and using the integrated RTC to provide wake ups bring it up to 1.8 uA. This leaves just 700 uA of average current to work with. Ok, but at some point, the processor has to do something. Each clock cycle has a fixed cost: For the AVR, I measured it at ~0.28 nanoamp seconds, meaning that the battery has enough power for 3,000 billion cycles. Individual clock cycles on an AVR128DA28 running at 32 kHz. However, it's almost always a good idea to use a slow clock: The chip will draw an extra 277 uA of current draw per MHz. At the default four MHz clock speed, that's just over a milliamp. There's no guarantee the battery will be able to supply that kind of power. Decoupling caps aren't going to save you here: 1 mA is enough to drain a rather big 1 uF capacitor at 1 volt per millisecond. (remember, no electrolytics allowed.) Since the MCU has a minimum voltage of 1.8 volts, and the batteries can go as low as two, it's only safe to run like this for 200 microseconds / 800 cycles! However, running at 32 kHz only draws an average of 10 microamps. There are still current pulses from each clock cycle, but there are small enough to that they only drop a 1 uF capacitor by 0.27 millivolts. The processor does draw more a bit more quiescent current while running then in sleep mode. This is why some people suggest you should run at the maximum clock speed to save power... but while it is more efficient on paper, it simply doesn't work with real batteries. This also lets us calculate how long it can run for: 10 microamps is 14 times the remaining 700 nanoamp budget, so the processor can be running 7% of the time. Also, on this particular MCU, wakeups cause a big current pulse: Because of stray capacitance, applying power to the processor costs a whole 2.62 nanoamp seconds. With a 1 uF capacitor, this would drain it by 2.62 mV. However, with smaller caps like 6.8 nF, it could would discharge them a whole 385 mV. Stuff like this is why I'd recommend using around a microfarad: A decent 1 uF (MLCC) ceramic rated at a few times the supply voltage will leak almost nothing. To be fair, the datasheet does recommend this value, but plenty of people are in the habit of using smaller ones: When you have a 5 volt supply, loosing a third of a volt is not a big deal. Using a 3-but-actually-2 volt battery, it's enough to drop below the chip's minimum operating voltage. Some parts claim a much lower sleep current (in the nanoamps), but that's without retaining memory: Most applications can't use these modes. Consider a data-logger. Because flash consumes the same amount of power when writing a few bytes or a kilobyte, being able to buffer readings actually saves power. ... although there are some applications where a feature like this does make sense: This is something you have to consider before taking sleep current specs at face value. ... it's cold ... the battery is close to being empty ... the battery is used ... you do nothing at all Clean : Testing the board as it came from the factory Humid : Breathing on it for a few seconds (99% RH, no visible condensation) Fingers : Touching it to get skin oils on the board Rosin : Spread some RMA flux and burned it with a soldering iron. https://ww1.microchip.com/downloads/en/DeviceDoc/AVR128DA28-32-48-64-DataSheet-DS40002183B.pdf : The discussed microcontroller. https://data.energizer.com/pdfs/cr2032.pdf : Example battery datasheet https://lcamtuf.substack.com/p/real-mlccs-and-inductors-have-curves : Another footgun with capacitors

0 views
DHH 4 days ago

Basecamp Five

I've been working on Basecamp for half my life, and nearly my entire professional career in software. The first code was written in the summer of 2003 when I was just 23. Now I'm 46, and we've just released the fifth major version.  It's an incredible update to a service that continues to help about a million users a day avoid dropping the ball when working with others. It's AI accessible, but not agent hysteric. It's still famously easy to use, still executes the basics beautifully, and still focuses on the small to medium-sized teams we've been serving in the Fortune 5,000,000 for decades. Here are just three of my favorite new features in Basecamp 5: Lexxy editor: Our new text editor finally brings tables, markdown, and live syntax highlighting for code to Basecamp. Oh, and voice notes. It's built on Meta's Lexical editor toolkit, and it's going to ship as the default for Action Text in the next major version of Rails. Keyboard accessible: After moving to Linux, building Omarchy, and acquiring a taste for mechanical keyboards, I've come to love navigating the computer primarily through hotkeys. So with a lot of effort, Basecamp is now a delight to drive through the keys, and you don't have to be a brainiac to remember them all: just hold down SHIFT, and they're revealed in the interface. SHIFT + S opens the sidebar, ESC moves focus between it and the main page, SHIFT + C starts composing a comment/chat line/answer. The permanent sidebar: If you live in Basecamp, like I do, it's to stay on top of all the new things that are constantly happening in a busy account, and that's just gotten so much faster with the new permanent sidebar. Before, we had a Hey! menu in the top bar. You'd get a little dot when something was new, then you'd open it, click, and the menu would close. If you had five things that were new, it'd be open-click-close, open-click-close, five times. Being able to zoom through these now with just the return key, tap, tap, tap, and I've read three new things. So good. And there's so much more. Jason put together a great summary on the new marketing site, which in itself is brand new too. A back-to-basics design in many ways. As our entire industry is getting swept up in agent hysteria (and I love AI as much as anyone!), we thought it better to focus on the human communication that's the cornerstone of Basecamp. The new site just speaks plainly to that mission and shows you the software right at the top. Another thing that's back is color, specifically in the logo. Basecamp's clever but flat paperclip logo has been replaced with a modern take of our original rolling mountains. In full three dimensions, with depth and a gradient. Love it.  Overall, I'm really proud of what we've built with Basecamp Five. We're inching in on a quarter of a century in service! We still have customers who signed up back in early 2004! This is the kind of legacy that makes me beam, and the new version is just ace.  If you've tried Basecamp in the past, it's time to take another look. If you haven't tried it yet, you're in for a treat.

0 views
Kelly Sutton 1 weeks ago

Moving on from React, 2 Years Later

It’s been an even busier year and change for Scholarly . We’re coming up on 3 years in business. We’ve raised a small round of funding from our existing investors, grown the team in both Denver and Seattle, and continue growing in all dimensions. I’m trying to do an annual review of a decision to move away from React in ~2023 to see how things are turning out. You can read the original posts, Moving on from React and Moving on from React, a Year Later . What a wild 18 months it’s been. Since the last post, we’ve moved from tab-completion and copy-paste LLM-aided development to full-on agents with things like Claude Code. We’ve also grown the team and we have reintroduced React (gasp!). The decision to reintroduce React was solely driven by React Flow . It’s the best diagram tool we found, and we thought it was worth eating our hat. Unlike some of the other libraries we use and pay for, it’s not currently packaged as Vanilla JavaScript. We’ve also deployed React in a select few areas where its state management yields the best customer experience. We ship this as small pieces of a page that is otherwise server-rendered. The React bits help us add the interactivity that we believe makes the best customer experience. For those keeping tabs, here’s how our Ruby/JS LOC has changed over time: A few reflections on the numbers above: Given the recent changes in software engineering, it’s hard to tell how much of this even matters anymore. Our roles as software engineers are changing with every model or harness upgrade. Agents and models have gotten a lot better at interpreting and using StimulusJS and Turbo . We use Claude Code with Opus 4.7 at the time of writing. Some of the rough edges of using Turbo with LLMs in the beginning feel completely gone now. Kept this one short to provide an update. Things are changing quickly, and it’s kind of interesting to think how much of this may or may not matter in the long run. If the LLM is writing our code and the customers have a great experience, how much does stack choice matter? Maybe we should index toward more complex technologies for humans but easier for LLMs to write? How much control should we cede? Thanks for reading. Until next time. Our codebase has 179k LOC of Ruby, compared to 61k from 18 months ago. A tripling! This can be somewhat attributed to our adoption of Sorbet for static type-checking. It just produces more verbose Ruby and provides some more safety that certain parts of our code base benefit from. Our JS LOC went from 4.1k to 14.8k in the same time frame. We’ve also adopted TypeScript here for some of our files that touch React. I’ve kept the linear trendline to simulate where we might have been with React. We’re still below where I’d predict we’d be had we stuck with it. You can clearly see where we made the cutover from React to Stimulus in August 2023, although it’s not as obvious since it’s so far in the past. Our Ruby LOC was growing super linearly last time, and that continues to be the case. I attribute this solely to Claude Code. It really whips the llama’s ass. Volume of LOC remains a liability, but the product capability has grown about this much or more in the meantime, so not concerning.

0 views
Max Bernstein 1 weeks ago

Travel notes: RubyKaigi Hakodate

I just got back from a three and a half week trip to Japan. It was the longest trip I have ever been on (aside from studying abroad in Germany, which felt different). I made the following wild circuit with only a backpack and a duffel: This trip was split into three parts: time with my immediate family, going to a conference, and then time with my partner. They were all great and also I am glad to be home. I’ll post my abbreviated travel notes here, including activity and food recommendations. We started in Tokyo but we were only there for about 40 hours. We focused our time mostly on arts and crafts: we did a kintsugi workshop, spent time at an artists cooperative, and then did a lot of walking around. This was a good intro to the trip, because everyone kept waking up at 4am and crashing at 7pm due to the jet lag. 4am wakeup makes for nice morning walks to 7-Eleven. I brought my family to T’s Tantan in Tokyo Station because I’m vegetarian and it’s otherwise hard to find ramen that approaches kosher in Japan. It continues to be great and I really appreciate having a steady vegetarian option available. Many years ago when I visited Tokyo there was a place that served a delicious tomato-based vegetarian ramen, but I hear it has since permanently closed. Bummer. We took the shinkansen to Kanazawa. I love the train. It’s fast. It’s quiet. You can eat your snacks on board and gaze out the window as the world whizzes by. It’s nice. We toured a soy sauce factory (meh; they don’t let you in the room where the magic happens) and the old town (pretty!) before finally eventually ending up at our small hotel in Toyama: Satoyama Auberge Maki No Oto. I highly recommend this hotel. It is beautiful, the staff is lovely, the food was excellent, and they were very accomodating of me being vegetarian. We continued on to Toyama, which is a port town. We got to talking with an older local guy who told us all about his favorite local spots. We learned after leaving that this guy has extraordinarily fancy taste and they were all either Michelin starred or at least Michelin rated and with a lead time of months. We opted to instead go to a local brewery, which had a ghost pepper beer (!) and pizza. We then moved on via train to Osaka, where we transferred to a car to head (eventually) to our hotel in the hills near Nara. We toured the Daimon sake brewery. They explained every little thing about the process, which was especially interesting to me, as I’ve done some small amount of homebrewing and I bake. They sounded similar. We had a tasting and even got to talk to Daimon-san. I recommend going. I also recommend the Akame 48 waterfalls walk/hike, which has some exquisite falls, and Murou Art Forest. They had some really wonderful installations. My brother and I parted ways from the rest of my family in Osaka: they headed further west and we headed north to Itō on the Izu peninsula. We got a surprise perfectly clear view of Fuji along the way. It’s beautiful there. They don’t seem to welcome foreigners in a lot of their restaurants (we were turned away several times) but one place had a guy who enthusiastically welcomed us in. We ended that evening enjoying a some food and a beer while also being stared at by a 300lb completely tattooed guy. It was a little unsettling but we left without incident. My brother and I made our way to Tokyo for the day before his flight and before my train north to Hakodate for RubyKaigi. I once again did that thing where I walked around in humid 80F heat with a large backpack and pants and was extraordinarily warm toward the end of the day. After about a liter of Aquarius on the train north I felt better. I stayed at Yunokawa Prince Hotel Nagisatei which I would like to especially call out for having an enormous, diverse, and very vegetarian friendly breakfast. Every morning I got to try new and tasty things and even feel full after. It was great. Hakodate is beautiful in the spring. I arrived at peak cherry blossom season and Goryokaku, their star shaped fort, is absolutely decked out in cherry blossoms. It is also moderately swarmed by tourists (in this case, three cruise ships). It didn’t feel over-crowded though. I enjoyed eating at The Bear King which had a vegetarian friendly option. The next day was the committer meeting. I don’t remember a ton from it other than people talking at length about the semantics of deep freezing an object (do you freeze its class? its class’s superclass? …?). I picked up my badge and also got to check out my colleague Chris Salzberg’s bar SOLENOID ! It’s a neat spot. I headed out to go find some dinner. This is about when I got a message on my phone that there was going to be an earthquake, so I walked back into the bar and said “hey, did you get this?” just before everything started shaking. It was the biggest earthquake I’ve experienced, but I was metaphorically not too shaken up. Then we got the tsunami warning. Chris’s bar is already something like 8 meters above sea level and at the foot of Mt Hakodate. With the city sirens going off and the police directing traffic with batons, though, I decided my best bet was just to march directly up the mountain to get more elevation. Since the tsunami wasn’t scheduled to arrive for about 20 or 30 minutes and my hotel was across the sea-level part of town, I parked myself on a little concrete post. Chris found me eventually. Someone told us that there was a middle school offering refuge, so we went and hung out on the side of the gymnasium. They were really nice about it. On Wednesday, the conference started. It was really well signed and organized. My usual complaint with conferences is that there’s nothing to eat for vegetarians (or that we get mashed with the gluten-free people and each group only gets a salad and bad bread) but that did not happen! They had really stellar vegetarian bento. They had a lot of leftovers toward the end of lunch so I even went and got a second. This was about when I started freaking out because my speaking slot was approaching and I wasn’t yet feeling my talk. Normally when I give a talk, I get up in front of people and I pace and gesticulate and productively complain and throw in some fun anecdotes and the audience, one way or another, ends up learning about JITs at scale, or Scheme semantics, or something. It’s what I’d done for my little lunch talk at Brown two weeks prior. I even titled that talk One must imagine compiler engineers happy so there was plenty of room for educational complaining. But this RubyKaigi talk was in front of an enormous crowd and toward a more general audience than I was used to addressing. The slides did not feel like they were flowing until about twenty minutes before my talk. In the end it went alright. I realized about 40 seconds in that I had way too much content so I ended up speaking rapidly for 30 minutes straight, completely unaware of the audience (which you can’t see anyway because of the lights). I only really noticed people when I made a dumb six-seven joke and Aaron laughed. The rest of the conference I was able to relax and enjoy other people’s talks. I got some good hallway track in, too. I think there’s a good group of people who are interested in Ruby tracing (for example, Perfetto in ZJIT ) so maybe we will make something happen. We had a nice small dinner at Yasai Bar Miruya , which was vegan (!) and had some nice sake. The host was very friendly, too. I nerd-sniped John and J into implementing a VM for the Universal Machine . This was a daunting homework assignment back in undergrad but it was a fun project later in life. S joined toward the end of the conference. She’s also vegetarian so we got some really excellent vegetarian ramen at MAIDO Ramen . Finally, S and I headed south on the shinkansen for Nikko. Nikko is small, beautiful, and a tourist day-trip town. Dinner closes early. Shops close earlier. Since we were staying there we had to make sure to track down and visit the one or two vegetarian places before they shuttered. S and I, along with J and J, took the bus up from Nikko, up the windiest switchbacks, to the Kegon Falls. We were going to take a boat across the lake, but the water level was too low for the dock on the other side, so we ended up half hiking and half taking a bus. Then we continued our hike through the Senjōgahara Marshland (beautiful), to the Yudaki Cascades (lovely), which also had a surprise restaurant and ice cream shop at the base! It’s called Yutaki Rest House . After some great (vegetarian friendly!! wow!!) udon, we marched up the waterfall and around Yuno Lake at the top to Yumoto Onsen. In order to make the last reasonable bus back to town, we just enjoyed putting our feet in the foot bath. One day was rainy. In the evening, J and I thought it would be fun to continue our Universal Machine implementations. As Norman Ramsey would say, “my implementation is 90 lines long and runs sandmark in under six seconds.” We also enjoyed doing a tour of the shrines right above Nikko. The shrines are resplendent against the backdrop of forest. Pro bus tip: you can either pay by IC card or credit card. No need to grab a ticket if you do that. S and I shipped our bags (thanks, Yamato) before continuing on to the small town of Moka, the staging area for our big pottery festival day. Unfortunately, there was no good way to get there: there was no reasonable series of trains and no taxi would take us. Ultimately we ended up taking the train to Utsonomiya and catching the long local bus to Moka. About twenty minutes into this ride, in the middle of nowhere, bus nearly empty, the bus driver pulled over and ran over to us looking kind of panicked. He asked where we were going and was visibly relieved when we said Moka. I suppose we are not the usual riders. Very nice of him. Upon arrival, S introduced me to CoCo ICHIBANYA, which is also super vegetarian friendly. I loved it. We ate really well before walking to our tiny hotel. We did not really know what to expect from the Mashiko pottery festival. The internet said it would be crowded and to arrive early, so we got up at 6:30am for estimated 7am departure on the tiny train from Moka to Mashiko. On most trains you can pay with an IC card but we were out in the sticks so we asked the only other guy on the platform how to pay for the train. He said he had no idea and that this was his first time here. When the train showed up completely packed to the gills and we had to (politely) push onto it, we started to realize that this was The Event and it was going to be mayhem. Also, fun fact: the way the Moka train payment works is that you grab a little ticket from the train, and, upon arrival, wait in line to present your ticket to two very overwhelmed looking people at a table, who charge you, and you pay in cash. Onto Mashiko: the festival was packed . There’s pottery everywhere the eye can see. There are tents and there are full buildings. It varies in quality and artistry from fine to jaw-droppingly spectacular. You could completely stock your kitchen from this fair alone and it would even be cost-effective. The main bummer for us is that we had to get pottery safely back home. We limited ourselves to a reasonable assortment but we really wanted to buy a beautiful painted 20 inch plate with a bird on a branch. After a ton of walking around, we took another long long bus back to Utsonomiya and continued onto Karuizawa. We didn’t know what to expect from Karuizawa but, having been, I could probably concisely describe it as “Aspen for people from Tokyo”. It was… fine. We loved our hotel, Tsuruya Ryokan. The manager was very excited when we borrowed a Studio Ghibli DVD from their collection. We continued on to Tokyo, our final stop. We our usual tour of stationery stores and bakeries—the bread was something to write home about (har har). We enjoyed a (vegetarian!! friendly!!) kaiseki meal at Hyoki Shabu-shabu Ginza before enjoying some live music at Rocky Top . We also recommend Jikasei MENSHO for vegetarian ramen. Bakery checklist: We had an uneventful and reasonably easy trip home. Whew. Long post for a long trip. See you next year in Miyazaki! BOUL’ANGE NIHONBASHI (check! good croissants) Bricolage bread & co (check! good everything) Brasserie Viron Marunouchi Beaver Bread Bricolage bread & co. Bartizan Bread Factory Gontran Cherrier Tokyo Aoyama Shop Comme’N Tokyo Shiomi Bakery The Little BAKERY <!– https://www.jocjapantravel.com/kanto-tokyo-bakeries/ –>

0 views
Rob Zolkos 3 weeks ago

Watch Your Agents

I’ve been telling developers to watch their logs for years. Not just when something is broken. Not just when production is on fire. Watch them while you are building. Your logs are the closest thing you have to x-ray vision for a web application. Click a button in the browser, watch the request move through the app, and you can see what is really happening behind the scenes. The habit is simple: keep the server log visible while you work. When you do, you start spotting problems long before they become production issues: The logs give you immediate feedback. They make the invisible visible. Coding agents need the same treatment. When you are working with an agent, do not just look at the final diff. Watch what it is doing. Watch the commands it runs, the files it opens, the mistakes it repeats, and the little bits of glue code it keeps inventing along the way. That is the agent equivalent of watching your development log. You are not only checking whether this turn succeeded. You are looking for patterns that can make future turns better. Most coding agents keep some kind of session history: transcripts, tool calls, command output, file edits, errors, retries, and sometimes timing information. Those logs are useful after the fact. Point the agent at its own session logs and ask it to look for patterns: A prompt I like for this: This is the same habit as watching the Rails log after clicking around a page. You are looking for the part of the system that is doing too much work, guessing too often, or hiding useful signal. A useful signal is when the model keeps generating code to do the same mechanical task. For example, imagine you have a skill for publishing blog posts. Every time you run it, the model writes a small Ruby or Python snippet to: If the agent is generating that code every time, that is a smell. The model is doing work that should probably be deterministic. Ask the agent to turn that behavior into a script: Then update the skill so future agents call the script instead of improvising the logic. Bad pattern: every publishing session, the agent manually inspects YAML front matter and tries to remember the required fields. Better pattern: create that exits non-zero when , , , or are missing or malformed. Now the agent does not need to reason about the rules from scratch. It runs the command and reacts to the result. Bad pattern: the agent repeatedly writes one-off Python to resize screenshots, compare image dimensions, or calculate visual diffs. Better pattern: create with clear output like: The agent can use the result without reinventing image processing each time. Bad pattern: the agent keeps constructing ad hoc SQL to answer common questions like “which users have duplicate active subscriptions?” or “which jobs are stuck?” Better pattern: create named scripts or Rails tasks: Now the workflow is repeatable, reviewable, and safe to run again. Bad pattern: the agent writes custom code every time it needs to build a fake webhook payload or API response. Better pattern: create or a small fixture library that produces known-good examples. The agent stops guessing at payload shapes and starts using something the test suite can trust. Moving repeated agent behavior into deterministic tools gives you a few wins: Watch the agent the way you watch your logs. When you see friction, repetition, or uncertainty, ask whether the agent needs better instructions or a better tool. Sometimes the answer is a clearer prompt. Sometimes it is a skill. And sometimes the best thing you can do is take the fragile reasoning out of the model entirely and give it a boring, deterministic script to call. That is not making the agent less useful. That is making the whole system more useful. the same query firing 50 times because of an N+1 a page that feels fine locally but is doing way too much work a slow query that needs an index an unexpected redirect or extra request a cache miss you thought was a cache hit a background job being enqueued more often than expected parameters coming through in a shape you did not expect What tasks did you repeat multiple times in this session? What code did you generate only to throw away later? Which commands failed, and what would have prevented those failures? Did you write any one-off scripts that should become checked-in tools? Did you repeatedly search for the same files or project conventions? Were there project rules you had to infer that should be documented? Which parts of the workflow were deterministic enough to automate? What should be added to , a skill, or a script? If a smaller model had to do this next time, what tools or instructions would it need? parse front matter validate the title, summary, badge, tags, and date derive the final filename move the draft into Dependability: the same input produces the same output. Determinism: fewer “creative” variations in routine work. Testability: scripts can have tests; improvised reasoning usually cannot. Reviewability: a script can be read, improved, and versioned. Cost: once the workflow is encoded, you may be able to use a smaller model for that task. Speed: future turns spend less time rediscovering the same procedure.

0 views

Rails Security, AI, and IBB

For quite a few years the Rails project has been working with the Internet Bug Bounty (IBB). The IBB is an organization that awarded cash to security researchers that reported issues to OSS projects participating in the IBB. For quite a while I wasn’t certain about my feelings toward the program because I felt like cash rewards could incentivize low quality reports as well as encourage reporters to “haggle” about the severity of a particular bug (the IBB paid more when the bug was more severe). In the beginning that certainly was the case. We were fielding many low quality reports, and people were haggling over severity. But the program evolved, and despite the never-ending haggling, I felt it did more good (rewarding security researchers) than bad (forcing the security team to wade through low quality reports). That is, until AI came along. Sometime in 2025 our team started getting inundated with low quality AI generated reports. I know for sure this wasn’t unique to just our team as well. Anyway, AI lowered the barrier to generate reports, so we were back in the era of wading through low quality reports. Only this time, the low quality reports were masquerading as high quality reports. AI made it easy to turn a bullshit problem into something that looked legit, and since there’s a possibility of money involved people tried to take advantage of the situation. We even had a report where someone forgot to delete the AI generated output and just uploaded the report as-is with the following text: I enjoy using AI, but I really don’t like AI being used on me. But that’s not what this post is about. Recently the IBB stopped accepting new submissions. In other words, they aren’t paying bounties to security researchers anymore. I don’t know for sure since I haven’t asked them directly, but I suspect this is due to so many projects being inundated with AI generated reports. I think putting a stop to bounties makes sense for the time being. Of course the downside is that legitimate researchers are no longer incentivized to report bugs to OSS projects. Finally, the Rails team didn’t actually handle paying out any of the bounties. After we accept and release fixes, the IBB took care of the bounties and we had no visibility into that process. Since the IBB has stopped accepting new submissions and paying bounties, we’re now tasked with playing customer support for IBB as many reporters are now asking us “are we getting paid?” I honestly don’t know what to make of this situation except that working in OSS security will always find new and interesting ways to suck. I don’t have any particular “call to action” for this post, but I hope that it gives people some kind of glimpse into how the tofu is made. Anyway, have a good day, and remember: It’s always Friday somewhere!

0 views
iDiallo 1 months ago

Don't use localhost:3000, use your own custom domain

After presenting a demo of how an internal tool works, I was flooded with questions. Not about the tool, but about why I had bought a domain just to run the demo. "Why didn't you use the staging server?" they asked. I was confused. I didn't buy a domain. I was running it locally. But instead of the URL being , it was a fully formed domain. . In fact, some people told me that they couldn't access the website on their devices. They thought I had to whitelist their IP to grant them access. To feel young again... Setting up a custom domain locally was common practice when I started web programming. But with the advent of Node.js (and rails?), everyone has resorted to just pointing to with an incrementing port number. The main reason is that the webserver is often bundled into the application itself. It’s easy to just run and call it a day. However, if you have multiple long-term projects running locally, especially if they need to communicate with one another, then managing a mental map of ports like , , and quickly gets tiring. This is where my old school approach shines. By combining the system hosts file with a reverse proxy like Nginx, you can run different projects locally with actual domain names. I usually end up with for active development, for a stable local build, and the actual production URL for the live site. Here is how to set it up. First, we need to tell your computer where to find these domains. Think of as your computer's personal contact list. When you type a URL, your computer looks here first. By adding an entry, you are telling your computer: "Don't bother checking the internet when I ask for myproject.com, I am actually talking about this machine." It creates a manual override that maps a friendly name directly to your machine's IP address. You can edit the file here: Linux/macOS: Windows: Open the file in your editor. In this file, right after the block of entries for Adobe (active.adobe.com...), add this line: Now, when you access those domains in your browser, they don't point to the wider internet, but directly to your own machine. Now that the domain is pointed to your own machine, we want to redirect it to the right application. If your app runs on port , navigating to will default to port and fail. This is where Nginx comes in. It listens on port and forwards the traffic to the specific port your app is running on. Here is a simplified Nginx config to make it work: Restart Nginx, and voilà! You have clean, professional URLs for your local environment. If you are running your services inside Windows Subsystem for Linux (WSL2), networking is handled a little differently because the Linux instance has its own virtual IP. You can get your instance's IP address with this command: You would use that IP address in your Windows hosts file instead of . After that demo, some people were disappointed to learn the trick. They thought I was so committed that I had bought a domain name just to give them the raw deal with my demo. Someone mused about a shirt with the words "real men don't use localhost:3000". That could have started a whole new motivational speaking career for me. A custom domain just looks very professional and is practical for separating environments. It just feels cooler than staring at all day. That's how you separate yourself from vibe-coders. Anyway, back to earth. I feel like this is a lost skill and I'm keeping it alive by sharing it. That's how you run a custom URL locally.

0 views
daniel.haxx.se 1 months ago

High-Quality Chaos

As I have been preparing slides for my coming talk at foss-north on April 28, 2026 I figured I could take the opportunity and share a glimpse of the current reality here on my blog. The high quality chaos era, as I call it. I complained and I complained about the high frequency junk submissions to the curl bug-bounty that grew really intense during 2025 and early 2026. To the degree that we shut it down completely on February 1st this year. At the time we speculated if that would be sufficient or if the flood would go on. Now we know. In March 2026, the curl project went back to Hackerone again once we had figured out that GitHub was not good enough. From that day, the nature of the security report submissions have changed. The slop situation is not a problem anymore. AI slop rate The report frequency is higher than ever. Recently it’s been about double the rate we had through 2025, which already was more than double from previous years. Number of hours between security reports The quality is higher. The rate of confirmed vulnerabilities is back to and even surpassing the 2024 pre-AI level, meaning somewhere in the 15-16% range. Confirmed vulnerability rate In addition to that, the share of reports that identify a bug, meaning that they aren’t vulnerabilities but still some kind of problem, is significantly higher than before. Share of reports that were bugs, not vulnerabilities Everything is AI now Almost every security report now uses AI to various degrees. You can tell by the way they are worded, how the report is phrased and also by the fact that they now easily get very detailed duplicates in ways that can’t be done had they been written by humans. The difference now compared to before however, is that they are mostly very high quality. The reporters rarely mention exactly which AI tool or model they used (and really, we don’t care), but the evidence is strong that they used such help. I did a quick unscientific poll on Mastodon to see if other Open Source projects see the same trends and man, do they! Friends from the following projects confirmed that they too see this trend. Of course the exact numbers and volumes vary, but it shows its not unique to any specific project. Apache httpd, BIND, curl, Django, Elasticsearch Python client, Firefox, git, glibc, GnuTLS, GStreamer, Haproxy, Immich, libssh, libtiff, Linux kernel, OpenLDAP, PowerDNS, python, Prometheus, Ruby, Sequoia PGP, strongSwan, Temporal, Unbound, urllib3, Vikunja, Wireshark, wolfSSL, … I bet this list of projects is just a random selection that just happened to see my question. You will find many more experiencing and confirming this reality view. When we ship curl 8.20.0 in the middle of next week – end of April 2026, we expect to announce at least six new vulnerabilities. Assuming that the trend keeps up for at least the rest of the year, and I think that is a fair assumption, we are looking at an estimated explosion and a record amount of CVEs to be published by the curl project this year. We might publish closer to 50 curl vulnerabilities in 2026. Number of published vulnerabilities Given this universal trend, I cannot see how this pattern can not also be spotted and expected to happen in many other projects as well. The tools are still improving. We keep adding flaws when we do bugfixes and add new features. Someone has suggested it might work as with fuzzing, that we will see a plateau within a few years. I suppose we just have to see how it goes. This avalanche is going to make maintainer overload even worse. Some projects will have a hard time to handle this kind of backlog expansion without any added maintainers to help. It is probably a good time for the bad guys who can easily find this many problems themselves by just using the same tools, before all the projects get time, manpower and energy to fix them. Then everyone needs to update to the newly released fixed versions of all packages, which we know is likely to take an even longer time. We are up for a bumpy ride.

0 views
Chris Coyier 1 months ago

Stories from Alaska Folk Fest 2026

[Folk Fest] is not an intellectual experience, it’s an emotional experience. Visiting Alaska gives me the feeling that people are chasing after when they travel: a little taste of what it’s like to be a part of another world. To live another version of life. Not just looking at it or fantasizing about it (which are fun too), but getting to live it for a little while. I’m lucky enough to have visited Juneau a number of times. My friend Justin Shoman lives there. President of the radio. His deep connection with the community makes the trip more fun than it might be otherwise, as I get to sidecar all that community goodness. Last year, I came up for the 50th annual Folk Fest , and it was a no-brainer to come back for the 51st. The 50th was such a milestone that documentarian Paige Sparks took the opportunity to make a literal movie about it, “50 Years of Folk Fest”. I caught a screening of it at KTOO and got to briefly meet Paige, who did a wonderful job. The documentary was a brisk 50 minutes and managed to explain the history without being boring, like how the original bylaws of the organization require the event to be free. It spotlighted some long-timers with zinger quotes, like the one at the top of this blog post, then focused on some of the new faces of folk fest, like Taylor Dallas and Annie Bartholomew , giving it modern relevance and freshness. A great thread in the documentary featured an awkward fella struggling with his own musical abilities and belonging. He blossomed into performing a really lovely original folk song that couldn’t have fit in anywhere better than Folk Fest. OH, I’M ALSO IN IT. There is a quick moment from an old-time jam at Amalga Distillery where you can see the back of my head. I loved that jam dearly last year and was sad that Amalga didn’t do it this year. They had make-your-own peanut butter and jam sandwiches (get it). C’mon that could have been a whole thing. When I landed in Juneau and walked out of security, I was relieved to see that my favorite plaque is still there. Thanks, plaque. I can’t wait to check out those additional displays throughout the terminal. I had some anxiety arriving. I didn’t get there until Thursday, DAYS LATE, so I had some FOMO — like I had already missed amazing opportunities. That feeling wore off quickly. I b-lined it to Devil’s Club , where I had tons of great jams last year. There was a great jam going on as I got there with Chaz from Ketchikan/Dude Mtn, Evan from Astoria/The Strongbacks, Rosemary from Fairbanks/Writing, and several others. Comradery was immediate. My friends Amy, Roger, Dave, Dennis, and Laura were there, all from various cities in Oregon. I think it was a first for most of them. I haven’t talked to them since leaving, but Amy was dreaming of getting two hotel suites next year instead of just one. One morning, I jammed with them in their hotel suite. It was a weird jam in the key of E, with the fiddles in calico tuning, which is fairly unusual for Old Time. I was on guitar and loving it. Heidi from Fairbanks is there, whom I love because of her unabashed love of banjos. The more banjos the better in her world (there are plenty of situations where people like to keep it to one banjo). She’s also very good, so I learn a lot. The book I read during the trip was an Alaska book I’ve been waiting to savor: Of Bears and Ballots . It delivers. It’s Heather Lende, of If You Lived Here, I’d Know Your Name fame. I’ve read a lot of Alaska books, but nobody evokes the feeling you get there like Heather, even as a mere visitor like me. I also picked up  The Tao of Raven , which I’ve only just started, but it starts with a lavishly wordy version of the fable where the Raven frees the sun, which I’m fond of. I have a version of the raven story that I typeset and letter-pressed myself, and my mom watercolored over, in my guest bathroom at home. Speaking of my banjo, I checked it on Alaska Airlines on the way up. I love my banjo, and it’s nice, but I’m not precious about it and don’t love schlepping things through airports. Some people gasp at the thought of checking an instrument. Well, here are some more points for them. The peg for my 5th string must have loosened and straightened out, causing a buzz as it went over the little mini nut on that string. That’s not an acceptable state to leave the banjo in for Folk Fest, so I had Justin swing by a shop to grab some wood glue, then did emergency surgery on it. I yanked out the peg with a channel lock, rotated it back correctly, then glued it up and hammered it back in. Not pretty, but it’s held up just fine since then. A bar that doesn’t seem to officially participate in Folk Fest (but is at the heart of it anyway) is The Triangle. It ends up being kind of a home base or where to go sit in lieu of any better idea. It’s a place that ends up generating memories for me. A drunk local buying us shots for listening to his life story. Two mandolin players trading fascinating chord transition licks. A beautiful woman frantically trying to find her friends, only to be calmly distracted by the historical photos on the wall. I promised to tell her what I know of them when she comes back, but alas. One of the many cool things KTOO does, in addition to the studio-audience shows, documentary screening, and all that, is to put every main stage performance on the radio. Every second of it! Plus they stream it so people around the world can listen. Driving around, or if we happen to be at Justin’s spot, we’d usually have it on. One thing we caught that way was Sea of Heartbreak (feat. Katy Harris, Caroline Oakley, Reeb Willms, Ava Honey, Pharis Romero). Kind of a supergroup of old-time ladies. I only know exactly who it is now, because it was so good on the air, I looked it up on the official website. One day, sitting at the Alaskan, I was chatting with the bartender, Morgan, who used to run the place. It seems people, bartenders especially, live in this palpable daze of excitement and exhaustion during Folk Fest. The next day, after a nice beach walk “up the road”, as they say, at Eagle Beach , we stopped into Squirez, a cozy little bar that overlooks Auke Bay. It was Morgan bartending again. There was an awful lot of bartender overlap like that. Just the night before, the day bartender at The Alaskan was working the door bar in the evening at The Crystal Saloon. Morgan is extra fun, though, as she travels a lot to interesting places and seems to be doing interesting things with her life, like starting a new gig at Uncruise. She also works at the Lucky Lady, although I didn’t see her there. At Squirez, she did a little rave about what’s so great about Folk Fest. It’s the end of winter (this was a rough one up there), and it’s before the cruise ships come. So it’s a week that feels like a special treat just for the locals. A beautiful gift. Morgan was on the same flight out on Tuesday morning as I was. It was nice to high five out along with another friend (a board member of KTOO) I met at the corndog brunch who had a daughter the same age as Ruby running around. That made me miss Ruby and think of my hope that Ruby and I get to share a love of music and community events one day. One particularly fun live show was Raisin’ Holy Hell at The Crystal Saloon. There were a bunch of rowdy old-timers in the band (some faces I recognized from the documentary) who really got after it and made a ruckus of a show. They played classics like Angeline the Baker and Stickin’ to the Union, mixed with Sublime covers and modern shit like that to switch it up. They had a drummer and a solid bass player holding it all together and making it more than worthy of the killer night slot it had. The whole audience was super into it, and I was having a great time. This feels weird to write, but one of the things that fed into the fun and the feeling of living a different life for a moment is that I’m essentially single now and approaching the point I’d be ready to date (long story, private). Chatting with single strangers can have that hey, is this… something? feeling that can be exciting if a little emotionally dangerous. In my real life, I’m a dad and a co-founder of a busy tech company, and I wouldn’t have it any other way. But once in a while, I can LARP as a freewheelin’ banjo-playin’ Alaskan. Another day, I popped into The Alaskan only to be perfectly on time to catch The Strongbacks , a sea-shanty group of five dudes that I quite like, hosting a “vocal jam”. I was surprised at how many sea shanty enthusiasts showed up. Half the people in the audience were mouthing along to the songs. An Irish session in the back of the bar didn’t stop playing for them, which made me furious. I considered saying something, but ultimately chose not to, as somehow nobody else seemed to care. Not even the bartender? Perhaps, as this wasn’t an official show and the jam had just as much right to make sound, asking them to stop would have been an injustice in its own right. Whatever, I’m still mad about it. The beauty of unamplified harmonizing voices should always take precedence over a mediocre Irish session. Just move! There is so much going on at Folk Fest, you’re definitely going to miss more than you do, even if you shortlist stuff you’re especially interested in. Here’s my list of things I would have liked to do but just… didn’t get to: That’s a big list. And yet: no regrets. Bocca al Lupo hosts a Corndog Bruch at 11am on Saturday. I missed it last year so I was glad to catch it this year. Arriving at 10:40am, there were already a few dozen people in line ahead of us. They passed out paper fliers detailing the gourmet corndogs that would be available. You were supposed to pass the paper back, but you could tell nobody wanted to actually be the one holding the paper. Way too much responsibility for a hungover Saturday morning. I had the elote and the honeybutter, both extraordinary, but I eyed up pickle-style with envy. The cashier was drinking a Bush NA. It sounded good at the time, so I ordered one. She had brought it from home. The band playing at the corndog brunch was The Heists , the last name of the lead couple, fleshed out by a great fiddler and bassist. Importantly: they replaced words in the songs with corndogs and corndog puns. Will the circle be a corndog and the like. I would have liked to be consulted on this endeavor, as I like to think I could have gotten the corndog integration density even higher. I recognized [Andrew] Heist from previous visits as I think he played in the band Taking Care of Bluegrass, which I’d seen a couple times, and saw again on this trip, but he didn’t seem to be in anymore. Possibly because he was in EVERY OTHER BAND . I saw them together again in The Boyfriend Girlfriend Bluegrass Band at the Alaskan. I saw him play with Raisin’ Holy Hell at The Crystal Saloon. I saw them in some very endearing moments in the documentary. I saw them play the main stage. I saw him out jamming. It’s a good thing they kick ass. There were so many times I was doubled over with laughter on this trip. Maybe that, all things considered, was the best part. I’ve come to think that laughing is my #1 bucket filler. One night at dinner, there was an appetizer called “Bread and Bones” (which turned out to be a bone marrow thing), but we weren’t sure, so we just made silly guesses about what it might be, and I haven’t laughed that hard in a long time. One day, sitting at Amalga (and I have absolutely no memory of how this came up), we opened up the Claude app on my phone and vibe-coded different trivia-style games. It competently crafted an “alive or dead” game with random celebrities, and we kept adding features and making variations. The new bar game is making your own. Justin is seeing someone. It was lovely to meet her. We spent a lot of time all together as a group of three (plus dogs!). She was kind, endearing, funny, and up for anything. I’m glad to have made another friend. I think three can be a magic number. There are more personalities and things going on to play off of. I need to remember this more specifically for friend trips: 3-5 is a good number range. Last year, for the 50th, the weather was shit. It was cold and rainy the entire time. That’s how it always is. I’m sure months of dark, wet weather generally have mental consequences for the Alaskan natives, but it doesn’t seem to affect people’s moods during Folk Fest. There was a bit in the documentary about where they are clear on the matter: it just doesn’t matter . Put on your coat. That was put to the test this year in an interesting way. While there were still big piles of snow everywhere, it was kinda nice out. Twice! Blue skies; warm sun. I was curious whether people would take to the streets, with outside jams, impromptu parties, and such. There was a little. I saw a couple of jams move chairs outside or play on the concrete outside the Sealaska Heritage Museum. It was kinda fun, but it wasn’t like this transformative thing for the festival. It was fun, but again, the weather just doesn’t seem to matter much. One of those nice days I popped into Devil’s Club to find the jam was Irish. Which is fine , but I’m not skilled enough in Irish to contribute much and there is usually enough going on I don’t need to force it. There was another fella sitting there, I noticed, who had a fiddle case, and we got to talking and turned out he played old time like me. So we found a little stoop over by Deckhand Dave’s, he flipped over an old, dirty bucket, and we played old-time duets for a couple of hours. Didn’t even catch his name. I only went to the main stage once this year. The very last night. There’s just so much to do, it’s not even weird to miss most of the main stage stuff. One way to engage with Folk Fest is to hang out at the main stage primarily, and I’m sure a ton of people do that, but the musician types are always seeking out gigs and jams, and the younger crowd (and people that just don’t care that much about folk music) take the opportunity to enjoy all the great human energy downtown. Bar hopping and seeing the many non-folk shows and such. I’m so glad I went to that last night, though. RO Shapiro had a powerful voice, sang beautiful songs alone on stage, and reminded us how important it is to support musicians. He had a wonderful song about how they all pass the same $20 bill around. I was stoked to see Riley Baugus, a banjo hero of mine. He was charming and funny and interesting in a way I definitely did not think he would be, and he managed to keep the huge audience captivated entirely alone with a banjo. He was there with The Red Hots , who I unfortunately missed. Willie Carlisle closed it up, playing with a couple of multi-instrumentalists (one of whom I got to jam a little with, incredibly). Willie is a monster with a huge voice, huge personality, and huge opinions. He’s got a kind of old timey way of speaking and choosing words. He felt like a modern embodiment of folk, blending instruments and styles that are quite different while carrying a consistent air of quality. He opened with a monster vocal-only The Balad of Penny Evans, a Steve Goodman song about Penny who’s husband dies in Vietnam and is none too happy about that. A song called Crittertown brought out a surprise friend in a giant possum costume to wander the audience (gave me big Northern Exposure feels). My favorite was Big Butt Billy, an extra-folky guitar number about a kinda gender-neutral waiter at a diner with an ass so incredible Willie breaks down into exasperated spoken word in the middle of the song, finding different wild-eyed words to praise the ass. One day in the afternoon, I was sitting in The Alaskan having a pint and waiting for Justin to get off work. There was a band setting up I’d never heard of: Big Sissy. Sisters from Connecticut. They played well and harmonized beautifully. I remember a First Aid Kit cover perfectly done. Fifteen minutes after their set was over, we had walked over to Griz Bar, and they all walked in. I got a chance to say hi and thank them for their amazing and unexpected set. It was a warm moment. Another day sitting on a stool at Griz Bar, there was a woman playing guitar really well and singing a Tom Waits cover. Rosemary was sitting, putting in little fiddle fills. They came over to the bar, and I got to buy them a drink, and the world felt warm again for another moment. She then played another Tom Waits cover. Yet another day at Griz, Dude Mountain was playing an acoustic set. It was packed, even in the drizzle. There was a large man dressed up as a kind of cartoon wizard. He didn’t look like he left the house much, honestly, but he was out now, and he brought his cat, which kinda crawled around on his shoulders. Then someone brought like a dozen Domino’s pizzas and passed them out for free. I’d say food isn’t particularly notable in Juneau. I had a steak dinner at SALT one night. The service was good. We laughed our asses off at stupid jokes. The steak was good, but everything else was fairly poor, honestly. They put this huge dollop of horseradish on my plate, camouflaged next to the au gratin potatoes, and I accidentally ate the entire thing. It was a real mouth problem for a minute there. My bad, I guess, but like, isn’t this a plating UX issue? I had a Pickle Rick at The Hanger. The Cubano at Devil’s Club. The Taco Bell replica Crunchwrap Supreme at the Imperial (regrettable but necessary). Pizza at the Island Pub over on Douglas was good, but gave me heartburn that was hard to kick. One night, we had a decent Indian spread at Spice. The vibes are a little sleepy; they didn’t seem to book any musicians this year, and the naan was a bit dry. The Mexican food at Mar y Sol is fine, but they are a dry restaurant, and no margs with Mexican is rough. Amy and crew had dinner there, and I got a text from her that they started a jam there, and honestly, that was really fun. Kinda brought Folk Fest to another area of town that doesn’t normally get it. The noon latte at Coppa was a 10. What you want out of a culinary experience in Juneau is to go out to Sand Bar in the valley and get the fried halibut. It’s literally all they do. The halibut comes from fishermen literally in Juneau. Even as a totally non-fish guy, I love it. I was sad to miss it this year. On my last full day there, I wanted to do some gift shopping. I called it Power Shopping because it was something I wanted to do, but wasn’t super in the mood for it, so the plan was hot’n’fast. I ended up getting: While Folk Fest officially ends on Sunday, and I imagine a lot of folks need to take off on Sunday or Monday, I scheduled my flight out on Tuesday on purpose because Monday is reserved for an all-day jam at The Imperial . The Imperial is right at the heart of downtown Juneau, but doesn’t seem to be an active participant in Folk Fest. Until Monday, when it’s absolutely taken over. All the stragglers show up there and all the musical styles represent. I listened to an alt-old-time jam singing Reeltime Travelers, a classic old-time jam, a country jam, and a monster cajun jam. It took me a while to get the nerve up to get my banjo and get in on it (my confidence ebbs and flows). Honestly, a couple of beers always helps, which I don’t love, but it is what it is. I ended up playing with Heidi again for a while, bookending the trip nicely, and then another group of lovely folks before feeling good about retiring the banjo for the trip. Lodestone library was hosting jams, and I peeked in and saw it, but I didn’t stop to jam, and should have. There is a new brewery in town, Harbor Mountain, that hosted stuff, but I never made it in there, even just to try a beer. I like the group Wool Pullers, who had a couple of shows, and I missed them both. I really wanted to see the band High Costa Living featuring the exuberant powerhouse that is Collette Costa , but the line at the door for that show at The Red Dog Saloon was just insane (hundreds long?) seemingly the entire night. I missed the rad metal band Bards of Mendenhall I missed The Red Hots (I should have gone to the live studio audience show at KTOO). I didn’t go to any dances. I’m dead scared of making a fool of myself at a dance, but I also want to get over it and do it. I didn’t do any workshops. I didn’t catch Caleb & Reeb, who had a LOT of shows. I saw them around a ton but didn’t seem them play, other than Reebs Sea of Heartbreak thing. I’ve still never even met Caleb, who’s a bit of a hero to me. A little intimidating. I missed the Canadian tuxedo party. I missed the cosmic truckstop brunch thing. A book from Sealaska Heritige Store . They had a Trickster basketball that was freakin’ art , but I just couldn’t justify traveling with it Some postcards and a book from Kindred Post A comic book at art supplies from Alaska Robotics (which had an incredible display of paintings of hikes in Juneau) T-Shirts from Treetop Obligatory shirts from Devil’s Club and The Alaskan

0 views
Nelson Figueroa 1 months ago

How to Install a Specific Version of a Homebrew Package with brew extract

I previously wrote about how to install older versions of homebrew packages . That method involves installing a package from a Ruby file but it’s outdated and doesn’t always work. There’s a better way with , although it still comes with caveats. I’ll be using as an example. Let’s say I wanted to install v0.145.0 because v0.146.0 introduced breaking changes that broke my theme. To install hugo v0.145.0: Note that this process will point your command to the older version, but you can switch between versions with . It will enable developer mode. This is normal and safe. Next, run . At the time of writing, it’s a 1.3GB download. This is necessary to get this working because Homebrew no longer keeps homebrew-core cloned locally. The command needs the full git history to search for older versions. Now we can use . This command will find a commit where the formula was at the version we want and copy that locally as . In this case we want Hugo v0.145.0, so we run : This isn’t needed for every formula and is something I ran into specifically with Hugo. Without this patch, you’ll run into errors. After running , edit the file: . Change this line: The reason we need to patch this file is because it prevents the error: It’s a mismatch between the path Homebrew expects ( ) vs the path that is created when using on Hugo ( ). Now that Hugo is extracted and patched, we can install with : Hugo v0.145.0 is now installed. There’s a warning with long output in the previous example due to the normal Hugo package being already installed but that is expected. Homebrew is now pointing the binary to v0.145.0 instead of the latest version (v0.160.1 at the time of writing). We can verify with : We can also see that Hugo v0.145.0 is installed along with the latest version with : Currently the command is pointing to v0.145.0. To have it point back to the regular version, run : And if we want to point back to the old version, run At first I expected to work right off the bat, but running both and is necessary to switch between versions properly. This is because homebrew tracks linked formulas and actual symlinks on disk separately. To help Homebrew track things properly we need to run both to clean the records, then to write the new symlinks. There’s no need to use to prevent the older version of Hugo from updating. Since this is a local copy, there is no remote repository that would be updated that would in turn update our local version. You can even try running to see the warning message: If you no longer need Hugo v0.145.0 you can run : If you don’t have any other packages you extracted with , you can also remove your local tap with Finally, if you don’t plan on using again in the future, you can remove the local clone of homebrew-core with . This will clean up the 1.3GB of files that was downloaded: Then re-link to the latest version with : Create a local tap with Tap homebrew/core which is a 1.3GB clone at the time of writing Extract the formula with Patch the formula. This isn’t needed for every formula. Install as usual https://docs.brew.sh/Manpage https://github.com/orgs/Homebrew/discussions/2941 https://emmer.dev/blog/installing-old-homebrew-formula-versions/

0 views
iDiallo 1 months ago

You paid for it, you should be comfortable in it

A friend of mine bought a Tesla Roadster back in the early 2010s. At the time, spotting a Tesla on the road was a rare event. Maybe even occasion enough to stop and take a picture. I never got the chance to photograph one, let alone drive one, until I met this new friend recently. This was my chance to experience the car firsthand. We walked to the parking structure to see it. As soon as he opened the door, something looked... off. On the outside, it was a pristine, six-figure roadster. But the inside looked completely custom. Not "custom" in the sense of a professional shop install, but more like the driver himself grabbed a hammer and chisel and made it his own. First, the driver's seat had been altered. It was much lower than usual and didn't match the passenger seat. My friend stands 6'7", and the Roadster is a tiny car. He physically couldn't fit, so he modified the seat rails to lower it. But that fix created a new problem: the door armrest now dug into his hip. So, he took a file to the interior panel, shaved it down, and 3D printed a smaller, ergonomic armrest. He even 3D printed a cup holder for the passenger side so his coffee was within reach. To me, the idea of taking a Dremel or a file to a $100,000+ car was unimaginable. You must be crazy to do it. He caught the look on my face and shrugged. "Hey, it's my car. I paid for it. I intend to be comfortable in it." I never thought of it like this. That sentiment stuck with me. Recently when I read an article by Kent Walters about filing the corners of his MacBook , those same feelings resurfaced. My work MacBook has edges so sharp that I've often felt like I was slicing my wrist on the chassis. I treated this as a design flaw I had to endure. But not Kent. He treated it as an obstacle to be removed. He literally filed down the corners of his laptop to ensure the machine he uses every day was comfortable. I may not have the guts to file my work issued MacBook, but I'm no stranger to customization... in software. I modify my tools constantly. I spend days tweaking my IDE, remapping keyboard shortcuts, and writing custom scripts until the software is unrecognizable to anyone else on my team. I don't think twice about rewriting a config file to make the tool fit my brain. When I was a kid, I always had a screw driver around, fixing a device that wasn't really broken. On the home computer, I modified everything. I once deleted all files to improve performance. It didn't work, but it led to a fruitful career. But somehow, when it comes to expensive hardware now, I freeze. I treat the physical object as a museum piece to be preserved. I bought a docking station to banish the laptop to a shelf, using an external mouse and keyboard to avoid touching the sharp chassis. I built a complex workaround to accommodate the tool, rather than performing the simple, brutal act of modifying the tool to accommodate me. We treat our physical tools as if they are on loan from the manufacturer. You'll see a musician buying a vintage guitar but refuses to adjust the action, terrified of ruining the "collector's value." Meanwhile, the working guitarist has sanded down the neck and covered it in stickers because it feels better in their hand. The software engineer accepts the default keybindings to avoid "bad habits," while the power user creates a layout that doubles their speed. If you own a tool, whether it's a car, a computer, or a line of code, you own the right to change it. The manufacturer designed it for the "average" user, but you are a specific human with specific needs. Remember grandma's couch in the living room? It had that plastic cover on it. It was so uncomfortable, but no one dared to remove it. The plastic was to preserve the sofa. No one got to enjoy it, instead everyone accommodated the couch only to preserve its value. A value that one ever benefits from. Don't let the perceived value of an object stop you from making it truly yours. A tool with battle scars is a tool that is loved.

0 views
Corrode 1 months ago

Cloudsmith

Rust adoption can be loud, like when companies such as Microsoft, Meta, and Google announce their use of Rust in high-profile projects. But there are countless smaller teams quietly using Rust to solve real-world problems, sometimes even without noticing. This episode tells one such story. Cian and his team at Cloudsmith have been adopting Rust in their Python monolith not because they wanted to rewrite everything in Rust, but because Rust extensions were simply best-in-class for the specific performance problems they were trying to solve in their Django application. As they had these initial successes, they gained more confidence in Rust and started using it in more and more areas of their codebase. CodeCrafters helps you become proficient in Rust by building real-world, production-grade projects. Learn hands-on by creating your own shell, HTTP server, Redis, Kafka, Git, SQLite, or DNS service from scratch. Start for free today and enjoy 40% off any paid plan by using this link . Made with love in Belfast and trusted around the world. Cloudsmith is the fully-managed solution for controlling, securing, and distributing software artifacts. They analyze every package, container, and ML model in an organization’s supply chain, allow blocking bad packages before they reach developers, and build an ironclad chain of custody. Cian is a Service Reliability Engineer located in Dublin, Ireland. He has been working with Rust for 10 years and has a history of helping companies build reliable and efficient software. He has a BA in Computer Programming from Dublin City University. Lee Skillen’s blog - The blog of Lee Skillen, Cloudsmith’s co-founder and CTO Django - Python on Rails Django Mixins - Great for scaling up, not great for long-term maintenance SBOM - Software Bill of Materials Microservice vs Monolith - Martin Fowler’s canonical explanation Jaeger - “Debugger” for microservices PyO3 - Rust-to-Python and Python-to-Rust FFI crate orjson - Pretty fast JSON handling in Python using Rust drf-orjson-renderer - Simple orjson wrapper for Django REST Framework Rust in Python cryptography - Parsing complex data formats is just safer in Rust! jsonschema-py - jsonschema in Python with Rust, mentioned in the PyO3 docs WSGI - Python’s standard for HTTP server interfaces uWSGI - A application server providing a WSGI interface rustimport - Simply import Rust files as modules in Python, great for prototyping granian - WSGI application server written in Rust with tokio and hyper hyper - HTTP parsing and serialization library for Rust HAProxy - Feature rich reverse proxy with good request queue support nginx - Very common reverse proxy with very nice and readable config locust - Fantastic load-test tool with configuration in Python goose - Locust, but in Rust Podman - Daemonless container engine Docker - Container platform buildx - Docker CLI plugin for extended build capabilities with BuildKit OrbStack - Faster Docker for Desktop alternative Rust in Production: curl with Daniel Stenberg - Talking about hyper’s strictness being at odds with curl’s permissive design axum - Ergonomic and modular web framework for Rust rocket - Web framework for Rust Cloudsmith Website Cian Butler’s Website Cian’s E-Mail

0 views
André Arko 1 months ago

Towards an Amicable Resolution with Ruby Central

Last week, three members of Ruby Central’s board published a new statement about RubyGems and Bundler , and this week they published an incident report on the events last year . The first statement reports that Ruby Central has now completed a third audit of RubyGems.org’s infrastructure: first by the sole remaining RubyGems.org maintainer , the second by Cloud Security Partners , and the third by Hogan Lovells. In all three cases, Ruby Central found no evidence of compromised end user data, accounts, gems, or infrastructure availability . I hope this can conclusively put to rest the idea that I have any remaining access to the RubyGems.org production systems, or that I caused any harm to the RubyGems.org service at any time. I also appreciate that Ruby Central is taking its share of responsibility, recognizing that its lack of communication with the former maintainers (including me) created confusion and frustration that contributed, in part, to how we ended up where we are today. Ruby Central board members Freedom, Brandon, and Ran state that their intent is now to work towards an amicable resolution. I salute their new commitment, and would like to do my part to help the RubyGems community move past these unfortunate events, with a resolution that puts the dispute fully behind us, and allows all of us to move forward. For my part, despite my claims against Ruby Central, and the threats they have directed against me, I am willing to completely settle all of my disputes with them, and pledge to take no legal action against Ruby Central regarding any of their actions prior to today. In exchange, I am requesting two things. First, I am asking Ruby Central to drop their legal threats, including releasing their claims against me and reimbursing my legal costs. Those costs arise from Ruby Central’s actions, including litigation threats, other escalations, and most recently contacting law enforcement. In addition to forcing me to retain counsel, these actions caused considerable stress and disruption. I am willing to provide invoices to ensure the reimbursement precisely matches only my actual costs. Second, I am asking Ruby Central lay our disagreement to rest with a public statement acknowledging that I did no harm to the RubyGems.org service. If Ruby Central fully drops their legal claims, and states I did not harm the RubyGems.org service, I would consider our disagreement amicably settled.

0 views
Chris Coyier 2 months ago

Hawai’i

I’m just back from the United States 50th state, a staggering 2,500 miles from the mainland. For the next week or two, I’ll pronounce it Ha-Vie-ee, like how it’s pronounced in the native Hawaiian language. A language, by the way, that only a few thousand people speak natively, no doubt due to the 91 years (1896-1987) where there was “strict physical punishment” for speaking it in schools. We humans are pretty damn uncool to each other sometimes. Ruby and I travelled there ( again! ) with some wonderful family friends, Matt, Becky, and their kids, Monroe and Zoey. A nice reminder of how rare and lovely it is to have a situation where the kids are friends, and the adults are friends, and everyone travels together well. We stayed in a villa at the Fairmont Kea Lani on Maui. I’ve been to Hawaii before, but this was my first time on Maui. It was a beautiful place to stay. A beautiful property and buildings right on the beach. The villa had two spacious rooms, a full kitchen, and a living room with a pull-out couch, on which all the kids slept together. I’ve stayed at fancy resorts before where the staff uses special greetings with guests. But in Hawaii, naturally, it’s “Aloha.” Probably because, ya know, a real word, and basically the whole brand of Hawaii. But I just can’t shake the feeling that it’s kinda cheesy. Like, do Hawaii long-timers say Aloha to each other? Like it’s 5:21 am and a local is getting a coffee at the gas station in a local neighborhood, do they say Aloha to the cashier? Do they get an Aloha back? I kept meaning to ask this of locals, but kept forgetting. Or not having the exact 1.5 beers in me it takes to reach that perfect level of fun and charm to ask strangers semi-intimate questions. If I were forced to guess, I’d guess Aloha is more of a thing they have to do at work with the tourists. Like your boss side-eyes you if you just say “Hello, good morning” instead. I never said it back, which felt weird. My goal was kind of a winkwink, it’s cool , you don’t have to do the cheesy tourist thing with me, I very promise I don’t care. The first night, we got checked in and b-lined it to Monkeypod . We’d all been there before (at a different location) and have talked about it endlessly. It’s a micro-chain with 4 locations across two islands. It’s just: great. They make a Mai Tai with Honey Liliko‘i Foam on the top which I have fond memories, and it was every bit as good as I remembered. I had wings and mahi-mahi tacos. 10/10. I never get the fish. I don’t like fish. I like specific little bits of seafood once in a while, but rarely cooked slabs of fish. So on that very first night, I decided I’d get fish every night on this trip. Maybe if I try enough of it, I’ll come around. It didn’t work. I struck out more times than I hit. But no big regrets. I tried. Timing-wise, it wasn’t the absolute perfect time to be in Hawaii. But it was spring break for our school district, so C’est la Vie. Unprecedented rain with some flooding. A rather ironic situation after the horrible fires just a few years back. We were watching the weather and reading the news weeks in advance, but things didn’t seem dire enough to cancel the trip. Honestly, some overcast weather isn’t the worst. None of us left with sunburns. It allows you to hang out outside all day, which you just can’t do on full-sun days, as it exhausts you. The first full day turned out to be one of the rainiest, and we spent most of it at the pool anyway. I got us a cabana that turned out to be awfully useful. Being in the pool in the rain is no big deal, but lying out in chairs in the rain is annoying. And you certainly can’t crack open the laptop or read a paperback. I did both that day and was loving it. We were trying to book an ATV tour for ourselves, and that was the one thing we just couldn’t get done. The rainstorms just weren’t letting it happen. Apparently, there was too much debris and whatnot on the trails; the places that offered these tours didn’t reopen until after we left. We started most mornings at the breakfast buffet, included in our fancy villa booking. It was pretty crowded as they couldn’t seat people outside in the wet. Then we’d hit the water without fail. A few days we did the ocean, but came to understand it really wasn’t a good time for that. Storms wash landcrap out to sea, making the water look muddy. Apparently, that’s worse than just looking ugly; it can harbor dangerous bacteria. The guy at 808 clothing told me that you’d have to be a real idiot to go out in it and that real Hawaiians would never. Last year, some lady had to have her legs sliced open to flush out the bacteria (or something? The guy was pretty weird). Also, later, our zip-line guide told us she loves to surf and wouldn’t go out because the “muddy” water is extra-attractive to sharks, since the low visibility helps them more than it helps their prey. Also later, we went to a surfing beach absolutely full of obviously local surfers. Turns out people don’t exactly speak for all people. We did some knee-deep ocean stuff because it’s hard to resist. One day we drove up to Paia, a northern coastal city with extreme charm. Unfortunately we got there when it was pouring pretty good, so we spent most of it hustling between store overhangs. You could really see how close to flooding everything can get, quickly. We mostly just did a little shopping, walking around, and snacking in Paia, and I didn’t take many photos there. It was super cute though, highly recommended. I sorta regret not buying a Ukulele bass from the music shop there as I’ve been eyeing one up like forever, ever since going on a trip with Brad Frost where he brought his. Which reminds me: we had the kids to Uke lessons at the Fairmont and it was kind of a mess. Probably skip that. The hostess at the bar we stopped at told us how to get down to the turtle beach nearby (Ho’okipa). It was really pouring when we got there, so we just parked for a while and watched the surfers. Really amazing to watch. Huge waves. The turtle beach didn’t disappoint! Hitting the pool was a daily event. The kids are old enough that we could shoo them out the door to the pool and not worry about it too much. Two of the kids had trackable wrist watches that could make calls, so that was extra convenient. There was a swim-up bar that I appreciated existing despite never getting around to using it. I did us the walk-up bar once, and the Zach Bryan impersonator bartended made me a cocktail despite it being almost an hour after it was supposed to close. He was being fawned over by two woman who wanted to make sure he had their number for later. I enjoyed that, naturally. Ruby’s favorite experience, and perhaps mine too, was the zip lining we did. We chose Haleakalā Zipline Tours as, well, it was open, and it’s location high up mid-island looked cool. It was. The two charming guides helped make it fun, showering us with bird-facts and about their conservation efforts. Ruby had to get over some fears of zip lining at all, which she did and of course now loves it. I left thinking of other zip lining we could to back home and hoping to see a ʻAlalā (Hawaiian crow). We hit Black Rock Pizza on the way home, my only non-fish dinner. The very last day, our friends moved on to another island, while we were hitting the redeye flight home. We had most of the day to kill, so we wandered around the property a bit, wandered some stores, then went to the local cinema to catch Project Hail Mary (fun!) and then off to the airport. Only a 5-hour flight back to Seattle compared to the 7-hour flight from Salt Lake City on the way there. We both slept a little and it went easy breezy.

0 views
Max Bernstein 2 months ago

Using Perfetto in ZJIT

Originally published on Rails At Scale . Look! A trace of slow events in a benchmark! Hover over the image to see it get bigger. Now read on to see what the slow events are and how we got this pretty picture. The first rule of just-in-time compilers is: you stay in JIT code. The second rule of JIT is: you STAY in JIT code! When control leaves the compiled code to run in the interpreter—what the ZJIT team calls either a “side-exit” or a “deopt”, depending on who you talk to—things slow down. In a well-tuned system, this should happen pretty rarely. Right now, because we’re still bringing up the compiler and runtime system, it happens more than we would like. We’re reducing the number of exits over time. We can track our side-exit reduction progress with , which, on process exit, prints out a tidy summary of the counters for all of the bad stuff we track. It’s got side-exits. It’s got calls to C code. It’s got calls to slow-path runtime helpers. It’s got everything. Here is a chopped-up sample of stats output for the Lobsters benchmark, which is a large Rails app: (I’ve cut out significant chunks of the stats output and replaced them with because it’s overwhelming the first time you see it.) The first thing you might note is that the thing I just described as terrible for performance is happening over twelve million times . The second thing you might notice is that despite this, we’re staying in JIT code seemingly a high percentage of the time. Or are we? Is 80% high? Is a 4.5% class guard miss ratio high? What about 11% for shapes? It’s hard to say. The counters are great because they’re quick and they’re reasonably stable proxies for performance. There’s no substitute for painstaking measurements on a quiet machine but if the counter for Bad Slow Thing goes down (and others do not go up), we’re probably doing a good job. But they’re not great for building intuition. For intuition, we want more tangible feeling numbers. We want to see things. The third thing is that you might ask yourself “self, where are these exits coming from?” Unfortunately, counters cannot tell you that. For that, we want stack traces. This lets us know where in the guest (Ruby) code triggers an exit. Ideally also we would want some notion of time: we would want to know not just where these events happen but also when. Are the exits happening early, at application boot? At warmup? Even during what should be steady state application time? Hard to say. So we need more tools. Thankfully, Perfetto exists. Perfetto is a system for visualizing and analyzing traces and profiles that your application generates. It has both a web UI and a command-line UI. We can emit traces for Perfetto and visualize them there. Take a look at this sample ZJIT Perfetto trace generated by running Ruby with 1 . What do you see? I see a couple arrows on the left. Arrows indicate “instant” point-in-time events. Then I see a mess of purple to the right of that until the end of the trace. Hover over an arrow. Find out that each arrow is a side-exit. Scream silently. But it’s a friendly arrow. It tells you what the side-exit reason is. If you click it, it even tells you the stack trace in the pop-up panel on the bottom. If we click a couple of them, maybe we can learn more. We can also zoom by mousing over the track, holding Ctrl, and scrolling. That will get us look closer. But there are so many… Fortunately, Perfetto also provides a SQL interface to the traces. We can write a query to aggregate all of the side exit events from the table and line them up with the topmost method from the backtrace arguments in the table: This pulls up a query box at the bottom showing us that there are a couple big hotspots: It even has a helpful option to export the results Markdown table so I can paste (an edited version) into this blog post: Looks like we should figure out why we’re having shape misses so much and that will clear up a lot of exits. (Hint: it’s because once we make our first guess about what we think the object shape will be, we don’t re-assess… yet .) This has been a taste of Perfetto. There’s probably a lot more to explore. Please join the ZJIT Zulip and let us know if you have any cool tracing or exploring tricks. Now I’ll explain how you too can use Perfetto from your system. Adding support to ZJIT was pretty straightforward. The first thing is that you’ll need some way to get trace data out of your system. We write to a file with a well-known location ( ), but you could do any number of things. Perhaps you can stream events over a socket to another process, or to a server that aggregates them, or store them internally and expose a webserver that serves them over the internet, or… anything, really. Once you have that, you need a couple lines of code to emit the data. Perfetto accepts a number of formats. For example, in his excellent blog post , Tristan Hume opens with such a simple snippet of code for logging Chromium Trace JSON-formatted events (lightly modified by me): This snippet is great. It shows, end-to-end, writing a stream of one event. It is a complete (X) event, as opposed to either: It was enough to get me started. Since it’s JSON, and we have a lot of side exits, the trace quickly ballooned to 8GB large for a several second benchmark. Not great. Now, part of this is our fault—we should side exit less—and part of it is just the verbosity of JSON. Thankfully, Perfetto ingests more compact binary formats, such as the Fuchsia trace format . In addition to being more compact, FXT even supports string interning. After modifying the tracer to emit FXT, we ended with closer to 100MB for the same benchmark. We can reduce further by sampling —not writing every exit to the trace, but instead every K exits (for some (probably prime) K). This is why we provide the option. Check out the trace writer implementation from the point this article was written. We could trace: Visualizations are awesome. Get your data in the right format so you can ask the right questions easily. Thanks for Perfetto! Also, looks like visualizations are now available in Perfetto canary. Time to go make some fun histograms… This is also sampled/strobed, so not every exit is in there. This is just 1/K of them for some K that I don’t remember.  ↩ two discrete timestamped begin (B) and end (E) events that book-end something, or an instant (i) event that has no duration, or a couple other event types in the Chromium Trace Event Format doc When methods get compiled How big the generated code is How long each compile phase takes When (and where) invalidation events happen When (and where) allocations happen from JITed code Garbage collection events This is also sampled/strobed, so not every exit is in there. This is just 1/K of them for some K that I don’t remember.  ↩

0 views
André Arko 2 months ago

How to Install a Gem

This post was originally given as a talk at SF Ruby Meetup . The slides are also available. Hello, and welcome to How To Install A Gem . My name is André Arko, and I go by @indirect on all the internet services. You might know me from being 1/3 of the team that shipped Bundler 1.0, or perhaps the 10+ years I spent trying to keep RubyGems.org up and running for everyone to use. More recently, I’ve been working on new projects: , a CLI to install Ruby versions and gems at unprecedented speeds, and gem.coop , a community gem server designed from the ground up so Bundler an can install gems faster and more securely than ever before. So, with that introduction out of the way, let’s get started: do you know how to install a gem? Okay, that’s great! You can come up and give this talk instead of me. I’ll just sit over here while you write the rest of this post. Slightly more seriously, do you know how converts the name that you give it into a URL to download a .gem file? It’s called the “compact index”, and we’ll see how it works very soon. Next, who in the audience knows how to unpack a .gem file? Do you know what format .gem files use, and what’s inside them? We’ll look at gem structure and gemspec files as well. Then, do you know where to put the files from inside the gem? Where do all of these files and directories get put on disk so we can use them later? Does anyone know off the top of their head? Once those files have been unpacked into the correct places, the last thing we need to know is how to require them. How do these unpacked files on disk get found by Ruby, so you can and have that actually work? This exercise was mostly to show that using gems every day actually skips over most of the way they work underneath. So let’s look at what a gem is, and examine how they work. By the end of this talk, you’ll know what’s inside a gem, where how RubyGems figures out what to download, and where and how that download gets installed so you can use it. And if you already everything we just talked about, please feel free to go straight to rv.dev and start sending us pull requests! First, we’re going to look at how the name of a gem becomes a URL for a .gem file. Let’s use as our example. Historically, there have been at least five or six different ways to look up information about a gem based on its name, but today there is one canonical way: the compact index. It’s so simple that you can do it yourself using curl. Just run , and you’ll be able to read the exact output that every tool uses to look up the versions of a gem that exist. Each line in the file describes one version of the gem, so let’s look at one line. We can break down that line with , and tackle each part one at a time. First, . That’s the version of that this line is about. So we now know for sure that exists. Next, a list of dependencies. The gem (version ) declares dependencies on a bunch of other gems: , , , , , , , , , , , , and . Each dependency has a version requirement attached, and for almost every gem it is exactly version , and only version . For , Rails is a little bit more flexible, and allows any version and up. The final section contains a checksum, a ruby requirement, and a rubygems requirement. The checksum is a sha256 hash of the .gem file that contains the gem, so after we download the gem we can check to make sure we have the right file by comparing that checksum. For this version of Rails, the required Ruby version is or greater, and the required RubyGems version is or greater. It’s up to the client to do something with that information, but hopefully you’ll see an error if you are using Ruby or RubyGems that’s too old. Great! So now we know the important information: Rails version is real, and strong, and is our friend. We can download it, and check the checksum against the checksum we were given in the info file line. Let’s do that now: Notice that the checksum produced by exactly matches the checksum we previously saw in our line from the info file: . That lets us know that we got the right file, and there were no network or disk errors. Now that we have the gem, we can investigate: what exactly is inside a gem? At this point, we’re going to pivot from the gem to the gem. There’s a good reason for that, and the reason is… the gem doesn’t actually have any files in it. So it’s a bad example. In order to show off what a gem looks like when it has files in it, we’ll use instead. So, we have our .gem file downloaded with curl. What do we do now? The first piece of secret knowledge that we need: gems are tarballs. That means we can open them up with regular old . Let’s try it. So what’s inside the .gem tarball is… another tarball. And also two gzipped files. Let’s look at the files first. As you might expect from its name, the file is a gzipped YAML file, containing checksums for the other two files. It’s maybe a bit silly to have multiple layers of checksumming here, but it does confirm that the outer layer of tarball and zip was removed without any errors. Okay, so what’s inside ? The answer is… Ruby, sort of. It’s a YAML-serialized instance of the class. We can see exactly what was put into this object at the time the gem was built. After snipping out the YAML that lists the dependencies (which we already looked at, because they are included in the info file), what’s left is some relatively simple information about the gem. Author, author’s email, description, homepage, license, various URLs. For the purposes of installing and using the gem, we care about exactly six pieces of information: , , , , , and . We’re going to combine those items with the files in the remaining file to get our unpacked and installed gem. Now that we know what’s in the gem specification, let’s look at what’s inside the data tarball. It matches up very closely with the long list of entries in the array in the gemspec. So now we have a bunch of files. Where are we going to put these files? Enter: the magic of RubyGems. The scheme that RubyGems has come up with is largely shaped by the constraints of how Ruby finds files to require, which we’re going to look at soon. For now, it is enough for us to know that RubyGems keeps track of a list of directories, a lot like the way works for your shell to find commands to run. To find the current directory, you can run . Here’s what that looks like: From this list, we can see that RubyGems organizes its own files into a few directories. To install a gem, we’re going to need to put the files we have into each of those directories, with specific paths and filenames. Just to recap, the files we need to place somewhere are: So let’s move the files into the directories we see RubyGems offers. First, cache the .gem file so RubyGems doesn’t need to download it again later: Then, add the gem specification so that RubyGems will be able to find it. There’s a small twist here, which is that the directory doesn’t contain YAML files, it contains Ruby files. So we also need to convert the YAML file back into a Ruby object, and then write out the Ruby code to create that object into a file that RubyGems can load later. Next, we need to put the files that make up the contents of the gem into the directory. One more thing we need to do: set up the executables provided by the gem. You can check out the files that RubyGems generates by looking in , but for our purposes we just need to tell RubyGems what gem and executable it needs to run, so we can do that: And with that, we’ve installed the gem! You can run the file that we just created to prove it: As we wrap up here, there are three aspects of gems that we haven’t touched on at all: docs, extensions, and plugins. We don’t have time to talk about them today in this meetup talk slot. Hopefully a future (longer) version of this talk will have space to include all of those things, because they are all super interesting, I promise. In the meantime, I will have to direct you to the docs for RDoc to learn more about docs, to the source code of or RubyGems itself if you want to learn more about gem extensions and plugins. There’s one last thing to figure out before we wrap up: how does find a gem for us to be able to use it? To explain that, we’ll have to drop down to some basic Ruby, and then look at the ways that RubyGems monkeypatches Ruby’s basic to make it possible to have gems with versions. The first thing to know about is that it works exactly like does in your shell. There’s a global Ruby variable named , and it’s an array of paths on disk. When you try to require something, Ruby goes and looks inside each of those paths to see if the thing you asked for is there. You can test this out for yourself in just a few seconds! Let’s try it. The Ruby CLI flag lets you add directories to the variable, and then the function looks inside that directory to find a file with the name that you gave to require. No magic, just a list to check against for files on disk. Now that you understand how the variable makes work, how does RubyGems work? You can’t just put ten different versions of into the and expect to still work. RubyGems handles multiple versions of the same file by monkeypatching . Let’s look at what happens when we , which is a file located inside the gem that we just installed. RubyGems starts by looking at all of the gem specifications, including the one we saved earlier. In each specification, it combines the name and version with the values in to come up with a path on disk. So for our just-installed gem, that would mean a path of: . RubyGems knows that directory contains a file named , so it is a candidate to be “activated”, which is what RubyGems calls it when a gem is added to your . As long as internal bookkeeping shows that no other versions of have already been added to the , we’re good! RubyGems adds this specific directory to the , and delegates to the original implementation of . Require finds the file at , reads it, and evaluates it. With that, we’ve done it! We have found, downloaded, unpacked, and installed a gem so that Ruby is able to run a command and load ruby files, without ever touching the command. If you’re interested in contributing to an open source project that works a lot with gems, we would love to work with you on , where we are working to create the fastest Ruby and gem manager in the world. And of course, if your company could use faster, easier, or more secure gems for developers, for CI, and for production deployments, we can help. We’d love to talk to you and you can find our contact information at spinel.coop . railties-8.1.3.gem (the .gem file itself) metadata.gz (the YAML Gem::Specification object from inside the gem) the unpacked data.tar.gz files (the contents of the gem)

0 views
Giles's blog 2 months ago

Writing an LLM from scratch, part 32e -- Interventions: the learning rate

I'm still working on improving the test loss for a from-scratch GPT-2 small base model, trained on code based on Sebastian Raschka 's book " Build a Large Language Model (from Scratch) ". In my training code, I have this code to create the optimiser: The values in there -- for the learning rate, and for the weight decay -- were just copied from the tiny training run that we do in section 5.2 of the book. What do those values actually mean, and are those really the right values for them? I felt I had a good handle on the learning rate, at least -- it's one of the first things you learn when you start looking at machine learning of any kind -- but how would you go about working out what the correct value for it was? On top of that, when I was reading the Chinchilla paper a while back, I noticed they repeatedly referred to a "cosine cycle" for the learning rate, which didn't fit into anything I'd learned about before. The weight decay was pretty much an unknown for me -- I know it is a parameter controlling the behaviour of the optimiser, but I don't know how it does that. In this post I want to look into the learning rate, and these mysterious cosines; I'll write a follow-up about the weight decay later. If you're reading this blog, you almost certainly know what the learning rate is, but let's go over it briefly to build a solid foundation. The way it's normally explained, using simple gradient descent, goes something like this. Let's assume that we're training a model with just one parameter, and it starts off set to − 5 . We run some training data through, and get a loss, let's say 44.44: We don't know what shape our loss curve is (if we did, we might be able to find the lowest loss algebraically), but we do know the differential of the parameter versus the loss at the point we've measured; it happens to be -13. That is reasonably large and negative: We use that information to say that we want to move in the direction of a larger value for our parameter -- that is, in our case where the gradient is negative, so we have a downhill slope towards the right, we want to increase the parameter to move rightwards on that chart, whereas if it were positive (an uphill slope) we'd want to decrease the parameter to move leftwards. Simply subtracting the gradient from the parameter would lead to an update in the right direction, but it would be a very large one in this case -- we'd move 13 units to the right -- so we multiply the gradient by a small positive number, the learning rate (often written as a lower-case eta, like this: η ), to move a small distance in that direction. Let's say η = 0.3 . That means we want to update our parameter: So now we run that through and get a new loss -- let's say it's 9.06 -- and a new gradient, which happens to be -5.2. Now we can do another update, and our parameter will become 0.46, so we use that and work out another loss and gradient, which come to 3.3816 and -2.08. Let's plot that one, but this time we'll draw back the veil and show the actual loss curve. Now, it's worth reiterating that while we're training this model we don't know what that curve looks like -- we're just finding points on it, along with its gradient at those points, and using that information to work out which parameter value to explore next. But it's pretty clear that as we continue, if the learning rate is set correctly, we'll get to the minimum eventually if the learning rate is the right kind of size, because -- due to the nice smooth U-shape of the curve, the gradient gets smaller the closer we get to the minimum 1 . It's also pretty clear that if the learning rate is smaller than an optimal value, in this simple case we will still find the right point, but it will take more steps because each one is smaller: And, of course, if the learning rate is too high, we might never converge -- we'd "bounce out of" the dip, and wind up with a parameter value that endlessly cycles between increasingly smaller and increasingly larger values, zooming off to infinity: OK, that's the basics. Why might we want to change from something that seems so logical and simple? A few paragraphs back I said: due to the nice smooth U-shape of the curve, the gradient gets smaller the closer we get to the minimum What if it doesn't? Imagine if we had something more like a V-shaped curve, like this: The gradient does not decrease as we get closer to the minimum, and so while we're in the downward-sloping part, each update is exactly the same distance: Now, eventually we'll jump over the minimum: In this example, I've used a gradient of − 8.33 on the downward-sloping part of the curve, and + 8.33 on the upward-sloping part, so that means that our next update just bounces us back to where we were before! Because the gradient isn't decreasing the closer we get to the minimum, we wind up just oscillating around it. That's not very helpful. That's a slightly contrived example (though not entirely -- intuitively, with functions like ReLU or GELU in our real LLMs, it's easy to imagine crazy loss landscapes). But it does show that perhaps we might want to add in our own "artificial" way to decrease the size of the steps we take over the course of training our model rather than just relying on the gradients naturally flattening out for us. Another way of looking at things is that as the model gets trained, we don't want batches of very new-looking data to cause big updates, taking us away from what was a good part of the loss landscape in terms of what we've seen so far. For example, imagine you've been training an LLM on a bunch of documents, which have so far been in English. Halfway through, it encounters a document in Byzantine Greek, the loss skyrockets, and you do a big update. That would be a problem! You might want it to learn a bit from it to push it slightly in a "the world is multi-lingual" direction, but you don't want it to lose a big chunk of the value from its previous training. You might also see a kind of connection to the way that people learn over the course of their lives -- for babies, everything is new and they "update their parameters" constantly as they try to understand the world. Children are still pretty flexible, but as we get older we tend to update our beliefs less and less. That's not always optimal, but as a heuristic it's pretty adaptive. Anyway, in general: for most training runs, we're going to want the learning rate to adjust over time. Most of the time this will be by reducing it, though there can be cases for increasing it again for periods. The general case of doing this is called "learning rate scheduling". There are a bunch of ways that people adjust the learning rate over the course of a train; here are a few that cropped up a lot while I was researching this. If we want the learning rate to go down over time, and we know how many steps we're training for, we can just set it to (say) 0.0004 for the first quarter of our train, then 0.0002 for the next, then 0.0001, then finish off with 0.00005, like this: That can work pretty well! But there is one obvious oddity -- the big step changes in learning rate mean that the exact placement of the drops and the training data before and after can matter. Why are we treating the data and the state of the model immediately before and immediately after so differently? It would make more sense to have a smoother schedule. What functions decay smoothly like that? An exponential curve does: let's say we just multiply the learning rate by a number that is a little smaller than one every step, so that it drops smoothly like this: But there are lots of other curves like that, and one is particularly interesting: As you change θ from 0 to π , the value of cos θ goes smoothly from 1 to − 1 , so it's easy enough to rescale that so that our learning rate follows the same curve: This is called a "cosine annealing" or "cosine decay" schedule, and was apparently inspired by the algorithms used for simulated annealing (an optimisation algorithm that was in turn inspired by how the atomic structures form in metals as they cool -- another one for the list of things to look into in the future...) That solves the mystery from earlier: the cosine that the Chinchilla paper was talking about was exactly this. As it turns out, the cosine decay scheduling curve is quite popular in deep learning, because it has what amounts to two well-defined phases -- an initial high learning rate where lots of exploration of the loss landscape can happen, followed by a smooth transition to something more like fine-tuning to optimise the location in whatever part of the loss landscape we've wound up in. Now, all of the above are assuming that we want the learning rate to start high and finish low, so that we can mimic the textbook gradient descent that we had at the start of this post. Intuitively that feels nice, but on further thought, the important thing is really that we have a low learning rate at the end of the train, so that we can find as close a point as possible for the minimum at the part of the loss landscape we've found ourselves in. But perhaps there's a case for having both high and low periods during the train, so that we don't get stuck in a local minimum -- something to jolt us out of where we were every now and then? 2 With a step function, that's easy: you could, for example, do this: With an exponential, you could do something like this: With cosine decay, of course, things are even easier, because the cosine function is inherently cyclical, so we can just do this: However, at least for our purposes, training an LLM using a Chinchilla-optimal number of training tokens, it makes sense to be guided by what the authors of the Chinchilla paper did. Appendix B says: We find that setting the cosine cycle length too much longer than the target number of training steps results in sub-optimally trained models, as shown in Figure A1. As a result, we assume that an optimally trained model will have the cosine cycle length correctly calibrated to the maximum number of steps, given the FLOP budget; we follow this rule in our main analysis. So, at this point, I think we have one important part of the intervention we want to make: we want to use a cosine learning rate scheduler, going from high near the start of the training run, down to low at the end over one cycle. Additionally, and also from appendix B in the paper: we use a 10x learning rate decay in line with Rae et al. (2021) ...which means that if our learning rate starts at η , then we want it to decay down to η / 10 by the end. So, we just need to work out an initial value for η , and let it rip, right? Well, not so fast... When our model is uninitialised, right at the start of the train, gradients are going to be pretty wild. It's going to be making random errors all of the time, and we'll be making huge jumps across the loss landscape. That sounds bad. Additionally those kind of wild jumps can get the optimiser into a -- well, sub-optimal -- state. I haven't read enough about optimisers yet to have a solid handle on that, but that can wait -- intuitively it makes some kind of sense that erratic gradient updates might confuse it. So, it makes a certain amount of sense to start off with a low learning rate so that we don't do that, and then to increase it gradually to the peak, and only then to schedule the gradual cosine decay. According to this (rather nice looking) masterclass on LLM training , it's typical to do this over "a few thousand steps or a small percentage (e.g., 1-10%) of the total training steps, depending on the dataset size and batch size", and we would just use a linear increase over that period: I think we should do that; a simple linear warmup at the start -- let's relatively arbitrarily say 5% of our training steps going up to our desired peak learning rate. So our learning rate schedule should look something like this: So far I've written a lot about how we vary the learning rate over time, and that's all been very useful. But we still need to know what the value should be initially! In smaller-scale experiments you might just try a bunch of different numbers to see what worked well, but at more than US$30 per train, that's not practical here. Unfortunately it's really quite hard to find good suggestions published anywhere. The GPT-2 paper is (as usual) reticent: The learning rate of each model was manually tuned for the best perplexity on a 5% held-out sample of WebText ...and if you search for "learning rate training llm", you'll see lots of results for when people are fine-tuning existing LLMs ( 2 × 10 − 4 comes up a lot), but almost nothing about when you're training one from scratch. I eventually came across this (long!) post from Hugging Face , which I definitely need to spend time going through in the future, because it covers a lot of the ground I've been going over in this post series. But for this post, I think the most relevant part is in the section " Scaling Laws for Hyperparameters ", where they include a figure from this DeepSeek paper . Here it is, with some of the (also relevant) surrounding text: In our trains we're using something like 5 × 10 18 total FLOPs. Now, they are specifically charting things in terms of non-embedding FLOPs, but I'm going to play a little fast and loose here and ignore that, so reading off their chart, that looks like we should be using about 1.4 × 10 − 3 as our learning rate. We can double-check that against their formula, where C is the compute budget: Nice, a close match! However, it's definitely worth noting that we're using a simple GPT-2 architecture, and they are using something quite different -- RMSNorm instead of LayerNorm, SwiGLU as the activation function on the feed-forward networks, Rotary Position Embedding rather than the fixed ones we're using, and so on. As a sanity check: you can see that they also give a formula for the optimal batch size in terms of tokens. For our FLOP budget, that comes in at 381,782, which is about 373 of our 1,024-token sequences. That is quite a lot higher than the 97-or-so sequences that we appeared to be optimal in our earlier experiments . That is a little concerning, though of course the 97 number came out of a very ad-hoc bit of curve-fitting. For now, I'm going to hope that that doesn't matter too much for the learning rate. This may come back to bite me; if the results of a train with 1.4 × 10 − 3 are radically worse than the existing rate of 4 × 10 − 4 , I'll have to do a bit more investigation. So, now I think we have all of the theoretical pieces in place to do a train. Let's move on to the practicalities. We started by looking at this: What should we change -- disregarding the until the next post? Based on the above, we want to do a linear warmup of about 5% of our steps, going up to a learning rate of 1.4 × 10 − 3 , followed by a cosine decay down to one tenth of that, 1.4 × 10 − 4 . What does that look like in code? The relevant API for scheduling the learning rate in PyTorch is, logically enough, in the module, and there are a bunch of different scheduling classes. You create your optimiser, then create a scheduler for the shape you want, and then you can call on the scheduler (after the on the optimiser) to adjust the optimiser's learning rate over time. Let's make that more concrete; one of the schedulers is , which is what we'll need for our linear warmup period. It takes as its parameters: Let's say that we want to go from almost-zero to our optimiser's learning rate over 1,600 steps -- we'd create our scheduler like this: ...then in our training loop, after we've done the scaled step of the optimiser, we'd also step the scheduler: This confused me a little bit the first time I saw it; after all, if the scheduler hasn't been "triggered" when we step the optimiser, how does the optimiser know what learning rate to use? Surely it would just use whatever it was initialised with? The answer is that when you create the optimiser, it stores away the learning rate that you give it in two places -- an "initial learning rate" and a "current learning rate". Next, when you create your scheduler, it uses the initial learning rate to work out the start and end values, and then sets the current one to the start value immediately. Just by creating a scheduler, you're changing the optimiser's current learning rate -- but not the initial one, which is important, as we'll see in a moment. So, we have a scheduler that handles our warmup period nicely. Another scheduler that's relevant to our interests is the CosineAnnealingLR . This takes: On creation, this scheduler will read in the optimiser's initial learning rate -- note, not the current one -- and then the first time it's stepped, it will set the current learning rate to that value, and then for steps after that it will reduce it so that it follows a nice cosine decay, reaching after steps. So those two cover the two regimes that we want -- the warmup and then the cosine decay. But now we need to put them together; we want to do one and then the other. There's a very useful class, , which allows you to chain schedulers and tell it when each one takes over from the previous one. Let's sketch out some code to use that to do a train with our new peak learning rate of 1.4 × 10 − 3 , a warmup of 1,600 steps, followed by a cosine decay for the next 32,000 steps to one tenth of the peak learning rate: That actually works quite nicely! I wrote a dummy training loop to plot the current learning rate over a fake train using code like the above , and got this: ...with the output confirming that the values were good at the "milestone" point, the start and the end: I was initially a bit surprised by that, as at the time I ran it, I didn't realise that there was that split between the initial and the current learning rates on the optimiser, so I thought that the cosine scheduler would pick up whatever tiny starting value the warmup scheduler had overwritten the optimiser's learning rate with -- but that split saves the day. That means that now we have the outline of how to schedule our learning rate. But before we can put that into the code, we need to think about how it affects our checkpoints. Just like the scheduler and the optimiser, the learning rate scheduler -- or, indeed, our two schedulers here -- contain information about the state of the train. That means that if we recover from a checkpoint, we need to provide them with the information they need. If we just created them afresh, they'd start from the beginning -- for example, if we restarted from step 20,000 in a train like the one above, we'd start a new warmup from pretty much zero, and then start a fresh cosine decay. That would be bad: (Dummy test code here .) Now, we could use the parameter to initialize them with the correct current global step. But they have a state dict, like most other PyTorch objects, so the simplest thing to do is just to write that to another checkpoint file: ...and then load it likewise: (Dummy test code here .) Conveniently, if you save the state dict of a , it will also include the state of all of its component schedulers, and likewise if you reload it, it will load the components' states back in too. The one thing you have to be careful about is what they warn about in the PyTorch docs: Initializing a scheduler overwrites its optimizer’s s. When restoring a checkpoint, initialize the scheduler before calling your optimizer's to avoid overwriting the loaded learning rates. Luckily enough, in our code as it stands, we create all of the things that are checkpointed -- the optimiser and the scaler so far, but shortly the scheduler as well -- before we load in the state dicts, so that drops out quite nicely. So, we have some sketched-out code -- it's time to put it in place for the real training run. I won't go through the details of the changes to my existing DDP training code, though you can see the diff here if you're interested. Much of the complexity was due to keeping backward compatibility so that we don't have to always use a learning rate scheduler; remember that in this mini-series, I'm trying making various changes ("interventions") to the training loop in isolation, seeing whether each one improves things. So it's important to be able to easily train with or without learning rate scheduling; I did that with a flag in the Implementation-wise, initially I was thinking that it would be easiest to always have a scheduler, and in the "non-scheduled" case to just set it to a linear one that didn't change the value over the course of the train. But in the end it turned out to be easier to use as being the switch to tell the training loop which "mode" it was in. The placement of the code to create the schedulers was also a little tricky; the "natural" place was just after the optimiser is created, like it is in the example code above. However, at that point, we don't know how many global steps we're going to have in the train, because we don't have the dataset -- which means that working out the numbers to pass in to the schedulers for the warmup and decay steps would be impossible. It turned out to be easiest to put it in the function , just after the datasets are loaded, as at that point we have all of the information we need. Anyway, that's the code done, so let's see what happens! I wanted to do two trains; one with the learning rate scheduling, and one with just the new value for the learning rate, instead of . I was expecting the updated learning rate alone to be too high and to cause a very choppy train, but had high hopes for the train with the scheduling. Here's how it did; the scheduled learning rate train first: Here's what the training loss looked like over that: Quite a few loss spikes early on in the train when the learning rate is at its peak, but nothing unmanageable -- and, as you'd expect, things calmed down quite a lot later on. I also charted the learning rate, to make sure it really was doing what I thought it was doing: So, a pretty smooth train, and we definitely did the right learning rate scheduling. Time to upload it to Hugging Face , and see what the evals look like. Firstly, the smoke test: Reasonably coherent, at least, though it's not super-impressive. On to the loss on our test set: That's our best loss so far! Let's put it into the table: So, it definitely looked like it was worth it. But was it the scheduling of the learning rate that helped, or just the change from 0.0004 to 0.0014? I kicked off a second run with no scheduling, just a learning rate of 0.0014, to see what would happen. After about an hour, I noticed that the loss chart had stopped updating. The last point had a maximum and minimum loss but no average -- but after that, nothing: However, the learning rate was still being charted, so the train was definitely running: Looking at the checkpoint metadata showed what had happened. At global step 1851, we had this 3 : ...and at the next checkpoint at step 2468, we had this: ...and the same for all checkpoints thereafter. Clearly the parameters had gone off the rails -- exactly what we'd expect with an excessive learning rate: There was no point in continuing the train, as it was pretty much certainly unrecoverable, so I stopped it. Out of interest, I downloaded the model, but I couldn't even run the smoke test on it: So it was pretty clear that just updating the learning rate to 0.0014 was actively harmful. No need to upload that one to HF! And time to wrap up this experiment. While this has been quite a long post, I've really only scratched the surface of how learning rates are set. If I were doing things in more detail, the best would probably be to do a "sweep" over multiple values to try to at least approximate the best possible rate for this model. That would be pretty expensive for me, though, so I decided to stick with the DeepSeek number. It might not be ideal for the specific architecture that I'm using, given how different that is to theirs, but given the results, it's a decent one compared to what I was using. 4 Something that I found interesting is that exactly how to schedule your learning rate is still an area being actively researched. Even in my relatively minimal research, I came across three alternatives to the mainstream warmup-cosine decay pattern: I'm sure there are many more. But for this train, I decided to stick to the mainstream, and the results were pretty good! To reiterate, this has been the most positive intervention so far: So I'll stick with that, and move on to the next thing: what is the parameter that we're passing in to the AdamW optimiser? Tune in next time :-) Yes, I am foreshadowing here.  ↩ To make my earlier analogy about learning rate decaying over time in people as they age even more dubious, we can imagine this as being rather like someone middle-aged going on an ayahuasca retreat ;-)  ↩ If you're wondering how we had a valid maximum and minimum in that first checkpoint when the average was NaN, here's why: You might wonder how large labs work out the right learning rate given their training runs run to millions of dollars. The answer is there in that DeepSeek paper, as that's one of the things they were doing. They scaled their model down from the billions of parameters that they wanted to train to various smaller models, and worked out the optimal learning rate for each of the smaller models by doing full trains on them. Once they had a mapping from model size to the ideal learning rate for their architecture, they could extrapolate that to the large ones that they wanted to train. The problem is that those "smaller" models are actually quite a lot larger than the one we're training here! And while we could potentially scale it down even further, I suspect that such truly tiny models (say, 1M parameters) wouldn't train well enough to give any meaningful results.  ↩ From the paper: Specifically, the learning rate of the model reaches its maximum value after 2000 warmup steps, and then decreases to 31.6% of the maximum value after processing 80% of the training tokens. It further reduces to 10% of the maximum value after 90% of the tokens. , which is the optimiser we're applying it to. , which the optimiser's learning rate is multiplied by to work out where we want to start up. , which is likewise applied to the optimiser's learning rate to work out the value we're heading for. , which is the number of steps over which it should go from the initial learning rate to the final one. , which lets the scheduler know how many steps into its schedule it currently is -- this defaults to , meaning it hasn't started yet. This can be useful if you're resuming from a checkpoint, but for our purposes we can ignore it. , which is the same as the 's. , which is the number of steps before it reaches its minimum , the minimum learning rate we want to get to. , again the same as the 's. Per the Hugging Face paper, some people do warmup, then pause at a set level for a while, then start the cosine decay (warmup-stable-decay). DeepSeek use a relatively simple stepped function after a warmup. 5 I came across a 2025 paper " Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs " which says that a linear decay (after a warmup) outperforms cosine. Yes, I am foreshadowing here.  ↩ To make my earlier analogy about learning rate decaying over time in people as they age even more dubious, we can imagine this as being rather like someone middle-aged going on an ayahuasca retreat ;-)  ↩ If you're wondering how we had a valid maximum and minimum in that first checkpoint when the average was NaN, here's why: ↩ You might wonder how large labs work out the right learning rate given their training runs run to millions of dollars. The answer is there in that DeepSeek paper, as that's one of the things they were doing. They scaled their model down from the billions of parameters that they wanted to train to various smaller models, and worked out the optimal learning rate for each of the smaller models by doing full trains on them. Once they had a mapping from model size to the ideal learning rate for their architecture, they could extrapolate that to the large ones that they wanted to train. The problem is that those "smaller" models are actually quite a lot larger than the one we're training here! And while we could potentially scale it down even further, I suspect that such truly tiny models (say, 1M parameters) wouldn't train well enough to give any meaningful results.  ↩ From the paper: Specifically, the learning rate of the model reaches its maximum value after 2000 warmup steps, and then decreases to 31.6% of the maximum value after processing 80% of the training tokens. It further reduces to 10% of the maximum value after 90% of the tokens. ↩

0 views
Alex White's Blog 2 months ago

Are Design Tools Relevant Anymore

I was a product designer for a few years. I had switched careers to design after suffering burn out as a software engineer. During those years, my entire day was spent in Figma, building high fidelity mockups, leading workshops and creating prototypes. While Figma helped me move quickly, rapidly iterating after receiving user feedback, the engineer part of me always felt it was a throwaway step. You build something, only to then have somebody else build it again in code. I recently had to put on my design hat again, putting together interactive prototypes around a few redesign ideas. At first, I reached for Figma, but after fiddling around for an hour, decided to go a different route. While prototyping in Figma used to be faster than building in code, that’s no longer true. With Claude Code, building out frontend components is fast . Much faster than messing with layers, frames and symbols in Figma. Let me explain. Enterprise apps have well defined brand guidelines. Colors, type, scale. They are often built off an existing component library (think Bootstrap, shadcn). This means you can use Claude in a way that follows the look and feel of your application, and is constrained to the components the development team leverages. The rails help keep Claude from going off into the deep end. Design then becomes focused on solving the user’s problem through UX, less fiddling around with UI. I can open Freeform on my iPad, sketch something out, and prompt Claude to leverage our foundation to make my sketch a reality. Then, I can dig into the code and tweak things to be just right. The result is a more interactive, true to life prototype that gives your engineering team a head start with coded components. You get better feedback from users and stakeholders as it’s easier to visualize what the final product looks like. You discover pitfalls that might not have shown up until an engineer was halfway into the card. On top of all that, you move a lot faster, you’re designing and building in 1 step rather than 2, giving your engineering team a head start once designs are finalized. So then, what’s the point of Figma and Sketch? You can tell Figma is battling with this reality by pushing Figma make. The issue is, it’s too constrained and produces poor results. You can’t link it to existing coded components, Tailwind configs, etc. On the other hand, usin my approach requires a technical background. You need to guide with framework suggestions, foundational setup and be able to takeover and tweak yourself. That said, there in the shorter term there’s likely still a place for Figma and Sketch at the table. Designing using the method I talked about requires a technical background, otherwise your results will be all over the place, and small tweaks will be next to impossible. As the technology gets better though, I’ll be surprised if Figma and Sketch survive the next couple of years.

0 views
André Arko 2 months ago

Four months of Ruby Central moving Ruby backward

From the moment RubyGems was first created in 2004, Ruby Central provided governance without claiming ownership , to support the Ruby community. Providing governance meant creating processes to provide stability and predictability. Avoiding ownership meant allowing the community to contribute, to the point where unpaid volunteers created and controlled the entirety of RubyGems.org for many years. Last year, Ruby Central flipped that successful formula on its head . They now claim ownership of both Bundler and RubyGems, but refuse to provide governance . Ruby Central now claims sole control over all code and decisions, despite paying for only a few percent of the work required to create and sustain the projects across 22 years. Instead of providing stable and predictable processes, Ruby Central suddenly hijacked the Bundler and RubyGems codebases away from the existing maintainers, shut out the community, and started issuing the threats to sue. When confronted by the former maintainers after the hijacking, Marty Haught of Ruby Central stated (in a recorded video call) on September 17 that “yeah, we shouldn’t have changed that”. On September 18, Marty went on to write: In the past, we’ve made the mistake of conflating ownership of the code with ownership of the infra, and vice versa, and we’d like to straighten this out so that we aren’t put in a legal bind that requires us to take control of the entire codebase when, we all agree, that is not proper or correct given the existing model. In the words of Ruby Central itself, “we all agree, [taking control of the entire codebase] is not proper or correct.” Since the beginning of this conflict, Ruby Central has privately admitted it was wrong to hijack the GitHub organization and steal the repos, but has refused to acknowledge this in public. Unfortunately, despite privately admitting their actions were wrong, Ruby Central has publicly continued to dig their hole deeper. Instead of owning up to their mistake, they secretly negotiated a deal with Matz for ruby-core to take over the stolen RubyGems and Bundler repository, further violating the project governance policies. If this situation were just about me personally, I could believe it sprang from from individual disagreements. Ruby Central claims they had good reasons to unilaterally kick me out of the project, even though I don’t think their claims hold water . With that said, regardless of what you think about me personally, the other five long-term maintainers have never gotten any explanation of why they were suddenly kicked out or bypassed entirely, all in violation of existing project governance. In her only public interview about the situation, Ruby Central Executive Director Shan Cureton defended stealing Bundler from its team of fifteen years by saying the removed team “didn’t need to have the story, and it wasn’t their story to have”. Ruby Central has made their position clear: if they steal your project, you are not entitled to know their reasons , and neither is anyone else. There is nothing “community-oriented” about stealing the most-used gem in Ruby and refusing to share your reasons with the community. Despite Ruby Central’s unacceptable treatment of both projects and maintainers, the former RubyGems and Bundler team said we want to move Ruby forward . We offered Ruby Central a path to move past their illegitimate GitHub takeover, past their vicious personal attacks, and past their threats to sue us. It has been four months since we made that offer, and Ruby Central has not accepted . While declining to accept our offer, Ruby Central has nonetheless found the time to propose new governance documents for RubyGems . In those documents, they explicitly require existing maintainers approve adding or removing team members. That rule was already present in the previous governance, and is the exact rule that Ruby Central violated to execute their takeover . When asked why they violated the previous governance, and why the new governance would be any more trustworthy, Ruby Central refused to respond substantively, and then the question itself was hidden by marking it “off topic” . Instead of working to resolve the situation, Ruby Central has spent 4 months rejecting requests for an explanation, while repeatedly threatening to sue me personally. After Ruby Central suddenly took over the Bundler repo, I sent them a standard trademark notice. They replied with a threat to sue me. When I later informed Ruby Central I had learned they violated state employment law, they simply replied with the same threat to sue me again. They are threatening to sue me for “hacking” them, despite their own analysis publicly concluding “no evidence that user data or production operations were harmed” . Without seeking common ground, or even looking for some sort of resolution we can just live with and move on from, Ruby Central has offered all of us — nothing . Ruby Central has made no offer in reply to outreach from the other five maintainers. To me, after four grueling months of private “negotiation”, their entire offer is nothing more than to refrain from suing. But only if I agree to everything that they want. They say I must agree that I have no claim on the name Bundler, despite helping create it and leading the Bundler team for the last 15 years. They say I must agree I was paid legally and fairly, when California law clearly states I was not. They say I must agree that Ruby Central can take over open source projects they host, any time they feel like it, with no explanation, and no consequences. I don’t agree. Letting this situation stay unaddressed sets a dangerous precedent for all open source projects written in Ruby. Ruby Central has resolved nothing. Don’t let their delaying tactics convince you otherwise. The Ruby community cannot trust Ruby Central with control over our gems until there is accountability for destroying the very governance they were supposed to be providing . Until accountability arrives, take action . Tell Ruby Central they owe everyone an explanation for violating the project governance around six long-term maintainers, not just me. Don’t sponsor, attend, or speak at RubyConf. Contribute to projects that aren’t controlled by Ruby Central. The exiled maintainers are working on new projects, with a focus on clear governance, long-term financial sustainability, and community input: Join the gem.coop beta, and stop using RubyGems.org. Use jwl instead of RubyGems. Use or Ruby Butler instead of Bundler. A better world is possible! Ruby Central might want to keep Ruby in the past, but we can work together to build Ruby a future .

0 views
Chris Coyier 2 months ago

FOREVERGREEN

In the first few minutes, Ruby says to me, “ This is like The Giving Tr ee “, and by the end, I was like, “ OK, you’re right .”

0 views