Raph Koster 1 weeks ago

Looking back at a pandemic simulator

It’s been six years now since the early days of the Covid pandemic. People who were paying super close attention started hearing rumors about something going on in China towards the end of 2019 — my earliest posts about it on Facebook were from November that year. Even at the time, people were utterly clueless about the mathematics of how a highly infectious virus spread. I remember spending hours writing posts on various different social media sites explaining that the Infection Fatality Rates and the R value were showing that we could be looking at millions dead. People didn’t tend to believe me: “SEVERAL MILLION DEAD! Okay, I’m done. No one is predicting that. But you made me laugh. Thanks.” You can do the math yourself. Use a low average death estimate of 0.4%. Assume 60% of the population catches it and then we reach herd immunity (which is generous): But that’s with low assumptions… It was like typing to a wall. In fact, it’s pretty likely that it still is, since these days, the discourse is all about how bad the economic and educational impact of lockdowns was — and not about the fact that if the world had acted in concert and forcefully, we could have had a much better outcome than we did. The health response was too soft , the lockdown too lenient, and as a result, we took all the hits. Of course, these days people also forget just how deadly it was and how many died, and so on. We now know that the overall IFR was probably higher than 0.4%, but very strongly tilted towards older people and those with comorbidities. We also now know that herd immunity was a pipe dream — instead we managed to get vaccines out in record time and the ordinary course of viral evolution ended up reducing the death rate until now we behave as if Covid is just a deadlier flu (it isn’t, that thinking ignores long-term impact of the disease). The upshot: my math was not that far off — the estimated toll in the US ended up being 1.2 to 1.4 million souls, and worldwide it’s estimated as between 15 and 28.5 million dead. Plenty of denial of this, these days, and plenty of folks blaming the vaccines for what are most likely issues caused by the disease in the first place. Anyway, in the midst of it all, tired of running math in my spreadsheets (yeah, I was tracking it all in spreadsheets, what can I say?), I started thinking about why only a few sorts of people were wrapping their heads around the implications. The thing they all had in common was that they lived with exponential curves. Epidemiologists, Wall Street quants, statisticians… and game designers. Could we get more people to feel the challenges in their bones? So… I posted this to Facebook on March 24th, 2020: Three weeks ago I was idly thinking of how someone ought to make a little game that shows how the coronavirus spreads, how testing changes things, and how social distancing works. The sheer number of people who don’t get it — numerate people, who ought to be able to do math — is kind of shocking. I couldn’t help worrying at it, and have just about a whole design in my head. But I have to admit, I kinda figured someone would have made it by now. But they haven’t. It’s not even a hard game to make. Little circles on a plain field. Each circle simply bounces around. They are generated each with an age, a statistically real chance of having a co-morbid condition (diabetes, hypertension, immunosuppressed, pulmonary issues…), and crucially, a name out of a baby book. They can be in one of these states: In addition, there’s a diagnosed flag. We render asymptomatic the same as healthy. We render each of the other states differently, depending on whether the diagnosed flag is set. They show as healthy until dead, if not diagnosed. If diagnosed, you can see what stage they are in (icon or color change). The circles move and bounce. If an asymptomatic one touches a healthy one, they have a statistically valid chance of infecting. Circles progress through these states using simple stats. We track current counts on all of these, and show a bar graph. Yes, that means players can see that people are getting sick, but don’t know where. The player has the following buttons. The game ticks through days at an accelerated pace. It runs for 18 months worth of days. At the end of it, you have a vaccine, and the epidemic is over. Then we tell you what percentage of your little world died. Maybe with a splash screen listing every name and age of everyone who died. And we show how much money you spent. Remember, you can go negative, and it’s OK. That’s it. Ideally, it runs in a webpage. Itch.io maybe. Or maybe I have a friend with unlimited web hosting. Luxury features would be a little ini file or options screen that lets you input real world data for your town or country: percent hypertensive, age demographics, that sort of thing. Or maybe you could crowdsource it, so it’s a pulldown… Each weekend I think about building this. So far, I haven’t, and instead I try to focus on family and mental health and work. But maybe someone else has the energy. I suspect it might persuade and save lives. Some things about this that I want to point out in hindsight. Per the American Heart Association, among adults age 20 and older in the United States, the following have high blood pressure: Per the American Diabetes Association, Per studies in JAMA, Next, realize that because the disease spreads mostly inside households (where proximity means one case tends to infect others), this means that protecting the above extremely large slices of the population means either isolating them away from their families, or isolating the entire family and other regular contacts. People tend to think the at-risk population is small. It’s not. The response, for Facebook, was pretty surprising. The post was re-shared a lot, and designers from across the industry jumped in with tweaks to the rules. Some folks re-posted it to large groups about public initiatives, etc. There was also, of course, plenty of skepticism that something like this would make any difference at all. The first to take up the challenge was John Albano, who had his game Covid Ops up and running on itch.io a mere six days later . You can still play it there! Stuck in the house and looking for things to do. Soooo, when a fellow game dev suggested a game idea and basic ruleset along with “I wish someone would make a game like this,” I took that as a challenge to try. Tonight (this morning?), the first release of COVID OPS has been published. John’s game was pretty faithful to the sketch. You can see the comorbidities over on the left, and the way the player has clicked on 72 year old Rowan — who probably isn’t going to make it. As he updated it, he added in more detailed comorbidity data, and (unfortunately, as it turns out) made it so that people were immune after recovering from infection. And of course, like the next one I’ll talk about, John made a point of including real world resource links so that people could take action. By April 6th, another team led by Khail Santia had participated in Jamdemic 2020 and developed the first version of In the Time of Pandemia. He wrote, The compound I stay at is about to be cordoned. We’ve been contact-traced by the police, swabbed by medical personnel covered in protective gear. One of our housemates works at a government hospital and tested positive for antibodies against SARS-CoV-2. The pandemic closes in from all sides. What can a game-maker do in a time like this? I’ve been asking myself this question since the beginning of community quarantine. I’m based in Cebu City, now the top hotspot for COVID-19 in the Philippines in terms of incidence proportion. This game would go on to be completed by a fuller team including a mathematical epidemiologist, and In the Time of Pandemia eventually ended up topping charts on Newgrounds when it launched there in July of 2020. This game went viral and got a ton of press across the Pacific Rim . The team worked closely with universities and doctors in the Philippines and validated all the numbers. They added local flavor to their levels representing cities and neighborhoods that their local players would know. Gregg Victor Gabison, dean of the University of San Jose-Recoletos College of Information, Computer & Communications Technology, whose students play-tested the game, said, “This is the kind of game that mindful individuals would want to check out. It has substance and a storyline that connects with reality, especially during this time of pandemic.” Not only does the game have to work on a technical basis, it has to communicate how real a crisis the pandemic is in a simple, digestible manner. Dr. Mariane Faye Acma, resident physician at Medidas Medical Clinic in Valencia, Bukidnon, was consulted to assess the game’s medical plausibility. She enumerated critical thinking, analysis, and multitasking as skills developed through this game. “You decide who are the high risks, who needs to be tested and isolated, where to focus, [and] how much funds to allocate….The game will make players realize how challenging the work of the health sector is in this crisis.” “Ultimately, the game’s purpose is to give players a visceral understanding of what it takes to flatten the curve,” Santia said. I think most people have no idea that any of this happened or that I was associated with it. I only posted the design sketch on Facebook; it got reshared across a few thousand people. It wasn’t on social media, I didn’t talk about it elsewhere, and for whatever reason, I didn’t blog about it. I have had both these games listed on my CV for a while. Oh, I didn’t do any of the heavy lifting… all credit goes to the developers for that. There’s no question that way more than 95% of the work comes after the high-level design spec. But both games do credit me, and I count them as games I worked on. A while back, someone on Reddit said it was pathetic that I listed these. I never quite know what to make of comments like that (troll much?!?). No offense, but I’m proud of what a little design sketch turned into, and proud of the work that these teams did, and proud that one of the games got written up in the press so much; ended up being used in college classrooms; was vetted and validated by multiple experts in the field; and made a difference however slight. Peak Covid was a horrendous time. Horrendous enough that we have kind of blocked it from our memories. But I lost friends and colleagues. I still remember. Back then I wrote, This is the largest event in your lifetime. It is our World War, our Great Depression. We need to rise the occasion, and think about how we change. There is no retreat to how it used to be. There is only through. A year later, the vaccine gave us that path through, and here we are now. But as I write this, we have the first human case of H5N5 bird flu; it was only a matter of time. Maybe these games helped a few people get through it all. They were played by tens of thousands, after all. Maybe they will help next time. I know that the fact that they were made helped me get through, that making them helped John get through, helped Khail get through — in his own words: In the end, the attempt to articulate a game-maker’s perspective on COVID-19 has enabled me to somehow transcend the chaos outside and the turmoil within. It’s become a welcome respite from isolation, a thread connecting me to a diversity of talents who’ve been truly generous with their expertise and encouragement. As incidences continue to rise here and in many parts of the world, our hope is that the game will be of some use in showing what it takes to flatten the curve and in advocating for communities most in need. So… at minimum, they made a real difference to at least three people. And that’s not a bad thing for a game to aspire to. 328 million people in the US. 60% of that is 196 million catch it. 0.4% of that is 780,000 dead. asymptomatic but contagious symptomatic 70% of asymptomatic cases turn symptomatic after 1d10+5 days. The others stay sick for the full 21 days. Percent chance of moving from symptomatic to severe is based on comorbid conditions, but the base chance is 1 in 5 after some amount of days. Percent chance of moving from severe to critical is 1 in 4, modified by age and comorbidities, if in hospital. Otherwise, it’s double. Percent chance of moving from critical to dead is something like 1 in 5, modified by age and comorbidities, if in hospital. Otherwise, it’s double. Symptomatic, severe, and critical circles that do not progress to dead move to ‘recovered’ after 21 days since reaching symptomatic. Severe and critical circles stop moving. Hover on a circle, and you see the circle’s name and age and any comorbidities (“Alison, 64, hypertension.”) Test . This lets them click on a circle. If the circle is asymptomatic or worse, it gets the diagnosed flag. But it costs you one test. Isolate . This lets them click on a circle, and freezes them in place. Some visual indicator shows they are isolated. Note that isolated cases still progress. Hospitalize . This moves the circle to hospital. Hospital only has so many beds. Clicking on a circle already in hospital drops the circle back out in the world. Circles in hospital have half the chance or progressing to the next stage. Buy test . You only have so many tests. You have to click this button to buy more. Buy bed . You only have this many beds. You have to click this button to buy more. Money goes up when circles move. But you are allowed to go negative for money . Lockdown. Lastly, there is a global button that when pressed, freezes 80% of all circles. But it gradually ticks down and circles individually start to move again, and the button must be pressed again from time to time. While lockdown is running, it costs money as well as not generating it. If pressed again, it lifts the lockdown and all circles can move again. At the time that I posted, I could tell that people were desperately unwilling to enter lockdown for any extended period of time; but “The Hammer and the Dance” strategy of pulsed lockdown periods was still very much in our future. I wanted a mechanic that showed population non-compliance. There was also quite a lot of obsessing over case counts at the time, and one of the things that I really wanted to get across was that our testing was so incredibly inadequate that we really had little idea of how many cases we were dealing with and therefore what the IFR (infection fatality rate) actually was. That’s why tests are limited in the design sketch. I was also trying to get across that money was not a problem in dealing with this. You could take the money value negative because governments can choose to do that. I often pointed out in those days that if the government chose, it could send a few thousand dollars to every household every few weeks for the duration of lockdown. It would likely have been less impact to the GDP and the debt than what we actually did. I wanted names. I wanted players to understand the human cost, not just the statistics. Today, I might even suggest that an LLM generate a little biography for every fatality. Another thing that was constantly missed was the impact of comorbidities. To this day, I hear people say “ah, it only affected the old and the ill, so why not have stayed open?” To which I would reply with: For non-Hispanic whites, 33.4 percent of men and 30.7 percent of women. For non-Hispanic Blacks, 42.6 percent of men and 47.0 percent of women. For Mexican Americans, 30.1 percent of men and 28.8 percent of women. 34.2 million Americans, or 10.5% of the population, have diabetes. Nearly 1.6 million Americans have type 1 diabetes, including about 187,000 children and adolescents 4.2% of of the population of the USA has been diagnosed as immunocompromised by their doctor

Data Analysis Science Statistics

pabloecortez 2 weeks ago

You can read the web seasonally

What if you read things around the web the way you watch movies or listen to music? A couple of days ago I made a post on Mastodon introducing lettrss.com, a project I made that takes a book in the public domain and sends one chapter a day to your RSS reader. Xinit replied with a great point about RSS feed management: This is fascinating, but I know how it would go based on the thousands of unread RSS feeds I've had, and the thousands of unheard podcasts I subscribed to. I'd end up with an RSS of unread chapters, representing a whole book in short order. Regardless of my inability to deal, it remains a great idea, and I will absolutely recommend while hiding my shame of a non-zero inbox. When I first started using RSS, I thought I'd found this great tool for keeping tabs on news, current events, and stuff I should and do care about. After adding newspapers, blogs, magazines, publications, YouTube channels and release notes from software I use, I felt a false sense of accomplishment, like I'd finally been able to wrangle the craziness of the internet into a single app, like I had rebelled against the algorithm™️. But it didn't take long to accumulate hundreds of posts, most of which I had no true desire to read, and soon after I abandoned my RSS reader. I came back to check on it from time to time, but its dreadful little indicator of unread posts felt like a personal failure, so eventually I deleted it entirely. Will Hopkins wrote a great post on this exact feeling. I don't actually like to read later : I used Instapaper back in the day, quite heavily. I built up a massive backlog of items that I'd read occasionally on my OG iPod Touch. At some point, I fell off the wagon, and Instapaper fell by the wayside. [...] The same thing has happened with todo apps over the years, and feed readers. They become graveyards of good intentions and self-imposed obligations. Each item is a snapshot in time of my aspirations for myself, but they don't comport to the reality of who I am. I couldn't have said it better myself. This only happens with long-form writing, whenever I come across an essay or blog post that I know will either require my full attention or a bit more time than I'm willing to give it in the moment. I've never had that issue with music. Music is more discrete. It's got a timestamp. I listen to music through moods and seasons, so much so that I make a playlist for every month of the year like a musical scrapbook. What if we took this approach to RSS feeds? Here's what I replied to Xinit: This is something I find myself struggling with too. I think I'm okay knowing some RSS feeds are seasonal, same as music genres throughout the year. Some days I want rock, others I want jazz. Similarly with RSS feeds, I've become comfortable archiving and resurfacing feeds. For reference, I follow around 10 feeds at any given time, and the feeds I follow on my phone are different from the ones on my desktop. You shouldn't feel guilty about removing feeds from your RSS readers. It's not a personal failure, it's an allocation of resources like time and attention.

Data Analysis Tutorial Web Development

annie's blog 4 weeks ago

Duck duck duck dichotomy

Have you ever played Duck Duck Goose 1 and the person who’s it keeps walking and walking and walking and walking around and never picks the goose? It’s really boring. There are very few actual dichotomies. Most choices are not binary. Most choices are more like: “Here is an array of options you can  recognize (the subset of a potentially infinite array of options you can’t even see because you’re only able to recognize what’s familiar). Pick one!” No wonder making decisions is so exhausting. I can spend a lot of time musing over the array of options, but eventually I  narrow it down to one option and then it’s time to make the real choice which is  a dichotomy: Yes, do it, action, go, forward. Choosing an option and then saying No to the option I selected for myself  is wild! Why would I do that? Because choice is dangerous. Exerting the force of my will upon the world, or at least attempting to do so, is a risk. Risk of pain, risk of failure, risk of being wrong (whatever that means), risk of ending up in a worse situation, risk of being misunderstood, risky risky risky! Sometimes it feels safer to just hang out, not move, wait and see. It isn’t safer, usually, but it feels  safer. Passivity is a way to live but it’s not the way I like to live. I like to happen. I like to be the thing that’s happening in my own life. I like to be the main character in my own story. And  I only get to happen by choosing. nothing happens and/or things happen to me but I never happen. I make choices all day long but most of those are inconsequential, like: what time will I get up, what food will I eat, will I be impatient or kind with my child, will I be impatient or kind with myself, will I make that phone call, will I go to the gym, will I worry, will I be grateful, will I floss today, will I finish this blog post, will I actually put away the clean laundry? The answer to that last one is No. It’s going to sit in the basket for a few days. These choices all seem inconsequential but maybe they aren’t. Tiny choices become a trend, the trend creates a groove, the groove becomes a rut and I walk the rut because it’s easier to stick with what’s familiar than to enact change, so here I am: that’s my life. I can change it by making different tiny choices, one after another. It’s not about the right choice or wrong choice or the accurate choice or idiotic choice or worst choice or best choice. It’s about exerting your will. Choosing something. Selecting an option and then acting on it. Saying Yes. Duck duck duck duck duck goose. It’s about the goose. It doesn’t matter who the goose is. It matters that you pick a goose. Otherwise there’s no game, just a bunch of kids sitting in a circle being bored and sad. Everyone sits in a circle. One person walks around the circle, tapping others and saying duck  until choosing a goose . The chosen goose tries to tag them before they sit down in the goose’s spot. nothing happens and/or things happen to me but I never happen. Everyone sits in a circle. One person walks around the circle, tapping others and saying duck  until choosing a goose . The chosen goose tries to tag them before they sit down in the goose’s spot.

Culture Data Analysis Writing

emiruz 1 months ago

Modelling beliefs about sets

Here is an interesting scheme I encountered in the wild, generalised and made abstract for you, my intrepid reader. Let \(X\) be a set of binary variables. We are given information about subsets of \(X\), where each update is a probability ranging over a concrete set, the state of which is described by an arbitrary quantified logic formula. For example, \[P\bigg\{A \subset X \mid \exists_{x_i, x_j \in A} \big(x_o \ne x_j))\bigg\} = p\] The above assigns a probability \(p\) to some concrete subset A, with the additional information that at least 1 pair of its members do not have the same value.

Data Analysis Statistics

Alex Molas 1 months ago

Bayesian A/B testing is not immune to peeking

Introduction Over the last few months at RevenueCat I’ve been building a statistical framework to flag when an A/B test has reached statistical significance. I went through the usual literature, including Evan Miller’s posts. In his well known “How Not to Run an A/B Test” there’s a claim that with Bayesian experiment design you can stop at any time and still make valid inferences, and that you don’t need a fixed sample size to get a valid result. I’ve read this claim in other posts. The impression I got is that you can peek as often as you want, stop the moment the posterior clears a threshold (eg $P(A>B) > 0.95$), and you won’t inflate false positives. And this is not correct. If you’re an expert in Bayesian statistics this is probably obvious, but it wasn’t for me. So I decided to run some simulations to see what really happens, and I’m sharing the results here in case it can be useful for others.

Testing Data Analysis Statistics

Dangling Pointers 1 months ago

SuSe: Summary Selection for Regular Expression Subsequence Aggregation over Streams

SuSe: Summary Selection for Regular Expression Subsequence Aggregation over Streams Steven Purtzel and Matthias Weidlich SIGMOD'25 A common use case for regular expressions is to check whether a particular input string matches a regular expression. This paper introduces a new use case, which is generating statistics over streams. Imagine an infinite stream of characters constantly being processed, and a dashboard which a person occasionally views. The dashboard shows statistics, e.g., “there have been matches in the last hours”. It is OK for these statistics to be approximate. The final quirk introduced by this paper is to allow for noise. In particular, all regular expressions implicitly accept “unexpected inputs” at any time and simply behave as if the unexpected input had never appeared. Counterbalancing this flexibility is a sliding time window which requires that if a particular subset of the input stream is going to match, it must do so with a bounded number of characters. A regular expression can be represented as an NFA . The N stands for non-deterministic , and that is the tricky bit. For example, say the regular expression to be matched against is . This matches strings which begin with , followed by 0 or more characters, followed by exactly 1 , followed by . Another way to write this expression is: ( stands for “one or more s”). Imagine you were designing a state machine to recognize strings which match this pattern. It starts off easy, the machine should expect the first input to be . The next expected input is , but then what? That first could be one of many represented by the , or it could be the last right before the . The state machine doesn’t know how to classify the inputs that it has seen until it sees the final (or some other non-B character, indicating no match). A common solution to this problem is like the metaverse, when that first B comes, split the universe into two, and then send all subsequent inputs to both. An easy way to implement this is to use a bit vector to track all possible states the NFA could be in. The NFA starts in the initial state (so only one bit is set), and transitions through states like you would expect (only one bit is set at a time). When that first comes however, two bits are set, indicating that there are two possible ways the input could match. The NFA above can be represented with 6 states: The following table shows how the bit vector could evolve over time (one new input accepted each time step, bit vectors are written with the least-significant bit on the right): The key insight in this paper is to track states with counters, not bits. Each counter represents the number of matches that have been seen thus far in the stream. For example, when an input comes along, the counter associated with state 1 is increased by one. When a comes along, the counter associated with state 4 is increased by the value of the counter associated with state 3. Section 5.1 of the paper explains how counter increment rules can be derived for a specific NFA. This system described by this paper contains multiple sets of counters. Some counters are global, and some are local to each input character. The purpose of the local counters is to allow for “garbage collection”. The idea is that the system keeps a sliding window of the most important recent inputs, along with global and local state counters. When a new input comes, the local state counters are used to determine which input in the sliding window to delete. This determination is made based on an estimate of how likely each input is to generate a match. When the user views the dashboard, then all matches within the sliding window are identified. It is OK if this process is expensive, because it is assumed to be rare. Fig. 9 has a comparison of this work (StateSummary) against other approaches: Source: https://dl.acm.org/doi/10.1145/3725359 Dangling Pointers State machines are notoriously hard to optimize with SIMD or specialized hardware because of the fundamental feedback loop involved. It would be interesting to see if this approach is amenable to acceleration. Subscribe now

Data Analysis

ava's blog 1 months ago

pain management

Allow me to crash out for a second. Since roughly a month, I’m experiencing a flareup in my spondyloarthritis (Ankylosing Spondylitis or Bechterew’s disease…). This is a type of arthritis that primarily affects the spine and usually some other joints. I first noticed it in the base of my right thumb that was painful and a bit stiff (this has now mostly resolved) and plantar fasciitis (the fascia in your foot arch, basically; my body loves attacking in this area for some reason, as I used to have frequent Achilles tendonitis as a teen). This first caused unexpected pain in some moments of walking and also resulted in issues using my phone, using a controller, and every day stuff that needs thumb mobility and pressure on the thumb. I also noticed general aches especially after resting and following some exercise. One example was having weirdly stiff elbows and shoulders after indoor cycling, which I hadn’t had in quite a while after treatment worked. This was followed by sacroiliitis (inflammation where hip and spine meet im the lower back) first on the right and now on both sides, and sharp pain in the upper thoracic spine (between the shoulder blades). That means while walking, sitting, and lying down, I have pain in the whole area of my lower back and hips, and as I breathe and my upper spine moves, I am in pain as well. Every time I breathe in, there’s a knife in my back. As nerves are affected too, I have shooting pains down my legs and into my shoulders and neck. My right leg occasionally randomly collapses away from under me due to this, but I haven’t fallen yet. Unfortunately, everything gets worse with rest (both sitting and lying down) but obviously, I can’t exercise 24/7. It’s generally difficult to hit the sweet spot each day where exercise helps and doesn’t further aggravate everything. I recently had such a great workout (30 minutes treadmill, 20 minutes cycling, 20 mins mix of yoga and pilates) that made me feel as if I had just gotten a dose of heavy painkillers, but that relief only lasted for about two hours max. I still need to sleep, study, and do an office job. I tried to go back to a low dose of Prednisone and it obviously helps a bit, but I don’t wanna be on it - I was on 80mg last year, tapered down to 50mg, and then couldn’t go lower for months until new treatment worked. I had the whole experience of side effects, even medically induced Cushing’s Disease and issues with my blood sugar. When I recently tried between 2mg-4mg, I was immediately back with the constant thirst and peeing (= blood sugar issues). It was so disrupting I had to stop. It’s sad seeing everything fall apart again. I see it in the way more stuff is lying around in the apartment than usual. Chores take longer or get procrastinated on. I am low energy. I barely go to the gym anymore and prefer to exercise at home. I heat up a heating pad for my back like 4 times a day, it’s not more than that only because I’m often too lazy and stubborn to do it more often. I try so hard not to take painkillers. You aren’t supposed to take ibuprofen with Crohn’s disease, but I have to sometimes. But when I max out my limit for it, I add paracetamol, which works less well but helps at least some. I’m especially careful with that so I don’t harm my liver. So it all becomes this big monster of trying to get the energy to exercise and making time for it in my day, then holding myself over with heating pads and stretches and distractions, before turning to painkillers as a last resort, and alternating/mixing them. I almost treat it like a luxury good, something to indulge in, because of weird shaming around it. I remember this absolutely disrespectful interview with a doctor I read this year in which he was clutching his pearls about people taking ibuprofen and that it’s so dangerous and poisonous and that people should just stop. He talked about it as if people just take these for fun over a papercut. I wish I could shit on his doormat. Peak example of a healthy and non-menstruating person with zero empathy. So every couple days, I allow myself to take them, and my inner monologue is really like “Oh well, I deserve this. I’m splurging on it. It’s okay for today, I held out long enough. But it is kind of extra. Maybe I could have skipped this one too. Is it even bad enough?” And then they kick in and I truly realize how bad it was. You get used to it after a while, your brain kind of tuning out some of it, but it’s still this constant static sound in the background that ruins everything. Realistically, if I’m being honest, I would need painkillers every morning and evening every single day. And if we’re being even more real, they would not be the freely available pills, but the highly controlled patches. But that also opens up a whole lot of other possible issues. It sucks! It fucking sucks. I throw myself into my studies, into my volunteer work, into lengthy blog posts and anything like that so there is finally some focus away from my body. If I’m in a flow state, I don’t have to be in here, I don’t have to witness this. I love slowly getting tired on the sofa and falling asleep while doing something else (like watching something) and I love being busy with something (like studying late) until I’m dead tired and then crashing into bed, falling asleep quickly. Because the alternative is going to bed in a timely manner and lying awake, being hyperaware of everything that hurts, and it starts hurting more and more as time goes on, and I’m lying there wondering how I can possibly manage the next 30 years like this, wishing it was over. I don’t have to endure this forever, of course. This flareup just needs to pass, or I need to switch medications, or I finally try and get a proper pain management going for these phases, and then everything goes back to normal. But in these moments, none of that matters. I just want it to be over. Every morning I get teleported back into this hurtful mess, and everything that would help causes more issues. It makes me angry and close to tears all the time, and makes me worry if I’ve developed antibodies to infliximab. My injection this week changed nothing. Next week will be super busy with traveling and attending events, and I’m tired of portioning out the relief. I’ll take what I need to make it, and I hope the rheumatology appointment the week after will be helpful. If anyone takes anything away from this, it should be the obvious fact that not all pain can be successfully treated with lifestyle changes and people aren’t necessarily taking “the easy way out” with painkillers. And if you look at people and think you know what causes their pain, you should consider that you never know what came first - the pain or the other things. With pain like that, it’s no wonder many people choose to avoid exercise, eat to feel happy, or self-medicate with drugs that are easier to get than a fent patch; and if people regularly get stuck on months of Prednisone, that does not help. My usually ~58kg self ballooned up to 75kg on ~6 months of Prednisone. After a year off, I’m 10kg down, 7 more to go. Reply via email Published 26 Oct, 2025

Science Programming Data Analysis

./techtipsy 1 months ago

Comparing the power consumption of a 30 year old refrigerator to a brand new one

Our apartment came with a refrigerator. It was alright, it made things cold, it kept them cold. It was also incredibly noisy, and no matter how much I fiddled with its settings, the compressor was always running and any ice cream left in the deep freeze part got rock solid. 1 When I hooked up one of my smart plugs to it, I soon learned why: one of the two compressors was running all the time. This lead to a huge block of ice forming on the back of the main compartment, and the deep freeze section icing up really quickly. I suspect that the thermostat may have been busted and contributed to the issue, but after trying to repair a dishwasher, getting cut about 10 times on my hands and losing, I had zero interest in attempting another home appliance repair on my own. The refrigerator was the UPO Jääkarhu ( jääkarhu means polar bear in Finnish), and the manual that the previous owner had still kept around had July 1995 on it, meaning that the refrigerator was about the same age as I am: 30 years old. Not bad at all for a home appliance! I shopped around for a new refrigerator and got a decent one that’s about the same size, except newer. I won’t mention the brand here because they didn’t pay me anything and this post really isn’t a refrigerator review, but it was in the low-to-midrange class, sporting a “no frost” feature, and could be bought for about 369 EUR in Estonia in the summer of 2025. Based on some napkin math, I assumed that within a few years, the electricity savings will cover the upfront cost of buying the new refrigerator, assuming that it doesn’t break down. After letting it run for a while, I had some data! Turns out that the old one consumed 3.7x more electricity compared to the new one. Here are some typical daily power consumption numbers: The difference is more noticeable if we zoom out a bit. Moving from ~78 kWh to ~21 kWh consumed each month is nice. Around the time we replaced the refrigerator, we also got a working dishwasher, and with those two combined I saw a solid 10-20% decrease in the overall power usage of the whole apartment. We went from using 334 kWh in June to 268 kWh in July, 298 kWh in August and 279 kWh in September. Remember that napkin math I made earlier? If we assume about 57 kWh savings per month, and an average electricity price of 17 cents per kWh (based on actual rates during August 2025), it will take about 38 months or a bit over 3 years for the new refrigerator to pay off in the most pessimistic scenario. The pay-off will likely be larger if we account for energy prices usually rising during winter. Don’t worry about the old refrigerator, we gave it away to a person who needed one for their new home in the short term as a stopgap until they get further with renovation work. Even got some good chocolate for that! The only point of concern with this change is that I don’t really trust the new refrigerator to last as long as the old one. The previous one was good for 30 years if you look past the whole ice buildup, heat and noise, but with the new one I suspect that it’s not going to last as long. At least my new refrigerator doesn’t have a Wi-Fi-connected screen on it! honestly, I miss that a lot, the ice cream was colder for longer, I ate it in smaller bites and savored it more.  ↩︎ old refrigerator: 2.6 kWh new refrigerator: 0.7 kWh honestly, I miss that a lot, the ice cream was colder for longer, I ate it in smaller bites and savored it more.  ↩︎

Hardware Electronics Data Analysis

Nick Khami 1 months ago

XGBoost Is All You Need

import LLMFeatureDemo from "../../components/blog/XGBoostIsAllYouNeed/LLMFeatureDemo.astro"; import ModelComparisonDemo from "../../components/blog/XGBoostIsAllYouNeed/ModelComparisonDemo.astro"; import FeatureImportanceDemo from "../../components/blog/XGBoostIsAllYouNeed/FeatureImportanceDemo.astro"; {/* <!-- TODO: Add a more concrete opening story - maybe a specific moment at the startup where you realized asking LLMs for direct answers was broken? --> */} I spent two and a half years at a well-funded search startup building systems that used LLMs to answer questions via RAG (Retrieval Augmented Generation). We'd retrieve relevant documents, feed them to an LLM, and ask it to synthesize an answer. I came out of that experience with one overwhelming conviction: we were doing it backwards. The problem was that we were asking LLMs "what's the answer?" instead of "what do we need to know?" LLMs are brilliant at reading and synthesizing information at massive scale. You can spawn infinite instances in parallel to process thousands of documents, extract insights, and transform unstructured text into structured data. They're like having an army of research assistants who never sleep and work for pennies. {/* <!-- TODO: Add personality - maybe joke about why you picked this problem, or a detail about trying the "ask LLM directly" approach first and failing? --> */} Forecasting how many rushing yards an NFL running back will gain in their next game is a perfect example of this architecture. It's influenced by historical statistics (previous yards, carries, opponent defense), qualitative factors (recent press coverage, injury concerns, offensive line health), and game context (Vegas betting lines, projected workload). {/* <!-- TODO: Add personality - show a real example of ChatGPT giving a plausible-sounding but wrong prediction? Make it funny? --> */} You could ask ChatGPT's Deep Research feature to predict every game in a week. It would use web search to gather context, think about each matchup, and give you predictions. This approach is fundamentally broken. It's unscalable (each prediction requires manual prompting and waiting), the output is unstructured (you'd need to manually parse each response and log it in a spreadsheet), it's unreliable (LLMs are trained to sound plausible, not to optimize for numerical accuracy), and you can't learn from it (each prediction is independent—there's no way to improve based on what worked). This is the "ask the LLM what's the answer" approach. It feels like you're doing AI, but you're really just creating an expensive, slow research assistant that makes gut-feel predictions. {/* <!-- TODO: Add personality - maybe contrast this with how a human would do feature engineering? Show the "aha" moment when you realized this approach? --> */} Instead of asking "How many yards will Derrick Henry rush for?", we ask the LLM to transform unstructured information into structured features. Search for recent press coverage and rate sentiment 1-10. Analyze injury reports and rate concern level 1-5. Evaluate opponent's run defense and rate weakness 1-10. This is scalable (run 100+ feature extractions in parallel), structured (everything becomes a number XGBoost can use), and improves over time (XGBoost learns which features actually matter). I started with basic statistical features from the NFL API: yards and carries from the previous week, 3-week rolling averages, that kind of thing. These are helpful, but they miss important context. So I had the LLM engineer seven qualitative features: press coverage sentiment, injury concerns, opponent defense weakness, offensive line health, Vegas sentiment, projected workload share, and game script favorability. An agent loop with web search processed context about each player and game to populate these features—searching for news in the week leading up to the game and rating each factor on a numerical scale. <LLMFeatureDemo /> Once we run this process for every running back each week, we end up with a dataset that has both statistical and LLM-engineered qualitative features. {/* <!-- TODO: Add personality - what were you hoping for? What did you expect to happen? --> */} I split the data chronologically—early weeks for training, later weeks for testing—and trained two models. A baseline using only statistical features (previous yards, carries, rolling averages), and an enhanced model using both statistical and LLM-engineered features. {/* <!-- TODO: Add personality - show your emotional reaction to seeing these numbers. Were you shocked? Skeptical? Did you run it again to make sure? --> */} <ModelComparisonDemo /> The LLM-enhanced model reduced prediction error by 22.6% . The baseline model was actually worse than just predicting the average yards (R² of -0.025), while the enhanced model explained 38.6% of the variance. But that's not the interesting part. The interesting part is what XGBoost actually learned. {/* <!-- TODO: Add personality - build up the surprise here. Maybe say "I looked at the feature importance rankings expecting to see..." --> */} <FeatureImportanceDemo /> Six of the top seven most important features are LLM-engineered. The top feature is average carries over the last 3 weeks (statistical). The second most important feature is press coverage sentiment (LLM). Then game script prediction (LLM), Vegas sentiment (LLM), projected workload share (LLM), offensive line health (LLM), and injury concern (LLM). I didn't tell XGBoost that press sentiment matters more than injury concerns, or that game script prediction is more important than offensive line health. The model discovered these patterns on its own by analyzing which features actually correlated with rushing yards. The most predictive LLM feature, press coverage sentiment, captures momentum and narrative that doesn't show up in raw statistics. When a running back is getting positive press coverage, they tend to get more carries and perform better. XGBoost found this signal and learned to weight it heavily. This is the power of the hybrid approach: LLMs transform messy, unstructured context into clean features. XGBoost discovers which features actually matter. Neither could do this alone. {/* <!-- TODO: Add personality - make the transition to "this is actually a bigger problem" more dramatic. Show frustration with the current state? --> */} This isn't just about NFL predictions. Email prioritization, Slack message routing, pull request quality assessment, prediction market opportunities, customer support triage—every one of these problems has the same structure. Some structured data combined with unstructured context that needs to be transformed into a prediction. The architecture is identical every time: use LLMs in parallel to extract features from unstructured data, combine with structured features, train XGBoost to find patterns, deploy and iterate. Setting this up from scratch takes way too much time. I want tools that make this trivial—upload your data, describe what you want to predict, and get back a trained model with a deployment-ready API. {/* <!-- TODO: Add personality - make this section angrier? More pointed? This is your villain reveal. --> */} The tools I'm describing could exist today. The technology is mature and proven. So why hasn't anyone built them? Random forests don't raise $1B rounds. Founders are building pure-LLM systems because that's what gets funded. VCs get excited about foundation models and AGI, not about elegant hybrid architectures that combine 2019-era XGBoost with LLM feature engineering. This is the real problem with modern AI development. Not that the technology isn't good enough—it's that incentives are backwards. VC-led engineering is bad engineering. The best technical solutions rarely align with what makes a compelling pitch deck. Everyone's building the wrong thing because they're building what raises money instead of what solves problems. If you're a builder who cares more about solving real problems than raising huge rounds, there's a massive opportunity here. Build the boring, practical tools that let people deploy these hybrid systems in minutes instead of weeks. Build what actually works instead of what sounds impressive. {/* <!-- TODO: Add personality - end on a more concrete note? What are YOU going to build next? What do you wish existed? --> */} The future of ML isn't pure LLMs or pure classical ML—it's knowing which tool to use for which job. Don't ask LLMs "what's the answer?" Ask them "what do we need to know?" Then let XGBoost find the patterns in those answers. Want to see the full implementation? Check out the complete Jupyter notebook walkthrough with all the code, data processing steps, training, and visualizations.

AI Data Analysis

DuckTyped 1 months ago

An Illustrated Introduction to Linear Algebra

This post assumes you know algebra, but no linear algebra. Lets dive in. There are two big ideas I want to introduce in the first chapter: Gaussian elimination, (which is not strictly a linear algebra thing, and has been around for years before linear algebra came along), and row picture versus column picture, which is a linear algebra thing. Let’s say you have a bunch of nickels and pennies, and you want to know how many of each do you need to have 23 cents . You could write that as an equation that looks like this: is the number of nickels you need, is the number of pennies you need. And you need to figure out the and values that would make the left-hand side work out to 23. And this one is pretty easy, you can just work it out yourself. You’d need four nickels and three pennies. So is four, is three. This kind of equation is called a linear equation . And that’s because when you plot this equation, everything is flat and smooth. There are no curves or holes. There isn’t a in the equation for example to make it curved. Linear equations are great because they’re much easier to work with than curved equations. Aside: Another solution for the above is 23 pennies. Or -4 nickels + 43 pennies. The point is you have two variables (x and y for nickels and pennies), and you are trying to combine them in different ways to hit one number . The trouble starts when you have two variables, and you need to combine them in different ways to hit two different numbers . That’s when Gaussian elimination comes in. In what world would you have to hit two different numbers? Does that seem outlandish? It’s actually very common! Read on for an example. Now let’s look at a different example. In the last one we were trying to make 23 cents with nickels and pennies. Here we have two foods. One is milk, the other is bread. They both have some macros in terms of carbs and protein: and now we want to figure out how many of each we need to eat to hit this target of 5 carbs and 7 protein. This is a very similar question to the one we just asked with nickels and pennies, except instead of one equation, we have two equations: Again we have an and a . Lets find their values. To solve these kinds of questions, we usually use Gaussian elimination . If you’ve never used Gaussian elimination, strap in. Step one is to rewrite this as a set of two equations: Now you subtract multiples of one equation from another to try to narrow down the value of one variable. Lets double that second equation: See how we have a and a now? Now we can add the two equations together to eliminate : We’re left with one equation and one variable. We can solve for : Aha, we know . Now we can plug that into one of the equations to find . We plug that in to one of the equations and find out that equals 1, and there we have answer: three milks, one bread, is what we need. This method is called Gaussian elimination, even though it was not discovered by Gauss. If you haven’t seen Gaussian elimination, congratulations, you learned a big idea! Gaussian elimination is something we will talk about more. It’s part of what makes linear algebra useful. We can also find the solution by drawing pictures. Let’s see how that works. Let’s plot one of these lines. First, we need to rewrite the equations in terms of . Reminder: first equation is for carbs, second for protein. x is number of milks, y is number of breads. Now let’s plot the graph for the first equation. Now, what does this line represent? It’s all the combinations of bread and milk that you can have to get exactly five carbs: So you can eat no milk and two-and-a-half breads, or two milks and one-and-a-half breads, or five milks and no bread, to get to exactly five carbs. All of those combinations would mean you have eaten exactly five carbs. You can pick any point that sits on this line to get to your goal of eating five carbs. Note: You can see the line goes into the negative as well. Technically, 5 breads and -5 milks will give you 5 carbs as well, but you can’t drink negative milks. For these examples, let’s assume only positive numbers for the variables. Now, let’s plot the other one. This is the same thing, but for protein. If you eat any of these combinations, you’ll have met the protein goal: You can pick a point that sits on the first line to meet the carb goal. You can pick a point that sits on the second line to meet the protein goal. But you need a point that sits on both lines to hit both goals. How would a point sit on both lines? Well, it would be where the lines cross . Since these are straight lines, the lines cross only once, which makes sense because there’s only a single milk and bread combo that would get you to exactly five grams of carbs and seven grams of protein. Now we plot the lines together, see where they intersect, and that’s our answer: Bam! We just found the solution using pictures. So that’s a quick intro to Gaussian elimination. But you don’t need linear algebra to do Gaussian elimination. This is a technique that has been around for 2,000 years. It was discovered in Asia, it was rediscovered in Europe, I think in the 1600s or something, and no one was really talking about “linear algebra”. This trick is just very useful. That’s the first big idea you learned. You can stop there if you want. You can practice doing this sort of elimination. It’s a very common and useful thing. What we just saw is called the “row picture”. Now I want to show you the column picture. I’m going to introduce a new idea, which is: instead of writing this series of equations, what if we write just one equation? Remember how we had one equation for the nickels and pennies question? What if we write one like that for food? Not a system of equations, just a single equation? What do you think that would look like? Something like this: It’s an equation where the coefficients aren’t numbers, they’re an “array” of numbers. The big idea here is: what if we have a linear equation, but instead of numbers, we have arrays of numbers? What if we treat , the way we treat a number? Can that actually work? If so, it is pretty revolutionary. Our whole lives we have been looking at just numbers, and now we’re saying, what if we look at arrays of numbers instead? Let’s see how it could work in our food example. What if the coefficients are an array of numbers? Well, this way of thinking is actually kind of intuitive. You might find it even more intuitive than the system of equations version. Each of these coefficients are called vectors . If you’re coming from computer science, you can kind of think of a vector as an array of numbers (i.e. the order matters). Lets see how we can use vectors to find a solution to the bread and milk question. Yeah, we can graph vectors . We can graph them either as a point, like I’ve done for the target vector here, or as an arrow, which is what I’ve done with the vector for bread and the vector for milk: Use the two numbers in the vector as the x and y coordinates. That is another big idea here: We always think of a set of coordinates giving a point, but you can think of vectors as an arrow instead of just a point . Now what we’re asking is how much milk and how much bread do we need, to get to that point? This is a pretty simple question. It’s simple enough that we can actually see it. Let me add some milks: And let me add a bread. Bingo bango, we’re at the point: Yeah, we literally add them on, visually. I personally find this more intuitive. I think the system of equations picture can confuse me sometimes, because the initial question was, “how much bread and how much milk should I eat?” The vector way, you see it in terms of breads and milks. The row way, you see it as one of the lines is the carbs, the other line is the protein, and the x and y axes are the amount of bread, which results in the same thing, but it’s a little more roundabout, a little more abstract. This one is very direct. We just saw that we can graph vectors too. Graphing it works differently from graphing the rows, but there is a graph we can make, and it works, which is pretty cool. What about the algebra way? Here is the equation again: Since we already know the answer, I’ll just plug that in: Now, the question is how does the left side equal the right side? The first question is how do you define this multiplication? Well, in linear algebra, it’s defined as, if you multiply a scalar by a vector, you just multiply it by each number in that vector: Now you are left with two vectors. How do you add two vectors? Well, in the linear algebra you just add the individual elements of each vector: And you end up with the answer. Congratulations, you’ve just had your first taste of linear algebra. It’s a pretty big step, right? Instead of numbers, we’re working with arrays of numbers. In future chapters, we will see why this is so powerful. That’s the first big concept of linear algebra: row picture vs column picture. Finally, I’ll just leave you with this last teaser, which is: how would you write these two equations in matrix notation? Like this: This is the exact same thing as before. You can write it as scalars times columns, as we had done before: or you can write it as a matrix times a vector, as above. Either one works. Matrices are a big part of linear algebra. But before we talk about matrices, we will talk about the dot product, which is coming up next. Check out Gilbert Strang’s lectures on linear algebra on YouTube . Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. P.S. Want more art? Check out my Instagram . Let’s say you have a bunch of nickels and pennies, and you want to know how many of each do you need to have 23 cents . You could write that as an equation that looks like this: is the number of nickels you need, is the number of pennies you need. And you need to figure out the and values that would make the left-hand side work out to 23. And this one is pretty easy, you can just work it out yourself. You’d need four nickels and three pennies. So is four, is three. This kind of equation is called a linear equation . And that’s because when you plot this equation, everything is flat and smooth. There are no curves or holes. There isn’t a in the equation for example to make it curved. Linear equations are great because they’re much easier to work with than curved equations. Aside: Another solution for the above is 23 pennies. Or -4 nickels + 43 pennies. The point is you have two variables (x and y for nickels and pennies), and you are trying to combine them in different ways to hit one number . The trouble starts when you have two variables, and you need to combine them in different ways to hit two different numbers . That’s when Gaussian elimination comes in. In what world would you have to hit two different numbers? Does that seem outlandish? It’s actually very common! Read on for an example. Food example Now let’s look at a different example. In the last one we were trying to make 23 cents with nickels and pennies. Here we have two foods. One is milk, the other is bread. They both have some macros in terms of carbs and protein: and now we want to figure out how many of each we need to eat to hit this target of 5 carbs and 7 protein. This is a very similar question to the one we just asked with nickels and pennies, except instead of one equation, we have two equations: Again we have an and a . Lets find their values. To solve these kinds of questions, we usually use Gaussian elimination . If you’ve never used Gaussian elimination, strap in. Gaussian elimination Step one is to rewrite this as a set of two equations: Now you subtract multiples of one equation from another to try to narrow down the value of one variable. Lets double that second equation: See how we have a and a now? Now we can add the two equations together to eliminate : We’re left with one equation and one variable. We can solve for : Aha, we know . Now we can plug that into one of the equations to find . We plug that in to one of the equations and find out that equals 1, and there we have answer: three milks, one bread, is what we need. This method is called Gaussian elimination, even though it was not discovered by Gauss. If you haven’t seen Gaussian elimination, congratulations, you learned a big idea! Gaussian elimination is something we will talk about more. It’s part of what makes linear algebra useful. We can also find the solution by drawing pictures. Let’s see how that works. Picture version Let’s plot one of these lines. First, we need to rewrite the equations in terms of . Reminder: first equation is for carbs, second for protein. x is number of milks, y is number of breads. Now let’s plot the graph for the first equation. Now, what does this line represent? It’s all the combinations of bread and milk that you can have to get exactly five carbs: So you can eat no milk and two-and-a-half breads, or two milks and one-and-a-half breads, or five milks and no bread, to get to exactly five carbs. All of those combinations would mean you have eaten exactly five carbs. You can pick any point that sits on this line to get to your goal of eating five carbs. Note: You can see the line goes into the negative as well. Technically, 5 breads and -5 milks will give you 5 carbs as well, but you can’t drink negative milks. For these examples, let’s assume only positive numbers for the variables. Now, let’s plot the other one. This is the same thing, but for protein. If you eat any of these combinations, you’ll have met the protein goal: You can pick a point that sits on the first line to meet the carb goal. You can pick a point that sits on the second line to meet the protein goal. But you need a point that sits on both lines to hit both goals. How would a point sit on both lines? Well, it would be where the lines cross . Since these are straight lines, the lines cross only once, which makes sense because there’s only a single milk and bread combo that would get you to exactly five grams of carbs and seven grams of protein. Now we plot the lines together, see where they intersect, and that’s our answer: Bam! We just found the solution using pictures. So that’s a quick intro to Gaussian elimination. But you don’t need linear algebra to do Gaussian elimination. This is a technique that has been around for 2,000 years. It was discovered in Asia, it was rediscovered in Europe, I think in the 1600s or something, and no one was really talking about “linear algebra”. This trick is just very useful. That’s the first big idea you learned. You can stop there if you want. You can practice doing this sort of elimination. It’s a very common and useful thing. The column picture What we just saw is called the “row picture”. Now I want to show you the column picture. I’m going to introduce a new idea, which is: instead of writing this series of equations, what if we write just one equation? Remember how we had one equation for the nickels and pennies question? What if we write one like that for food? Not a system of equations, just a single equation? What do you think that would look like? Something like this: It’s an equation where the coefficients aren’t numbers, they’re an “array” of numbers. The big idea here is: what if we have a linear equation, but instead of numbers, we have arrays of numbers? What if we treat , the way we treat a number? Can that actually work? If so, it is pretty revolutionary. Our whole lives we have been looking at just numbers, and now we’re saying, what if we look at arrays of numbers instead? Let’s see how it could work in our food example. What if the coefficients are an array of numbers? Well, this way of thinking is actually kind of intuitive. You might find it even more intuitive than the system of equations version. Each of these coefficients are called vectors . If you’re coming from computer science, you can kind of think of a vector as an array of numbers (i.e. the order matters). Lets see how we can use vectors to find a solution to the bread and milk question. Step one: graph the vectors. Yeah, we can graph vectors . We can graph them either as a point, like I’ve done for the target vector here, or as an arrow, which is what I’ve done with the vector for bread and the vector for milk: Use the two numbers in the vector as the x and y coordinates. That is another big idea here: We always think of a set of coordinates giving a point, but you can think of vectors as an arrow instead of just a point . Now what we’re asking is how much milk and how much bread do we need, to get to that point? This is a pretty simple question. It’s simple enough that we can actually see it. Let me add some milks: And let me add a bread. Bingo bango, we’re at the point: Yeah, we literally add them on, visually. I personally find this more intuitive. I think the system of equations picture can confuse me sometimes, because the initial question was, “how much bread and how much milk should I eat?” The vector way, you see it in terms of breads and milks. The row way, you see it as one of the lines is the carbs, the other line is the protein, and the x and y axes are the amount of bread, which results in the same thing, but it’s a little more roundabout, a little more abstract. This one is very direct. The algebra way We just saw that we can graph vectors too. Graphing it works differently from graphing the rows, but there is a graph we can make, and it works, which is pretty cool. What about the algebra way? Here is the equation again: Since we already know the answer, I’ll just plug that in: Now, the question is how does the left side equal the right side? The first question is how do you define this multiplication? Well, in linear algebra, it’s defined as, if you multiply a scalar by a vector, you just multiply it by each number in that vector: Now you are left with two vectors. How do you add two vectors? Well, in the linear algebra you just add the individual elements of each vector: And you end up with the answer. Congratulations, you’ve just had your first taste of linear algebra. It’s a pretty big step, right? Instead of numbers, we’re working with arrays of numbers. In future chapters, we will see why this is so powerful. That’s the first big concept of linear algebra: row picture vs column picture. Finally, I’ll just leave you with this last teaser, which is: how would you write these two equations in matrix notation? Like this: This is the exact same thing as before. You can write it as scalars times columns, as we had done before: or you can write it as a matrix times a vector, as above. Either one works. Matrices are a big part of linear algebra. But before we talk about matrices, we will talk about the dot product, which is coming up next. Additional reading Check out Gilbert Strang’s lectures on linear algebra on YouTube .

Data Analysis Tutorial Programming

Shayon Mukherjee 1 months ago

An MVCC-like columnar table on S3 with constant-time deletes

Parquet is excellent for analytical workloads. Columnar layout, aggressive compression, predicate pushdown, but deletes require rewriting entire files. Systems like Apache Iceberg and Delta Lake solve this by adding metadata layers that track delete files separately from data files. But what if, for fun, we built something (arguably) simpler? S3 now has conditional writes (If-Match, If-None-Match) that enable atomic operations without external coordination. Let’s explore how we might build a columnar table format on S3 that gets most of Parquet’s benefits while supporting constant-time deletes.

Data Analysis Cloud Database

Rik Huijzer 2 months ago

Companies are Lying About AI Layoffs

Companies are saying that the economy and AI are causing the large amount of tech layoffs, but Vanessa Wingårdh looked at the H-1B visa data and noticed that the companies are lying. She argues that the American employees are being replaced by cheaper H-1B visa workers. WSJ also reported on this due to some senators speaking up. As proof she uses the H-1B data from the U.S. Citizenship and Immigration Services. When filtering by fiscal years 2023, 2024 and 2025, the number of Beneficiaries Approved for some of the big corporations as of 30 June 2025 are shown in the second column below. I al...

Career Business Data Analysis

DYNOMIGHT 2 months ago

Dear PendingKetchup

PendingKetchup comments on my recent post on what it means for something to be heritable : The article seems pretty good at math and thinking through unusual implications, but my armchair Substack eugenics alarm that I keep in the back of my brain is beeping. Saying that variance was “invented for the purpose of defining heritability” is technically correct, but that might not be the best kind of correct in this case, because it was invented by the founder of the University of Cambridge Eugenics Society who had decided, presumably to support that project, that he wanted to define something called “heritability”. His particular formula for heritability is presented in the article as if it has odd traits but is obviously basically a sound thing to want to calculate, despite the purpose it was designed for. The vigorous “educational attainment is 40% heritable, well OK maybe not but it’s a lot heritable, stop quibbling” hand waving sounds like a person who wants to show but can’t support a large figure. And that framing of education, as something “attained” by people, rather than something afforded to or invested in them, is almost completely backwards at least through college. The various examples about evil despots and unstoppable crabs highlight how heritability can look large or small independent of more straightforward biologically-mechanistic effects of DNA. But they still give the impression that those are the unusual or exceptional cases. In reality, there are in fact a lot of evil crabs, doing things like systematically carting away resources from Black children’s* schools, and then throwing them in jail. We should expect evil-crab-based explanations of differences between people to be the predominant ones. *Not to say that being Black “is genetic”. Things from accent to how you style your hair to how you dress to what country you happen to be standing in all contribute to racial judgements used for racism. But “heritability” may not be the right tool to disentangle those effects. Dear PendingKetchup, Thanks for complimenting my math (♡), for reading all the way to the evil crabs, and for not explicitly calling me a racist or eugenicist. I also appreciate that you chose sincerity over boring sarcasm and that you painted such a vibrant picture of what you were thinking while reading my post. I hope you won’t mind if I respond in the same spirit. To start, I’d like to admit something. When I wrote that post, I suspected some people might have reactions similar to yours. I don’t like that. I prefer positive feedback! But I’ve basically decided to just let reactions like yours happen, because I don’t know how to avoid them without compromising on other core goals. It sounds like my post gave you a weird feeling. Would it be fair to describe it as a feeling that I’m not being totally upfront about what I really think about race / history / intelligence / biological determinism / the ideal organization of society? Because if so, you’re right. It’s not supposed to be a secret, but it’s true. Why? Well, you may doubt this, but when I wrote that post, my goal was that people who read it would come away with a better understanding of the meaning of heritability and how weird it is. That’s it. Do I have some deeper and darker motivations? Probably. If I probe my subconscious, I find traces of various embarrassing things like “draw attention to myself” or “make people think I am smart” or “after I die, live forever in the world of ideas through my amazing invention of blue-eye-seeking / human-growth-hormone-injecting crabs.” What I don’t find are any goals related to eugenics, Ronald Fisher, the heritability of educational attainment, if “educational attainment” is good terminology, racism, oppression, schools, the justice system, or how society should be organized. These were all non-goals for basically two reasons: My views on those issues aren’t very interesting or notable. I didn’t think anyone would (or should) care about them. Surely, there is some place in the world for things that just try to explain what heritability really means? If that’s what’s promised, then it seems weird to drop in a surprise morality / politics lecture. At the same time, let me concede something else. The weird feeling you got as you read my post might be grounded in statistical truth. That is, it might be true that many people who blog about things like heritability have social views you wouldn’t like. And it might be true that some of them pretend at truth-seeking but are mostly just charlatans out to promote those unliked-by-you social views. You’re dead wrong to think that’s what I’m doing. All your theories of things I’m trying to suggest or imply are unequivocally false. But given the statistical realities, I guess I can’t blame you too much for having your suspicions. So you might ask—if my goal is just to explain heritability, why not make that explicit? Why not have a disclaimer that says, “OK I understand that heritability is fraught and blah blah blah, but I just want to focus on the technical meaning because…”? One reason is that I think that’s boring and condescending. I don’t think people need me to tell them that heritability is fraught. You clearly did not need me to tell you that. Also, I don’t think such disclaimers make you look neutral. Everyone knows that people with certain social views (likely similar to yours) are more likely to give such disclaimers. And they apply the same style of statistical reasoning you used to conclude I might be a eugenicist. I don’t want people who disagree with those social views to think they can’t trust me. Paradoxically, such disclaimers often seem to invite more objections from people who share the views they’re correlated with, too. Perhaps that’s because the more signals we get that someone is on “our” side, the more we tend to notice ideological violations. (I’d refer here to the narcissism of small differences , though I worry you may find that reference objectionable.) If you want to focus on the facts, the best strategy seems to be serene and spiky: to demonstrate by your actions that you are on no one’s side, that you don’t care about being on anyone’s side, and that your only loyalty is to readers who want to understand the facts and make up their own damned mind about everything else. I’m not offended by your comment. I do think it’s a little strange that you’d publicly suggest someone might be a eugenicist on the basis of such limited evidence. But no one is forcing me to write things and put them on the internet. The reason I’m writing to you is that you were polite and civil and seem well-intentioned. So I wanted you to know that your world model is inaccurate. You seem to think that because my post did not explicitly support your social views, it must have been written with the goal of undermining those views. And that is wrong. The truth is, I wrote that post without supporting your (or any) social views because I think mixing up facts and social views is bad. Partly, that’s just an aesthetic preference. But if I’m being fully upfront, I also think it’s bad in the consequentialist sense that it makes the world a worse place. Why do I think this? Well, recall that I pointed out that if there were crabs that injected blue-eyed babies with human growth hormone, that would increase the heritability of height. You suggest I had sinister motives for giving this example, as if I was trying to conceal the corollary that if the environment provided more resources to people with certain genes (e.g. skin color) that could increase the heritability of other things (e.g. educational attainment). Do you really think you’re the only reader to notice that corollary? The degree to which things are “heritable” depends on the nature of society. This is a fact. It’s a fact that many people are not aware of. It’s also a fact that—I guess—fits pretty well with your social views. I wanted people to understand that. Not out of loyalty to your social views, but because it is true. It seems that you’re annoyed that I didn’t phrase all my examples in terms of culture war. I could have done that. But I didn’t, because I think my examples are easier to understand, and because the degree to which changing society might change the heritability of some trait is a contentious empirical question. But OK. Imagine I had done that. And imagine all the examples were perfectly aligned with your social views. Do you think that would have made the post more or less effective in convincing people that the fact we’re talking about is true? I think the answer is: Far less effective. I’ll leave you with two questions: Question 1: Do you care about the facts? Do you believe the facts are on your side? Question 2: Did you really think I wrote that post with with the goal of promoting eugenics? If you really did think that, then great! I imagine you’ll be interested to learn that you were incorrect. But just as you had an alarm beeping in your head as you read my post, I had one beeping in my head as I read your comment. My alarm was that you were playing a bit of a game. It’s not that you really think I wanted to promote eugenics, but rather that you’re trying to enforce a norm that everyone must give constant screaming support to your social views and anyone who’s even slightly ambiguous should be ostracized. Of course, this might be a false alarm! But if that is what you’re doing, I have to tell you: I think that’s a dirty trick, and a perfect example of why mixing facts and social views is bad. You may disagree with all my motivations. That’s fine. ( I won’t assume that means you are a eugenicist.) All I ask is that you disapprove accurately. xox dynomight My views on those issues aren’t very interesting or notable. I didn’t think anyone would (or should) care about them. Surely, there is some place in the world for things that just try to explain what heritability really means? If that’s what’s promised, then it seems weird to drop in a surprise morality / politics lecture.

Statistics Data Analysis

Entropic Thoughts 2 months ago

Wiggling Into Correlation

Statistics Data Visualization Data Analysis

fasterthanli.me 3 months ago

The science of loudness

My watch has a “Noise” app: it shows d B , for decibels. Your browser does not support the video tag. My amp has a volume knob, which also shows decibels, although.. negative ones, this time. Your browser does not support the video tag. And finally, my video editing software has a ton of meters — which are all in decibel or decibel-adjacent units. Your browser does not support the video tag. How do all these decibels fit together?

Data Analysis Music

emiruz 3 months ago

A short statistical reasoning test

Here are a few practical questions of my own invention which are easy to comprehend but very difficult to solve without statistical reasoning competence. They are provided in order of difficulty. The answers are at the end. If you find errors or have elegant alternative solutions, please email me (address in bio)! QUESTIONS 1. Sorting fractions under uncertainty You are given the number of trials and successes for a set of items, and you are asked to sort them by the fraction #successes / #trials.

Statistics Data Analysis

DYNOMIGHT 3 months ago

Futarchy’s fundamental flaw — the market — the blog post

Here’s our story so far: Markets are a good way to know what people really think. When India and Pakistan started firing missiles at each other on May 7, I was concerned, what with them both having nuclear weapons. But then I looked at world market prices: See how it crashes on May 7? Me neither. I found that reassuring. But we care about lots of stuff that isn’t always reflected in stock prices, e.g. the outcomes of elections or drug trials. So why not create markets for those, too? If you create contracts that pay out $1 only if some drug trial succeeds, then the prices will reflect what people “really” think. In fact, why don’t we use markets to make decisions? Say you’ve invented two new drugs, but only have enough money to run one trial. Why don’t you create markets for both drugs, then run the trial on the drug that gets a higher price? Contracts for the “winning” drug are resolved based on the trial, while contracts in the other market are cancelled so everyone gets their money back. That’s the idea of Futarchy , which Robin Hanson proposed in 2007. Why don’t we? Well, maybe it won’t work. In 2022, I wrote a post arguing that when you cancel one of the markets, you screw up the incentives for how people should bid, meaning prices won’t reflect the causal impact of different choices. I suggested prices reflect “correlation” rather than causation, for basically the same reason this happens with observational statistics. This post, it was magnificent. It didn’t convince anyone. Years went by. I spent a lot of time reading Bourdieu and worrying about why I buy certain kinds of beer. Gradually I discovered that essentially the same point about futarchy had been made earlier by, e.g., Anders_H in 2015, abramdemski in 2017, and Luzka in 2021. In early 2025, I went to a conference and got into a bunch of (friendly) debates about this. I was astonished to find that verbally repeating the arguments from my post did not convince anyone. I even immodestly asked one person to read my post on the spot. (Bloggers: Do not do that.) That sort of worked. So, I decided to try again. I wrote another post called ” Futarky’s Futarchy’s fundamental flaw” . It made the same argument with more aggression, with clearer examples, and with a new impossibility theorem that showed there doesn’t even exist any alternate payout function that would incentivize people to bid according to their causal beliefs. That post… also didn’t convince anyone. In the discussion on LessWrong , many of my comments are upvoted for quality but downvoted for accuracy, which I think means, “nice try champ; have a head pat; nah.” Robin Hanson wrote a response , albeit without outward evidence of reading beyond the first paragraph. Even the people who agreed with me often seemed to interpret me as arguing that futarchy satisfies evidential decision theory rather than causal decision theory . Which was weird, given that I never mentioned either of those, don’t accept the premise the futarchy satisfies either of them, and don’t find the distinction helpful in this context. In my darkest moments, I started to wonder if I might fail to achieve worldwide consensus that futarchy doesn’t estimate causal effects. I figured I’d wait a few years and then launch another salvo. But then, legendary human Bolton Bailey decided to stop theorizing and take one of my thought experiments and turn it into an actual experiment. Thus, Futarchy’s fundamental flaw — the market was born. (You are now reading a blog post about that market.) I gave a thought experiment where there are two coins and the market is trying to pick the one that’s more likely to land heads. For one coin, the bias is known, while for the other coin there’s uncertainty. I claimed futarchy would select the worse / wrong coin, due to this extra uncertainty. Bolton formalized this as follows: There are two markets, one for coin A and one for coin B. Coin A is a normal coin that lands heads 60% of the time. Coin B is a trick coin that either always lands heads or always lands tails, we just don’t know which. There’s a 59% it’s an always-heads coin. Twenty-four hours before markets close, the true nature of coin B is revealed. After the markets closes, whichever coin has a higher price is flipped and contracts pay out $1 for heads and $0 for tails. The other market is cancelled so everyone gets their money back. Get that? Everyone knows that there’s a 60% chance coin A will land heads and a 59% chance coin B will land heads. But for coin A, that represents true “aleatoric” uncertainty, while for coin B that represents “epistemic” uncertainty due to a lack of knowledge. (See Bayes is not a phase for more on “aleatoric” vs. “epistemic” uncertainty.) Bolton created that market independently. At the time, we’d never communicated about this or anything else. To this day, I have no idea what he thinks about my argument or what he expected to happen. In the forum for the market, there was a lot of debate about “whalebait”. Here’s the concern: Say you’ve bought a lot of contracts for coin B, but it emerges that coin B is always-tails. If you have a lot of money, then you might go in at the last second and buy a ton of contracts on coin A to try to force the market price above coin B, so the coin B market is cancelled and you get your money back. The conversation seemed to converge towards the idea that this was whalebait. Though notice that if you’re buying contracts for coin A at any price above $0.60, you’re basically giving away free money. It could still work, but it’s dangerous and everyone else has an incentive to stop you. If I was betting in this market, I’d think that this was at least unlikely . Bolton posted about the market. When I first saw the rules, I thought it wasn’t a valid test of my theory and wasted a huge amount of Bolton’s time trying to propose other experiments that would “fix” it. Bolton was very patient, but I eventually realized that it was completely fine and there was nothing to fix. At the time, this is what the prices looked like: That is, at the time, both coins were priced at $0.60, which is not what I had predicted. Nevertheless, I publicly agreed that this was a valid test of my claims. I think this is a great test and look forward to seeing the results. Let me reiterate why I thought the markets were wrong and coin B deserved a higher price. There’s a 59% chance coin B would turns out to be all-heads. If that happened, then (absent whales being baited) I thought the coin B market would activate, so contracts are worth $1. So thats 59% × $1 = $0.59 of value. But if coin B turns out to be all-tails, I thought there is a good chance prices for coin B would drop below coin A, so the market is cancelled and you get your money back. So I thought a contract had to be worth more than $0.59. If you buy a contract for coin B for $0.70, then I think that’s worth Surely isn’t that low. So surely this is worth more than $0.59. More generally, say you buy a YES contract for coin B for $M. Then that contract would be worth It’s not hard to show that the breakeven price is Even if you thought was only 50%, then the breakeven price would still be $0.7421. Within a few hours, a few people bought contracts on coin B, driving up the price. Then, Quroe proposed creating derivative markets. In theory, if there was a market asking if coin A was going to resolve YES, NO, or N/A, supposedly people could arbitrage their bets accordingly and make this market calibrated. Same for a similar market on coin B. Thus, Futarchy’s Fundamental Fix - Coin A and Futarchy’s Fundamental Fix - Coin B came to be. These were markets in which people could bid on the probability that each coin would resolve YES, meaning the coin was flipped and landed heads, NO, meaning the coin was flipped and landed tails, or N/A, meaning the market was cancelled. Honestly, I didn’t understand this. I saw no reason that these derivative markets would make people bid their true beliefs. If they did, then my whole theory that markets reflect correlation rather than causation would be invalidated. Prices for coin B went up and down, but mostly up. Eventually, a few people created large limit orders, which caused things to stabilize. Here was the derivative market for coin A. And here it was market for coin B. During this period, not a whole hell of a lot happened. This brings us up to the moment of truth, when the true nature of coin B was to be revealed. At this point, coin B was at $0.90, even though everyone knows it only has a 59% chance of being heads. The nature of the coin was revealed. To show this was fair, Bolton did this by asking a bot to publicly generate a random number. Thus, coin B was determined to be always-heads. There were still 24 hours left to bid. At this point, a contract for coin B was guaranteed to pay out $1. The market quickly jumped to $1. I was right. Everyone knew coin A had a higher chance of being heads than coin B, but everyone bid the price of coin B way above coin A anyway. In the previous math box, we saw that the breakeven price should satisfy If you invert this and plug in M=$0.90, then you get I’ll now open the floor for questions. Isn’t this market unrealistic? Yes, but that’s kind of the point. I created the thought experiment because I wanted to make the problem maximally obvious, because it’s subtle and everyone is determined to deny that it exists. Isn’t this just a weird probability thing? Why does this show futarchy is flawed? The fact that this is possible is concerning. If this can happen, then futarchy does not work in general . If you want to claim that futarchy works, then you need to spell out exactly what extra assumptions you’re adding to guarantee that this kind of thing won’t happen. But prices did reflect causality when the market closed! Doesn’t that mean this isn’t a valid test? No. That’s just a quirk of the implementation. You can easily create situations that would have the same issue all the way through market close. Here’s one way you could do that: On average, this market will run for 30 days. (The length follows a geometric distribution ). Half the time, the market will close without the nature of coin B being revealed. Even when that happens, I claim the price for coin B will still be above coin A. If futarchy is flawed, shouldn’t you be able to show that without this weird step of “revealing” coin B? Yes. You should be able to do that, and I think you can. Here’s one way: First, have users generate public keys by running this command: Second, they should post the contents of the when asking for their bit. For example: Third, whoever is running the market should save that key as , pick a pit, and encrypt it like this: Users can then decrypt like this: Or you could use email… I think this market captures a dynamic that’s present in basically any use of futarchy: You have some information, but you know other information is out there. I claim that this market—will be weird. Say it just opened. If you didn’t get a bit, then as far as you know, the bias for coin B could be anywhere between 49% and 69%, with a mean of 59%. If you did get a bit, then it turns out that the posterior mean is 58.5% if you got a and 59.5% if you got a . So either way, your best guess is very close to 59%. However, the information for the true bias of coin B is out there! Surely coin B is more likely to end up with a higher price in situations where there are lots of bits. This means you should bid at least a little higher than your true belief, for the same reason as the main experiment—the market activating is correlated with the true bias of coin B. Of course, after the markets open, people will see each other’s bids and… something will happen. Initially, I think prices will be strongly biased for the above reasons. But as you get closer to market close, there’s less time for information to spread. If you are the last person to trade, and you know you’re the last person to trade, then you should do so based on your true beliefs. Except, everyone knows that there’s less time for information to spread. So while you are waiting till the last minute to reveal your true beliefs, everyone else will do the same thing. So maybe people sort of rush in at the last second? (It would be easier to think about this if implemented with batched auctions rather than a real-time market.) Anyway, while the game theory is vexing, I think there’s a mix of (1) people bidding higher than their true beliefs due to correlations between the final price and the true bias of coin B and (2) people “racing” to make the final bid before the markets close. Both of these seem in conflict with the idea of prediction markets making people share information and measuring collective beliefs. Why do you hate futarchy? I like futarchy. I think society doesn’t make decisions very well, and I think we should give much more attention to new ideas like futarchy that might help us do better. I just think we should be aware of its imperfections and consider variants (e.g. commiting to randomization ) that would resolve them. If I claim futarchy does reflect causal effects, and I reject this experiment as invalid, should I specify what restrictions I want to place on “valid” experiments (and thus make explicit the assumptions under which I claim futarchy works) since otherwise my claims are unfalsifiable? Markets are a good way to know what people really think. When India and Pakistan started firing missiles at each other on May 7, I was concerned, what with them both having nuclear weapons. But then I looked at world market prices: See how it crashes on May 7? Me neither. I found that reassuring. But we care about lots of stuff that isn’t always reflected in stock prices, e.g. the outcomes of elections or drug trials. So why not create markets for those, too? If you create contracts that pay out $1 only if some drug trial succeeds, then the prices will reflect what people “really” think. In fact, why don’t we use markets to make decisions? Say you’ve invented two new drugs, but only have enough money to run one trial. Why don’t you create markets for both drugs, then run the trial on the drug that gets a higher price? Contracts for the “winning” drug are resolved based on the trial, while contracts in the other market are cancelled so everyone gets their money back. That’s the idea of Futarchy , which Robin Hanson proposed in 2007. Why don’t we? Well, maybe it won’t work. In 2022, I wrote a post arguing that when you cancel one of the markets, you screw up the incentives for how people should bid, meaning prices won’t reflect the causal impact of different choices. I suggested prices reflect “correlation” rather than causation, for basically the same reason this happens with observational statistics. This post, it was magnificent. It didn’t convince anyone. Years went by. I spent a lot of time reading Bourdieu and worrying about why I buy certain kinds of beer. Gradually I discovered that essentially the same point about futarchy had been made earlier by, e.g., Anders_H in 2015, abramdemski in 2017, and Luzka in 2021. In early 2025, I went to a conference and got into a bunch of (friendly) debates about this. I was astonished to find that verbally repeating the arguments from my post did not convince anyone. I even immodestly asked one person to read my post on the spot. (Bloggers: Do not do that.) That sort of worked. So, I decided to try again. I wrote another post called ” Futarky’s Futarchy’s fundamental flaw” . It made the same argument with more aggression, with clearer examples, and with a new impossibility theorem that showed there doesn’t even exist any alternate payout function that would incentivize people to bid according to their causal beliefs. There are two markets, one for coin A and one for coin B. Coin A is a normal coin that lands heads 60% of the time. Coin B is a trick coin that either always lands heads or always lands tails, we just don’t know which. There’s a 59% it’s an always-heads coin. Twenty-four hours before markets close, the true nature of coin B is revealed. After the markets closes, whichever coin has a higher price is flipped and contracts pay out $1 for heads and $0 for tails. The other market is cancelled so everyone gets their money back. Let coin A be heads with probability 60%. This is public information. Let coin B be an ALWAYS HEADS coin with probability 59% and ALWAYS TAILS coin with probability 41%. This is a secret. Every day, generate a random integer between 1 and 30. If it’s 1, immediately resolve the markets. It it’s 2, reveal the nature of coin B. If it’s between 3 and 30, do nothing. Let coin A be heads with probability 60%. This is public information. Sample 20 random bits, e.g. . Let coin B be heads with probability (49+N)% where N is the number of bits. do not reveal these bits publicly. Secretly send these bits to the first 20 people who ask.

Data Analysis Business Statistics

Robert Alexander's Tech Blog 3 months ago

The most popular email providers

I downloaded the Tranco list of top domain names and ran DNS MX lookups to detect mail servers. I used the top 100,000 domain names, which should be a reasonably sample of large and popular sites. I found that 72% of the scanned domains had a mail server configured (MX record was present). Here are the most popular email hosting platforms, as of August 2025. I’ve aggregated by the second-level domain name of the MX records, then truncated to providers supporting at least 25 of the scanned domain names. Google and Microsoft grab the top spots, acting as email provider for about 60% of the email-enabled domains analyzed. A diverse group of hosting providers, registrars, email and security products form the long-tail. The scanning code is available on GitHub

Data Analysis

DYNOMIGHT 3 months ago

Heritability puzzlers

The heritability wars have been a-raging. Watching these, I couldn’t help but notice that there’s near-universal confusion about what “heritable” means. Partly, that’s because it’s a subtle concept. But it also seems relevant that almost all explanations of heritability are very, very confusing. For example, here’s Wikipedia’s definition : Any particular phenotype can be modeled as the sum of genetic and environmental effects: Phenotype ( P ) = Genotype ( G ) + Environment ( E ). Likewise the phenotypic variance in the trait – Var ( P ) – is the sum of effects as follows: Var( P ) = Var( G ) + Var( E ) + 2 Cov( G , E ). In a planned experiment Cov( G , E ) can be controlled and held at 0. In this case, heritability, H ², is defined as H ² = Var( G ) / Var( P ) H ² is the broad-sense heritability. Do you find that helpful? I hope not, because it’s a mishmash of undefined terminology, unnecessary equations, and borderline-false statements. If you’re in the mood for a mini-polemic: Reading this almost does more harm than good. While the final definition is correct, it never even attempts to explain what G and P are, it gives an incorrect condition for when the definition applies, and instead mostly devotes itself to an unnecessary digression about environmental effects. The rest of the page doesn’t get much better. Despite being 6700 words long, I think it would be impossible to understand heritability simply by reading it. Meanwhile, some people argue that heritability is meaningless for human traits like intelligence or income or personality. They claim that those traits are the product of complex interactions between genes and the environment and it’s impossible to disentangle the two. These arguments have always struck me as “suspiciously convenient”. I figured that the people making them couldn’t cope with the hard reality that genes are very important and have an enormous influence on what we are. But I increasingly feel that the skeptics have a point. While I think it’s a fact that most human traits are substantially heritable, it’s also true the technical definition of heritability is really weird, and simply does not mean what most people think it means. In this post, I will explain exactly what heritability is, while assuming no background. I will skip everything that can be skipped but—unlike most explanations—I will not skip things that can’t be skipped. Then I’ll go through a series of puzzles demonstrating just how strange heritability is. How tall you are depends on your genes, but also on what you eat, what diseases you got as a child, and how much gravity there is on your home planet. And all those things interact. How do you take all that complexity and reduce it to a single number, like “80% heritable”? The short answer is: Statistical brute force. The long answer is: Read the rest of this post. It turns out that the hard part of heritability isn’t heritability. Lurking in the background is a slippery concept known as a genotypic value . Discussions of heritability often skim past these. Quite possibly, just looking at the words “genotypic value”, you are thinking about skimming ahead right now. Resist that urge! Genotypic values are the core concept, and without them you cannot possibly understand heritability. For any trait, your genotypic value is the “typical” outcome if someone with your DNA were raised in many different random environments. In principle, if you wanted to know your genotypic height, you’d need to do this: Since you can’t / shouldn’t do that, you’ll never know your genotypic height. But that’s how it’s defined in principle—the average height someone with your DNA would grow to in a random environment. If you got lots of food and medical care as a child, your actual height is probably above your genotypic height. If you suffered from rickets, your actual height is probably lower than your genotypic height. Comfortable with genotypic values? OK. Then (broad-sense) heritability is easy. It’s the ratio Here, is the variance , basically just how much things vary in the population. Among all adults worldwide, is around 50 cm². (Incidentally, did you know that variance was invented for the purpose of defining heritability?) Meanwhile, is how much genotypic height varies in the population. That might seem hopeless to estimate, given that we don’t know anyone’s genotypic height. But it turns out that we can still estimate the variance using, e.g., pairs of adopted twins, and it’s thought to be around 40 cm². If we use those numbers, the heritability of height would be People often convert this to a percentage and say “height is 80% heritable”. I’m not sure I like that, since it masks heritability’s true nature as a ratio. But everyone does it, so I’ll do it too. People who really want to be intimidating might also say, “genes explain 80% of the variance in height”. Of course, basically the same definition works for any trait, like weight or income or fondness for pseudonymous existential angst science blogs. But instead of replacing “height” with “trait”, biologists have invented the ultra-fancy word “phenotype” and write The word “phenotype” suggests some magical concept that would take years of study to understand. But don’t be intimidated. It just means the actual observed value of some trait(s). You can measure your phenotypic height with a tape measure. Let me make two points before moving on. First, this definition of heritability assumes nothing. We are not assuming that genes are independent of the environment or that “genotypic effects” combine linearly with “environmental effects”. We are not assuming that genes are in Hardy-Weinberg equilibrium , whatever that is. No. I didn’t talk about that stuff because I don’t need to. There are no hidden assumptions. The above definition always works. Second, many normal English words have parallel technical meanings, such as “field” , “insulator” , “phase” , “measure” , “tree” , or “stack” . Those are all nice, because they’re evocative and it’s almost always clear from context which meaning is intended. But sometimes, scientists redefine existing words to mean something technical that overlaps but also contradicts the normal meaning, as in “salt” , “glass” , “normal” , “berry” , or “nut” . These all cause confusion, but “heritability” must be the most egregious case in all of science. Before you ever heard the technical definition of heritability, you surely had some fuzzy concept in your mind. Personally, I thought of heritability as meaning how many “points” you get from genes versus the environment. If charisma was 60% heritable, I pictured each person has having 10 total “charisma points”, 6 of which come from genes, and 4 from the environment: If you take nothing else from this post, please remember that the technical definition of heritability does not work like that . You might hope that if we add some plausible assumptions, the above ratio-based definition would simplify into something nice and natural, that aligns with what “heritability” means in normal English. But that does not happen. If that’s confusing, well, it’s not my fault. Not sure what’s happening here, but it seems relevant. So “heritability” is just the ratio of genotypic and phenotypic variance. Is that so bad? I think… maybe? How heritable is eye color? Close to 100%. This seems obvious, but let’s justify it using our definition that . Well, people have the same eye color, no matter what environment they are raised in. That means that genotypic eye color and phenotypic eye color are the same thing. So they have the same variance, and the ratio is 1. Nothing tricky here. How heritable is speaking Turkish? Close to 0%. Your native language is determined by your environment. If you grow up in a family that speaks Turkish, you speak Turkish. Genes don’t matter. Of course, there are lots of genes that are correlated with speaking Turkish, since Turks are not, genetically speaking, a random sample of the global population. But that doesn’t matter, because if you put Turkish babies in Korean households, they speak Korean. Genotypic values are defined by what happens in a random environment, which breaks the correlation between speaking Turkish and having Turkish genes. Since 1.1% of humans speak Turkish, the genotypic value for speaking Turkish is around 0.011 for everyone, no matter their DNA. Since that’s basically constant, the genotypic variance is near zero, and heritability is near zero. How heritable is speaking English? Perhaps 30%. Probably somewhere between 10% and 50%. Definitely more than zero. That’s right. Turkish isn’t heritable but English is. Yes it is . If you ask an LLM, it will tell you that the heritability of English is zero. But the LLM is wrong and I am right. Why? Let me first acknowledge that Turkish is a little bit heritable. For one thing, some people have genes that make them non-verbal. And there’s surely some genetic basis for being a crazy polyglot that learns many languages for fun. But speaking Turkish as a second language is quite rare , meaning that the genotypic value of speaking Turkish is close to 0.011 for almost everyone. English is different. While only 1 in 20 people in the world speak English as a first language, 1 in 7 learn it as a second language. And who does that? Educated people. Some argue the heritability of educational attainment is much lower. I’d like to avoid debating the exact numbers, but note that these lower numbers are usually estimates of “narrow-sense” heritability rather than “broad-sense” heritability as we’re talking about. So they should be lower. (I’ll explain the difference later.) It’s entirely possible that broad-sense heritability is lower than 40%, but everyone agrees it’s much larger than zero. So the heritability of English is surely much larger than zero, too. Say there’s an island where genes have no impact on height. How heritable is height among people on this island? There’s nothing tricky here. Say there’s an island where genes entirely determine height. How heritable is height? Again, nothing tricky. Say there’s an island where neither genes nor the environment influence height and everyone is exactly 165 cm tall. How heritable is height? It’s undefined. In this case, everyone has exactly the same phenotypic and genotypic height, namely 165 cm. Since those are both constant, their variance is zero and heritability is zero divided by zero. That’s meaningless. Say there’s an island where some people have genes that predispose them to be taller than others. But the island is ruled by a cruel despot who denies food to children with taller genes, so that on average, everyone is 165 ± 5 cm tall. How heritable is height? On this island, everyone has a genotypic height of 165 cm. So genotypic variance is zero, but phenotypic variance is positive, due to the ± 5 cm random variation. So heritability is zero divided by some positive number. Say there’s an island where some people have genes that predispose them to be tall and some have genes that predispose them to be short. But, the same genes that make you tall also make you semi-starve your children, so in practice everyone is exactly 165 cm tall. How heritable is height? ∞%. Not 100%, mind you, infinitely heritable. To see why, note that if babies with short/tall genes are adopted by parents with short/tall genes, there are four possible cases. If a baby with short genes is adopted into random families, they will be shorter on average than if a baby with tall genes. So genotypic height varies. However, in reality, everyone is the same height, so phenotypic height is constant. So genotypic variance is positive while phenotypic variance is zero. Thus, heritability is some positive number divided by zero, i.e. infinity. (Are you worried that humans are “diploid”, with two genes (alleles) at each locus, one from each biological parent? Or that when there are multiple parents, they all tend to have thoughts on the merits of semi-starvation? If so, please pretend people on this island reproduce asexually. Or, if you like, pretend that there’s strong assortative mating so that everyone either has all-short or all-tall genes and only breeds with similar people. Also, don’t fight the hypothetical.) Say there are two islands. They all live the same way and have the same gene pool, except people on island A have some gene that makes them grow to be 150 ± 5 cm tall, while on island B they have a gene that makes them grow to be 160 ± 5 cm tall. How heritable is height? It’s 0% for island A and 0% for island B, and 50% for the two islands together. Why? Well on island A, everyone has the same genotypic height, namely 150 cm. Since that’s constant, genotypic variance is zero. Meanwhile, phenotypic height varies a bit, so phenotypic variance is positive. Thus, heritability is zero. For similar reasons, heritability is zero on island B. But if you put the two islands together, half of people have a genotypic height of 150 cm and half have a genotypic height of 160 cm, so suddenly (via math) genotypic variance is 25 cm². There’s some extra random variation so (via more math) phenotypic variance turns out to be 50 cm². So heritability is 25 / 50 = 50%. If you combine the populations, then genotypic variance is Meanwhile phenotypic variance is Say there’s an island where neither genes nor the environment influence height. Except, some people have a gene that makes them inject their babies with human growth hormone, which makes them 5 cm taller. How heritable is height? True, people with that gene will tend be taller. And the gene is causing them to be taller. But if babies are adopted into random families, it’s the genes of the parents that determine if they get injected or not. So everyone has the same genotypic height, genotypic variance is zero, and heritability is zero. Suppose there’s an island where neither genes nor the environment influence height. Except, some people have a gene that makes them, as babies, talk their parents into injecting them with human growth hormone. The babies are very persuasive. How heritable is height? We’re back to 100%. The difference with the previous scenario is that now babies with that gene get injected with human growth hormone no matter who their parents are. Since nothing else influences height, genotype and phenotype are the same, have the same variance, and heritability is 100%. Suppose there’s an island where neither genes nor the environment influence height. Except, there are crabs that seek out blue-eyed babies and inject them with human growth hormone. The crabs, they are unstoppable. How heritable is height? Again, 100%. Babies with DNA for blue eyes get injected. Babies without DNA for blue eyes don’t. Since nothing else influences height, genotype and phenotype are the same and heritability is 100%. Note that if the crabs were seeking out parents with blue eyes and then injecting their babies, then height would be 0% heritable. It doesn’t matter that human growth hormone is weird thing that’s coming from outside the baby. It doesn’t matter if we think crabs should be semantically classified as part of “the environment”. It doesn’t matter that heritability would drop to zero if you killed all the crabs, or that the direct causal effect of the relevant genes has nothing to do with height. Heritability is a ratio and doesn’t care. So heritability can be high even when genes have no direct causal effect on the trait in question. It can be low even when there is a strong direct effect. It changes when the environment changes. It even changes based on how you group people together. It can be larger than 100% or even undefined. Even so, I’m worried people might interpret this post as a long way of saying heritability is dumb and bad, trolololol . So I thought I’d mention that this is not my view. Say a bunch of companies create different LLMs and train them on different datasets. Some of the resulting LLMs are better at writing fiction than others. Now I ask you, “What percentage of the difference in fiction writing performance is due to the base model code, rather than the datasets or the GPUs or the learning rate schedules?” That’s a natural question. But if you put it to an AI expert, I bet you’ll get a funny look. You need code and data and GPUs to make an LLM. None of those things can write fiction by themselves. Experts would prefer to think about one change at a time: Given this model, changing the dataset in this way changes fiction writing performance this much. Similarly, for humans, I think what we really care about is interventions. If we changed this gene, could we eliminate a disease? If we educate children differently, can we make them healthier and happier? No single number can possibly contain all that information. But heritability is something . I think of it as saying how much hope we have to find an intervention by looking at changes in current genes or current environments. If heritability is high, then given current typical genes , you can’t influence the trait much through current typical environmental changes . If you only knew that eye color was 100% heritable, that means you won’t change your kid’s eye color by reading to them, or putting them on a vegetarian diet, or moving to higher altitude. But it’s conceivable you could do it by putting electromagnets under their bed or forcing them to communicate in interpretive dance. If heritability is high, that also means that given current typical environments you can influence the trait through current typical genes . If the world was ruled by an evil despot who forced red-haired people to take pancreatic cancer pills, then pancreatic cancer would be highly heritable. And you could change the odds someone gets pancreatic cancer by swapping in existing genes for black hair. If heritability is low, that means that given current typical environments , you can’t cause much difference through current typical genetic changes . If we only knew that speaking Turkish was ~0% heritable, that means that doing embryo selection won’t much change the odds that your kid speaks Turkish. If heritability is low, that also means that given current typical genes , you might be able change the trait through current typical environmental changes . If we only know that speaking Turkish was 0% heritable, then that means there might be something you could do to change the odds your kid speaks Turkish, e.g. moving to Turkey. Or, it’s conceivable that it’s just random and moving to Turkey wouldn’t do anything. But be careful. Just because heritability is high doesn’t mean that changing genes is easy. And just because heritability is low doesn’t mean that changing the environment is easy. And heritability doesn’t say anything about non-typical environments or non-typical genes. If an evil despot is giving all the red-haired people cancer pills, perhaps we could solve that by intervening on the despot. And if you want your kid to speak Turkish, it’s possible that there’s some crazy genetic modifications that would turn them into unstoppable Turkish learning machine. Heritability has no idea about any of that, because it’s just an observational statistic based on the world as it exists today. Heritability: Five Battles by Steven Byrnes. Covers similar issues in way that’s more connected to the world and less shy about making empirical claims. A molecular genetics perspective on the heritability of human behavior and group differences by Alexander Gusev. I find the quantitative genetics literature to be incredibly sloppy about notation and definitions and math. (Is this why LLMs are so bad at it?) This is the only source I’ve found that didn’t drive me completely insane. This post focused on “broad-sense” heritability. But there a second heritability out there, called “narrow-sense”. Like broad-sense heritability, we can define the narrow-sense heritability of height as a ratio: The difference is that rather than having height in the numerator, we now have “additive height”. To define that, imagine doing the following for each of your genes, one at a time: For example, say overall average human height is 150 cm, but when you insert gene #4023 from yourself into random embryos, their average height is 149.8 cm. Then the additive effect of your gene #4023 is -0.2 cm. Your “additive height” is average human height plus the sum of additive effects for each of your genes. If the average human height is 150 cm, you have one gene with a -0.2 cm additive effect, another gene with a +0.3 cm additive effect and the rest of your genes have no additive effect, then your “additive height” is 150 cm - 0.2 cm + 0.3 cm = 150.1 cm. Note: This terminology of “additive height” is non-standard. People usually define narrow-sense heritability using “additive effects ”, which are the same thing but without including the mean. This doesn’t change anything since adding a constant doesn’t change the variance. But it’s easier to say “your additive height is 150.1 cm” rather than “the additive effect of your genes on height is +0.1 cm” so I’ll do that. Honestly, I don’t think the distinction between “broad-sense” and “narrow-sense” heritability is that important. We’ve already seen that broad-sense heritability is weird, and narrow-sense heritability is similar but different. So it won’t surprise you to learn that narrow-sense heritability is differently -weird. But if you really want to understand the difference, I can offer you some more puzzles. Say there’s an island where people have two genes, each of which is equally likely to be A or B. People are 100 cm tall if they have an AA genotype, 150 cm tall if they have an AB or BA genotype, and 200 cm tall if they have a BB genotype. How heritable is height? Both broad and narrow-sense heritability are 100%. The explanation for broad-sense heritability is like many we’ve seen already. Genes entirely determine someone’s height, and so genotypic and phenotypic height are the same. For narrow-sense heritability, we need to calculate some additive heights. The overall mean is 150 cm, each A gene has an additive effect of -25 cm, and each B gene has an additive effect of +25 cm. But wait! Let’s work out the additive height for all four cases: Since additive height is also the same as phenotypic height, narrow-sense heritability is also 100%. In this case, the two heritabilities were the same. At a high level, that’s because the genes act independently. When there are “gene-gene” interactions, you tend to get different numbers. Say there’s an island where people have two genes, each of which is equally likely to be A or B. People with AA or BB genomes are 100 cm, while people with AB or BA genomes are 200 cm. How heritable is height? Broad-sense heritability is 100%, while narrow-sense heritability is 0%. You know the story for broad-sense heritability by now. For narrow-sense heritability, we need to do a little math. So everyone has an additive height of 150 cm, no matter their genes. That’s constant, so narrow-sense heritability is zero. I think basically for two reasons: First, for some types of data (twin studies) it’s much easier to estimate broad-sense heritability. For other types of data (GWAS) it’s much easier to estimate narrow-sense heritability. So we take what we can get. Second, they’re useful for different things. Broad-sense heritability is defined by looking at what all your genes do together. That’s nice, since you are the product of all your genes working together. But combinations of genes are not well-preserved by reproduction. If you have a kid, then they breed with someone, their kids breed with other people, and so on. Generations later, any special combination of genes you might have is gone. So if you’re interested in the long-term impact of you having another kid, narrow-sense heritability might be the way to go. (Sexual reproduction doesn’t really allow for preserving the genetics that make you uniquely “you”. Remember, almost all your genes are shared by lots of other people. If you have any unique genes, that’s almost certainly because they have deleterious de-novo mutations. From the perspective of evolution, your life just amounts to a tiny increase or decrease in the per-locus population frequencies of your individual genes. The participants in the game of evolution are genes. Living creatures like you are part of the playing field. Food for thought.) Phenotype ( P ) is never defined. This is a minor issue, since it just means “trait”. Genotype ( G ) is never defined. This is a huge issue, since it’s very tricky and heritability makes no sense without it. Environment ( E ) is never defined. This is worse than it seems, since in heritability, different people use “environment” and E to refer to different things. When we write P = G + E , are we assuming some kind of linear interaction? The text implies not, but why? What does this equation mean? If this equation is always true, then why do people often add other stuff like G × E on the right? The text states that if you do a planned experiment (how?) and make Cov( G , E ) = 0, then heritability is Var( G ) / Var( P ). But in fact, heritability is always defined that way. You don’t need a planned experiment and it’s fine if Cov( G , E ) ≠ 0. And—wait a second—that definition doesn’t refer to environmental effects at all. So what was the point of introducing them? What was the point of writing P = G + E ? What are we doing? Create a million embryonic clones of yourself. Implant them in the wombs of randomly chosen women around the world who were about to get pregnant on their own. Convince them to raise those babies exactly like a baby of their own. Wait 25 years, find all your clones and take their average height. If heritability is high, then given current typical genes , you can’t influence the trait much through current typical environmental changes . If you only knew that eye color was 100% heritable, that means you won’t change your kid’s eye color by reading to them, or putting them on a vegetarian diet, or moving to higher altitude. But it’s conceivable you could do it by putting electromagnets under their bed or forcing them to communicate in interpretive dance. If heritability is high, that also means that given current typical environments you can influence the trait through current typical genes . If the world was ruled by an evil despot who forced red-haired people to take pancreatic cancer pills, then pancreatic cancer would be highly heritable. And you could change the odds someone gets pancreatic cancer by swapping in existing genes for black hair. If heritability is low, that means that given current typical environments , you can’t cause much difference through current typical genetic changes . If we only knew that speaking Turkish was ~0% heritable, that means that doing embryo selection won’t much change the odds that your kid speaks Turkish. If heritability is low, that also means that given current typical genes , you might be able change the trait through current typical environmental changes . If we only know that speaking Turkish was 0% heritable, then that means there might be something you could do to change the odds your kid speaks Turkish, e.g. moving to Turkey. Or, it’s conceivable that it’s just random and moving to Turkey wouldn’t do anything. Heritability: Five Battles by Steven Byrnes. Covers similar issues in way that’s more connected to the world and less shy about making empirical claims. A molecular genetics perspective on the heritability of human behavior and group differences by Alexander Gusev. I find the quantitative genetics literature to be incredibly sloppy about notation and definitions and math. (Is this why LLMs are so bad at it?) This is the only source I’ve found that didn’t drive me completely insane. Find a million random women in the world who just became pregnant. For each of them, take your gene and insert it into the embryo, replacing whatever was already at that gene’s locus. Convince everyone to raise those babies exactly like a baby of their own. Wait 25 years, find all the resulting people, and take the difference of their average height from overall average height. The overall mean height is 150 cm. If you take a random embryo and replace one gene with A, then the there’s a 50% chance the other gene is A, so they’re 100 cm, and there’s a 50% chance the other gene is B, so they’re 200 cm, for an average of 150 cm. Since that’s the same as the overall mean, the additive effect of an A gene is +0 cm. By similar logic, the additive effect of a B gene is also +0 cm.

Statistics Data Analysis

Evan Hahn 4 months ago

Notes from "The Weather Machine: A Journey Inside the Forecast"

In The Weather Machine: A Journey Inside the Forecast , author Andrew Blum gives a high-level overview of global weather forecasting. What complexity hides behind the simple interfaces of our daily weather apps? Three main points stuck out to me: “The weather machine has to be a global system, and it won’t work any other way.” International collaboration is critical. Weather models need lots of global data to be effective. Weather technology, like the internet , is closely tied to military interests. Some quotes: Blum describes weather models as simulations, tweaked with real data. “Rather than a meat grinder that transmogrifies the weather of the present to the weather of the future (a one-way process), I now pictured two spinning earths side by side. I saw the real earth, the planet we live on, in the view we have gained from traveling to space. And I saw the model earth, its simulated atmosphere swirling with clouds and storms, with the bonus capability of being able to run in fast-forward, into the future.” The European Centre for Medium-Range Weather Forecasts is the “current and undisputed champion” weather model, according to this book. The book covered a wide range of topics and was fairly short, so it felt a little shallow. But it had been on my list for months and was a nice, quick read. “The weather machine has to be a global system, and it won’t work any other way.” International collaboration is critical. Weather models need lots of global data to be effective. Weather technology, like the internet , is closely tied to military interests. Some quotes: “In a very practical sense, there were few clear distinctions between weather satellites and reconnaissance satellites, or cargo-carrying rockets and intercontinental ballistic missiles. It worked both ways. The military uses justified the meteorological efforts. The military efforts benefited the meteorological uses.” The NASA Jet Propulsion Laboratory “was haunted by the space program’s fundamental duality: They would create the machines that would open up a new era in human exploration—truly out to the edges of the solar system—while also creating the technologies that could destroy humanity.” “The only caveat written into the charter was that the World Weather Watch be used for peaceful purposes only.” Blum describes weather models as simulations, tweaked with real data. “Rather than a meat grinder that transmogrifies the weather of the present to the weather of the future (a one-way process), I now pictured two spinning earths side by side. I saw the real earth, the planet we live on, in the view we have gained from traveling to space. And I saw the model earth, its simulated atmosphere swirling with clouds and storms, with the bonus capability of being able to run in fast-forward, into the future.” The European Centre for Medium-Range Weather Forecasts is the “current and undisputed champion” weather model, according to this book.

Science Data Analysis