matduggan.com 2 days ago

You Know GDPR Is Good Based on Who Hates It

FDR has always been one of my favorite presidents, second maybe to Lincoln. Both were men the establishment assumed were one of them until, to their horror, they governed like they weren't. Both could trash the opposition in one breath and take the moral high ground in the next. One of my favorite Roosevelt lines growing up came from Madison Square Garden, October 1936, standing before a crowd that included plenty of people who wanted him dead: What I love is that he doesn't argue with the hate. He doesn't say they're wrong to hate him, or that the hate is unfair. He says the hate is evidence. The process is working. I've always treated it as a metric: if you're doing something hard and nobody hates it, you probably aren't doing it. If the right people hate it and those people happen to be some of the worst people alive, so much the better. By that standard, the GDPR (Europe's General Data Protection Regulation) is doing beautifully. It is impossible to go anywhere in a technology space online without hitting a wave of commentary about how stupid GDPR is. It was written by bureaucrats who don't understand the amazing potential of unrestricted technology. These US-based critiques almost always lean on the oldest trick in cyberlibertarianism: we don't have time to regulate, we must simply adapt and ride the wave. Nobody has time for government. Of all GDPR's consequences, none gets more attention than the cookie banner, which critics present as the inevitable result of government meddling. Blaming GDPR for the cookie banner is like blaming the health inspector for the roaches. The banner is deliberate vandalism, a dark pattern engineered to exhaust you before you can learn anything about the surveillance apparatus humming behind the "OK." Ironically the banner designed to hide the machine has taught the public more about the machine than a thousand podcasts ever will. Even non-technical people stop at "your data is shared with 996 partners." "For a sports score website?" So why is the tech commentary community so loud about this? Because they understand what's at stake. If consent must be freely given and easy to refuse, the industry's power shrinks exponentially. People might decide who has their data, how long it's kept, and what it was collected for. You can only imagine how that thought keeps a Meta executive up at night when he's not eating endangered animals, or ignoring calls from his children whose names he has forgotten while on a tacky yacht. Consider the following example. Did you know Google was doing this every single time you searched on Google ? Did your dad? So even in the most maliciously compliant form the regulation does provide value and information. Remember the hatred is the metric. How did we get here, where US tech companies end up regulated by Brussels? Why isn't the US government regulating US corporations anymore? If the rules are so terrible, why did nobody choose market exit? Has the EU become the world's "privacy cop" or, in the inverse, the biggest player to protect a fundamental human right to privacy? Ah who doesn't remember where they were on GDPR Day. Since we're all socialists in the EU, we stood up from our government issued desks and gave the required three cheers for regulation, then resumed being on vacation for 6 weeks. Obviously after stopping by my free doctor on my way to the airport. On May 25th, 2018, GDPR took effect to the sound of American commentary, the way fireworks take effect to the sound of dogs. You can tell American CEOs were aware of the regulation based on the speed by which they copied the language from it. Right before it took effect Brad Smith, the president of Microsoft, tweeted "We believe privacy is a human right." Tim Cook was right behind, telling CNN that "privacy is a fundamental human right". The framing of privacy as a human right is one of the key elements of the EU approach with GDPR. This is in stark contract with the US legal system which views information privacy as more of a market problem. You are all informed individuals in the wide marketplace of data exchanges and are left mostly to your own devices. In theory there should be regulations by the US of things like unfairness, deceptions and other market failures but in practice that doesn't happen. Europe is no stranger to this fight. The German state of Hesse passed the world's first data protection law in 1970, also known as the year the Beatles broke up, and set a standard we still fail to meet today: At the launch of GDPR there were 126 countries with data privacy laws of some sort. What you see with this sea of legislation is an overwhelming consensus that what GDPR was attempting to do was correct. In fact you see a pretty high level of global convergence of standards. All 126 laws descend from the same commandments the OECD carved in 1980: collect only what you need, say what it's for, keep it safe, let people see and correct it, and don't be a creep about any of this. Fifty years later, the American internet industry is still stuck on commandment one. So first the often-repeated sentiment that this is a flight of EU fancy is straight up incorrect. Something you could describe as the "European standard" for data privacy quickly became a global standard. Anu Bradford calls it the Brussels Effect: Europe regulates, the world complies, because despite American bluster, Europe is a market nobody can leave. In the US financial market, companies who surrendered the EU market would have quickly found themselves with new CEOs as their previous leaders suddenly discovered health problems or a deep love of their families that they had ignored for years. Especially given the increasingly frosty relationship between the US and China, companies that were pushed out of China due to regulation and increased domestic competition cannot lose the EU market. It's also difficult to screen a lot of services for EU customers, especially because the laws follow the personal data of EU residents whenever and wherever the information is transferred outside of the EU. Think of it like trying to sort luggage based on what stickers are on the outside. The EU is also set up in such a way where enforcement of such a law becomes possible. Every member state has a Data Protection Authority, who are charged with assisting individuals in protecting their rights, advising domestic legislatures on the functioning of existing regulation and finally enforcing the law. The EU is also different from the US in that it is open to exploring precautionary regulatory action. We see this with the EU Artifical Intelligence Act which tried almost immediately to get some controls on the industry right at the beginning. Link So the combination of a robust option for enforcement combined with an increased appetite for regulation in general and a high level of respect for personal privacy made the EU the logical source for this legislation. Want to know what the teeth look like? In 2011, an Austrian law student named Max Schrems asked Facebook for everything it had on him and got back 1,200 pages, much of it stuff he'd never volunteered. He filed a complaint from a dorm room. Four years later, the Court of Justice of the EU had voided Safe Harbor, the transatlantic data treaty, on the strength of it. A college student complaint killed an international agreement signed by presidents and prime ministers....multiple times. The best proof of GDPR's power is what it did to Japan. On January 23rd, 2019 the EU and Japan reached a deal which allowed for the free flow of personal data between the two economies. It established an overarching privacy law with a core set of individual rights and enforcement by independent supervisory authorities. The process took 2 years, which isn't a surprise because before this process Japan had very weak, swiss-cheese regulations. In 2014 Graham Greenleaf chose the title "The Illusion of Protection" for his chapter about Japan in an overview of Asian privacy laws. The private sector was basically unregulated, it has "easily manipulated exceptions" to its rules concerning the use and disclose of personal data, its absence of provisions for sentivie information and had no restrictions on data exports. It was nowhere near the level of protections that the EU would expect for information sharing. You can draw a straight line from a bargaining table in Brussels to new rights for a retiree in Osaka. Japanese data brokers hate it, which again is the point . Why hasn't the US sought the same arrangement? Because everyone involved knows the application would be denied. Which raises the real question: why can't the country that invented most of this technology produce a rule for it? Why is the US stuck pretending there's no reason to regulate while the rest of the world moves on? There's no better text on what has happened in the US with the data economy than The Age of Surveillance Capitalism by Shoshana Zuboff. It's a good read and I won't ruin it for you. First, the definition of Surveillance Capitalism from the book. Didn't read that? I don't blame you. In English they found oil and the oil was us. And as us Americans love to do, we immediately discovered a sudden love of freedom in the presence of oil. Effectively here's what happened. The 9/11 terrorist attacks had a ripple effect in US regulations, effectively derailing the momentum that the domestic US regulatory organizations had about starting to build frameworks around personal data. The focus became on security and not privacy. Quickly public intelligence agencies and the fledgling surveillance capitalist business in Silicon Valley found each other and carved out the concept of "surveillance exceptionalism". If you were spying to keep us safe, then it's not spying it's public service. As time went on, the corruption of American politics rendered the possibility of regulation less and less feasible at the federal level. Google and Facebook poured tens of millions into lobbying (for non-US readers, lobbying is a nice way of saying bribery that is legal). There also became an "open door" between government and tech, with 197 people moving back and forth from Washington to the Googleplex. What these companies learned is that by studying our behavioral data during periods of relaxation or play was the most value, allowing them to accurately and reliably push people towards profitable outcomes. US consumers were aware of this to some extent, often repeating phrases like "If it's free, then you are the product". That was true before, but in the new digital economy it is no longer true. We're not even the product anymore, which is why all these companies no longer give a solitary shit about whether their stuff is good or fun to use. We're the raw material that they mine. They know every single thing about us and we don't know anything about them. They accumulate all the data from us but not for us or for our benefit. Behavioral modification though the accumulation and manipulation of this data is now the wealth engine of the United States. Now we are in a market failure scenario. No individual company will disarm first, and any regulator can be captured for what these companies spend on catering. Only law with real enforcement teeth changes the math. And the US cannot pass laws with teeth anymore, not because voters don't want them (polling says they do), but because the pipeline is purchased. When someone says America "can't" regulate tech, they mean it the way a hostage "can't" reach the phone. GDPR didn't spread because Europe is heroic. GDPR spread because Brussels is what fills the room when Washington leaves it. Thanks to the amazing https://decryptads.com you can see what it looks like in real time. Let's take The Verge. This is a relatively straightforward tech commentary website that has a paywall. It should have a very simple supply chain in terms of advertising. What we see is the opposite. There is a giant network of companies trading, selling and bidding on the information from this website. This isn't the fault of The Verge, this is just how the machine operates. Your data is like fish at a fish market. Everyone gets to inspect the merchandise and decide whether they want to buy without you being involved. Ironically we only get this information because of the and which exist to combat advertising fraud. Even a well-run website operated by technologically literate people geared towards tech enthusiasts behind a paywall is not immune to this system. What you see here is that there is no fucking escape . If you want to have a large web presence and have bills to pay, you need to participate. And The Verge is doing it right! 59 declared partners and none of them are actively preparing for war with the US. In the US though this is not a new problem. The economic historian Karl Polanyi came up with the concept of "Double Movement". You can read it here. Think of it like this. Imagine you strip American Capitalism down to a tug of war. In the first round, the businesses go first. This is the "let the invisible hand decide" part. Everything becomes a product you can buy and sell including land, work, even money itself. Think of this like removing all the referees from a game and letting players do whatever they want. Then there is round 2, which is "wait this is hurting people". People demand protections like minimum wage laws, workers rights and trade regulations because the "free market" never seems to regulate itself but is causing real harm. The referees come back, but now with a rulebook written by the players who got hurt. His argument is that a totally free market is a fantasy. It's a fantasy because markets need government support to operate. Even people who pretend they despise government interference rely on them. Copyright and trademark doesn't matter when you need to train an LLM, but it matters a whole lot when I start marketing my iWatch smart watch. That rhythm of abuse, then correction governed American capitalism for a century. For surveillance capitalism, round two never comes. There is no functional federal counterweight as I write this in 2026. Some states are trying, and good for them, but regulating the internet state by state isn't progress it's just stopping the bleeding. Meanwhile the brokers buying and selling your life exist entirely outside your scrutiny, have no fear of comprehensive reform, and can swing a statehouse election for what they spend on a quarter's catered lunches. Entities this powerful cannot coexist with a functional democracy — a fact they seem to have considered, given the rise of the "Nerd Reich". https://www.npr.org/2026/08/10/nx-s1-5925350/the-nerd-reich-tracks-the-unmasking-of-silicon-valleys-true-politics So the order of operations is simple. Washington can't act, so Brussels does. Brussels acts, and the world follows, because the market is too big to leave. That is the whole story of GDPR: not European ambition, but American absence. A sign of the declining empire if you will. Roosevelt gave that speech at Madison Square Garden on the last night of October 1936, and a week later he won forty-six states. The people who hated him got Maine, Vermont, and the next ninety years of being wrong about everything they hated. That's the thing about being hated by the right people which is it is a currency and a valuable one. Privacy regulation will get there too. Not because the industry repents because industries don't, but because this is how every one of these stories ends: seatbelts, smoking sections, lead paint. Normal, then scandalous, then unthinkable. Someday someone will ask what an ad network for children was and refuse to believe the answer. And the executives who fought this will be on a boat somewhere, explaining that they were for it all along. Accept all.

Programming Security Data

▲ 0

0 views

matduggan.com 1 weeks ago

In Purgatory Everyone Likes Crepes: My Time at a Greek All-Inclusive

I can’t remember the last time I was actually hot. It’s a thought that keeps looping through my head as I stand next to a children’s play structure in a Greek all-inclusive waterpark. The sun is beating down on an international community of mostly parents drinking watered-down cocktails out of plastic cups while their children scream and run in circles. Surrounding this little paddock of plastic is a lazy river, filled with mothers and fathers staring blankly into the middle distance as they slowly orbit the water toys. It’s all concrete and aggressively bright primary colors, suffused with the smell of gyros being cooked by a Filipino staff behind me. Living in Denmark, it’s rare for the sun to shine directly on you. Typically, everything outside looks like it has a blue filter placed over the lens—like how TV shows put a yellow filter over the camera to establish that you are now in Mexico. Sometimes in Copenhagen you might get hot, but then it will immediately start to hail on you, and you suddenly have a different, worse problem. Greece, by contrast, feels like the sun is noticeably closer to the earth. It feels personal, like a heat lamp pressed against your skull. My daughter has been playing peek-a-boo with a cute baby on a bench next to this water playset for long enough that I feel a moral obligation to go over and say hi to the mom. She’s a nice woman, watching four kids at a waterpark alone with a casual calm I actively envy. I’m a little stressed keeping eyes on just one, but she has four kids ranging from around ten to two, running around and disappearing beneath the waves, only to emerge just before I would panic. She has a small sunburn on the small of her back that I assume she either can’t reach or forgot—a small, raw triangle of red. We chat briefly. She asks where I live; I ask the same. I’m surprised when she answers Russia, mostly because I don’t get much exposure to Russians in Denmark. I clearly make a face without meaning to, because she quickly adds, "Their father is off fighting in the war." Ironically, I think she added this to head off me judging her as a single mother. Instead, revealing the father is a Russian soldier has made the part of my brain that regulates politeness completely short-circuit. What is the social etiquette for the wife and children of a military power you don’t support? I’ve never even considered this question, standing here in my swim trunks, my too-pale stomach exposed to a hostile world. I don’t want her children to be orphans, obviously. Neither she nor they did anything to start this conflict. But I’m worried that expressing vague positivity will end like my conversation with an Israeli in Copenhagen who took my neutral statement, "I hope the conflict ends soon," and responded with a wink and, "It will, with Trump at the helm." "I hope he comes home soon," I say, keeping my face perfectly neutral. She nods and resumes throwing a ball around with her children. She seems completely unaffected by thinking about her husband, which I think is actually the strange magic of the all-inclusive resort. It is a geopolitical anesthesia. The world is burning down, but here, the ice cream machine is still running. On most topics, I don’t consider myself a snob. I enjoy a Michelin-star restaurant, and I enjoy a nice slice of 7-Eleven pizza at 2 AM. I'll finish a thick, sad Classic of Literature and then immediately pick up Space Marines vs. The Metal Skeleton Wizard on the Moon . But on travel, I’ve always looked down on the all-inclusive crowd. Sitting one rung above a Carnival Cruise on the ladder of human decay, the all-inclusive always seemed like the most cynical way to say you had legally stepped foot in another country. My first real exposure to this was in Puerto Rico. I would leave my bespoke 20-room boutique hotel and scoff at the barbed-wire fence surrounding the American mega-resort down the road. "Why would you fly to the Caribbean to eat at a steakhouse?" I’d say, dripping with scorn to my wife, who would nod as we walked through dark neighborhoods on our way to eat dinner in someone’s actual living room that Google Maps called a restaurant. As I traveled the world, I saw these giant compounds everywhere. I swore I’d never become one of those placid, sunburned people wandering around a sculpted grass lawn, slightly buzzed at 11 AM. So imagine my surprise when I found myself finishing my second tequila sunrise before lunch, staring at a too-blue pool. The resort’s "activity crew" was out on Crepe Island—so named because it had a full crepe station on an island in the middle of the pool. "Really, it's a crepe peninsula," I say to nobody. Nobody cares. They are aggressively not paying attention to anyone else. This is not a community. A line of unlicensed, Disney-and-Nintendo-adjacent costumes are paraded out, with "Merio" closest to me. Kids kept trying to give Merio a high-five. I don't know if it was the suit or if the guy inside was just sick of his job, but he kept waiting too long to high-five the kids back, inevitably wacking them in the face or the back of the head as they moved on. Over and over, an excited child would run up, followed by a bizarre, lagging delay before the inflated hand would strike. "At some point, that's just hitting kids," I said to the ether. The woman next to me put her AirPods back in. My wife had booked this trip as a reward to our daughter. Having been a remarkably good sport as we dragged her to such famously child-friendly vacations like "A Sheer Cliff Overlooking Icebergs in Greenland" and "Endless Climbing Up and Down Stairs in Florence," we wanted to give her a chance to just be a kid. When we asked her what her favorite vacation was, she responded with zero hesitation. "Lalandia," she’d say, before resuming coloring in her coloring books with so much force you could see the head of the marker get pushed back inside the plastic casing. Since Lalandia is effectively a massive, indoor all-inclusive in Denmark, we figured the same concept in Greece would at least be a warm alternative. We boarded a chartered flight direct to Greece, my first time on a flight where everything was delivered in Danish. I felt like a spy hiding among the locals, fearful they'd discover the American among them. Safe from prying eyes, I was pleased to watch the Danes engage in behaviors they usually made fun of us for. Loud conversations were had, crunchy snacks were aggressively unwrapped, and general uncouth behavior typically shunned was suddenly fair game at 30,000 feet. Upon landing, the tour coordinator told my wife and me that we had to "run" to make our bus. My wife took off for the toilet; I grabbed the kid and the big suitcase and sprinted for the van. We then sat in the van waiting to take off for over an hour. What I hadn't realized in my heroic rush to the van was that I had left a small carry-on sitting in the middle of the airport. It was filled with snacks, coloring books, and also my wife’s wallet and passport. The first night at our all-inclusive was understandably tense. Variations on the word "idiot" were thrown around. The next day, we got the good news that the bag was safe and sound at the airport police station on the island of Kos. So while my family went off to the pool, I went to the lobby and called a cab. My driver arrived 30 minutes later and immediately asked, "Is it ok if you sit up front so we also take these beautiful people downtown?" The Scottish woman in the back blushed at this. I, lacking the social architecture to recognize warmth, simply thought: Wow, bold move saying that in front of her husband. I got into the passenger seat. This, as I realized later, totally broke down the driver-passenger social construct. After dropping the charming Scottish couple off in the downtown area, my driver started telling me literally everything about his life. For the next forty minutes, I learned how his brother had been a firefighter, then one day went on vacation to Vietnam, fell in love with a Vietnamese woman, and opened the only authentic Greek gyro restaurant in Hanoi. He had a booming, theatrical laugh and was poured into a skin-tight polo shirt. "Why are you going to pick up this suitcase your wife lost? DO WE EVEN NEED TO ASK?? HAHAHA! WOMEN!" Then he’d get deadly serious and remind me, "Children, they're the most important thing." I had spoken maybe six words at this point in the trip. I forced a laugh, silently calculating the statistical probability of being murdered and buried in a Greek olive grove. It would be a beautiful place to die, though I worried they’d use a photo of me from this trip—pale, squinting in a baseball cap—for the Netflix documentary thumbnail. Some true crime podcast would inevitably conclude I had it coming. At one point, he pulled the cab over to show me a flat patch of grass. "This is where I saved the island from a wildfire," he said proudly. I couldn't see any sign of fire; it just looked like a normal field with the grass crushed. I didn't say anything, as up to this point, the conversation had been delightfully one-sided. I had mostly sat sweating through my shirt, nodding. He told me how someone had thrown a cigarette out of their car, and it had started to smolder. He had stopped and used his legally required fire extinguisher to put it out. "It's good that I had this," he said, patting the dash. "This is why a robot can never replace taxi drivers." He then stared at me as if I were an assassin sent from Silicon Valley. "You betcha," I said, sweating. I wanted to assure him I wasn't there to replace him with an app. My amoral peers back home were already working on it, but I personally wouldn't kill his livelihood. Just people who looked and sounded exactly like me. I kept poking around silently instead, but then he caught me looking at the giant bulldozer sitting twenty meters away, its tracks leading directly to the field. I'm a genius like that. "Yeah, alright, also the bulldozer helped a bit to put the fire out. But mostly, it was my fire extinguisher." The bulldozer was merely an accessory after the fact. He then changed the subject by telling me that Tom Hanks had sat in his taxi "right where you are sitting." "Did you know Tom Hanks loves Greece and we love him? He is such a friend of Greece that we gave him a passport." Is Tom Hanks Greek? I thought, staring out the window. Why are there so many interior decoration stores? Is everyone on this island redoing their bathroom? "Tom is so nice, we love him." My brain returned absolutely no information about Tom Hanks, which frankly is typical of my mind. He clearly wanted me to respond with a Tom Hanks factoid, but all I could think to say was, "Yeah, his son was really good in Fargo ." "HIS SON? Who is talking about his son?" He took the conversation back over, explaining how Tom Hanks’s wife had sat in the back of the cab while Tom sat in the front. Weird choice, Tom, I thought, looking for a sign we were wrapping this trip up. When we arrived at the small airport, he parked randomly outside the main building, basically just the corner of the road that led from the airport back to town. "Wait, are you coming in with me?" I asked, confused. He nodded, saying it would "take way too long" if I went in by myself. Suddenly we were concerned about efficiency. He marched us in, knowing everyone who worked there. I found myself in a small police office that definitely didn't understand the separation of church and state. There was a massive crucifix on the wall, and each police officer's desk was covered with a Jesus mousepad and a large picture of the Virgin Mary. My cab driver and the police officer negotiated the release of the suitcase in front of me without bothering to involve me. At some point, I was told I could take the suitcase and get out of here. Nobody had asked me any questions except to quiz me about the contents of the bag. On the way back, we stopped for tea because, truly, what is a taxi meter at this point? We'd been in the car together for like two hours. He told me how his mother-in-law had almost died from a heart attack. "It's good that she could get to Athens in time," he said. He then told me how during the off-season he harvests olives from his family farm and has them pressed down the street. "It's heaven working the fields with your family." I imagined my siblings and I being asked to harvest olives in the beating sun and immediately envisioned four body bags in the shade of an olive tree. I nodded, staring out the window at a fire extinguisher store across the street, where what looked like a 12-year-old boy rolled a cigarette seemingly one-handed and lit it while sitting on top of a barrel. The long grass around the fire extinguisher store seemed primed for a fire. What happens if a wildfire hits a fire extinguisher store? I pondered, while my driver went on about the beauty of the Mediterranean or something I wasn't paying attention. Then we packed it up and went back to the hotel. "Thanks for a nice morning, Mark," he said fondly to me as I got out. My name is Mat. But I guess if you've saved the island from a wildfire, you can call me whatever you want. I got back just after Mini Disco. This was a one-hour dancing marathon for the kids in the semi-enclosed theater. With two fully manned drink stations on either side of the stage, it was an opportunity for children to dance while their parents drank like an asteroid was imminent. The songs stayed the same every night, but on this first night, we were going to learn a valuable lesson: Always buy the t-shirt. Apparently, while I was learning the ins and outs of the Greek cab business, a nice woman had asked my wife and daughter if they wanted to buy a t-shirt for the Mini Disco. My wife, smelling a tourist scam, had hard-declined. What she didn't realize was that the climax of the Mini Disco was a formal t-shirt presentation ceremony. Each child's name was called, they were presented with a t-shirt, and then they were allowed to hug Leo the Lion. My daughter didn't understand that she didn't have a t-shirt. So, when the name "Eleanor"—which is not her name—was called, my daughter hopped up and snagged the shirt. This meant my wife had to rush onto the stage, rip the t-shirt out of our daughter's hands, and hand it to the sheepish little girl whose shirt it actually was, all while holding a cocktail. It was a masterclass in parenting under the influence. The next day, we were on the hunt to put in an order for the t-shirt, having made the most serious promises parents can make to a child. After we chased down the red-haired French woman running the kids' merch table, we went to the main pool. She didn't seem surprised to see us come crawling back. I got the sense the first day she was doing the sales pitch, then after that she just waited for us to come to her. There is a weird etiquette to pools and British people. They will rush out the second you are legally allowed to put a towel down on a chair, meaning from 8 AM to 1 PM, there isn't a single open seat. The British treat pool chairs the way trench soldiers treated no-man's-land: as hotly contested territory worth dying over. But if you are lazy like my family is, you just wait. They start to leave the pool around 1 PM, and you can get a great seat with zero work. The next day, you repeat the entire cycle. That evening, kiddo got her shirt. It was a proud moment; she clutched her blaze-pink t-shirt to her chest and teared up with pride. Soon, the days started to blend together. At some point, we all got an emergency text message telling us about wildfires on a neighboring island. I assumed the resort staff would need to calm down the crowds, as you could pretty clearly see the smoke from the fire. The sky was turning a dark, orange-brown in the distance, and the air smelled like burning rubber. Nobody cared at all. A thousand people looked at a Greek text message they couldn't read, put their phones down, and picked up their beach reads. The apocalypse was met with a shrug. As an allegory for global warming, there is something particularly bleak about people dancing in a pool to 90s boy band hits as a thick plume of wildfire smoke shoots up into the sky. Thankfully, here nobody pretends to care about anything, so I was able to slip back into a soothing apathy. In a world that was constantly asking me to pay attention to some fresh horror, this was a place that asked nothing of you. Toward the end of the trip, we decided to head down to the small town and were dropped off at Dolphin Square. We started walking around, looking at Google Maps to see what we should look at. There was a Roman House, which is basically an empty lot full of pieces of old Roman architecture and a sign saying I needed to pay 10 euros a person. Since nobody was there to collect this 10 euros, the sign felt more aspirational than practical. Someone should pay this Greek island 10 euros, but not today, I guess. But I looked at the old rocks. My daughter was unimpressed. She looked at the ruins, looked at me, and asked if she could get ice cream. "Denmark has a lot of old stuff," she said, which, in her defense, is true. We walked around a bit more, buying little tourist trinkets. I'll never understand who buys the t-shirts that fill the small streets of these towns. Who is the demographic for a shirt that says "I heart my boyfriend"? And more importantly, where is he? Does he buy the t-shirt for you before you go on a solo trip without him? But we went to a small restaurant and had a proper meal for the first time in days. Despite the heroic efforts of the staff at the resort, it was actually hard to eat there meal after meal. The culinary philosophy seemed to be "cook the will to live out of it." Everything was either fried and dried out beyond belief, a hard puff pastry, or so bland you couldn't really tell what was going on. The lamb sorta tasted like the chicken, which tasted a lot like the pork, which tasted like a cry for help. At one point, we went to an Asian-themed restaurant on the resort, and I was served "duck with Chinese pancakes." The duck was well-cooked, but the pancake was an Old El Paso flour tortilla. I couldn't help but feel like the duck died for no reason. At one of these meals, where my daughter pounded another plate of french fries and pizza, a British woman at the table next to us told me how her entire family looked forward to this trip. She started to tear up as she explained that she and her husband had worked extra hours this year to make it so their kids could come with them. "It just makes me burst with pride to think about," she said. As she spoke, one of her kids sat beside her, wearing noise-cancelling headphones, staring at an iPad, and methodically eating an entire pepperoni pizza without making eye contact with another human soul. By the end of our week, I was eating meals of plain bread with watermelon and coffee. My daughter was consuming what looked like a kilo of Nutella, and my wife and I settled down for another cycle. We only ended up going to the actual beach, which was beautiful and maybe 200 meters from our hotel room, once. The ocean was beautiful, with rolling hills in the background, perfect warm water that was the saltiest water I've ever been in. It was like swimming in a giant, warm tear. Everyone but me hated it because it wasn't as nice as the pools. "Ugh, there are ROCKS!" my daughter shouted, upset at the audacity of the ocean for existing. Honestly, the rocks hurt a fucking ton, but I was too self-righteous to admit it. "Let's enjoy nature, everybody!" I yelled, bleeding from the feet. I thought by the end of my time with these people that I would come to think less of them. I expected to leave despising their complacency. Instead, I found myself weirdly protective of these folks. They are just trying to make some childhood memories with their kids, trying to manufacture something, anything that looks like a normal childhood in a world burning down. I won't lie, the all-inclusive life isn't the life for me. It is a place out of time, where the enjoyment of it requires a total suspension of your relationship to the outside world. But I do now understand the appeal of not being challenged. In a time when everything is being questioned and every tradition and norm is falling apart, this is a callback to frankly an easier time to be alive. And we ended up getting the t-shirt, which, when you break it down, is really the most important part.

AI UI

▲ 0

0 views

matduggan.com 2 weeks ago

OTel Isn't Going Well (And I Made A Spreadsheet About It)

For years now one of the most reliable complaints I hear when I try to drag a team off their vendor specific SDK and onto OpenTelemetry is some variation of: "why does it seem like this isn't done yet?" Vendor SDKs for observability are, to put it charitably, idiot-proof. You install the thing, dashboards just load data, someone else worries about how all those pieces fit together, and you get on with your life. OpenTelemetry, by contrast, greets you at the door with a lot of "experimental" stamps and roughly six different ways to accomplish any given task. In OpenTelemetry's defense this was never what they were going for as a project. I've always respect that they stuck to their guns by attempting to build a truly vendor agnostic system that really doesn't care what you do with the data. I have never gotten a sense of a vendor being strongly preferred with OTel, which is quite the feat considering how lucrative and contentious the observability ecosystem was. Also considering that the maintainers of this project are largely employed by exclusively those companies. As the years wore on, I started to get nervous. Conversations in the semantic-conventions repo drag on and on and on. Different languages had dramatically different stories. Golang and Dotnet were first class citizens, but other languages lagged years behind the others. I started asking a lot of probing questions before recommending OpenTelemetry to smaller teams who didn't have the time, budget, or emotional bandwidth for it. Auto-instrumentation was genuinely magical, but the cliff between "auto-instrument works" and "now I have to manually instrument something" was steep enough that you owed people a warning before you pushed them off it. This narrative has been going on for awhile in the observability space, a vague sense of "something is wrong in Otel-land". But let's try to generate some actual data here. Is there an actual problem, or is this something where the perception by the community of slow progress is imaginary? Is the problem not enough maintainers, too big of a scope, or something in-between? My guess when I started was "oh this is your classic open-source bit off more than they can chew". Not enough maintainers, not enough budget. Now there is some of that, but there's also something else going on. The actual problem happening inside of OpenTelemetry is a three way crash. You have a binary stability gate which, when combined with a very small bench of actual maintainers means there is understandable worry about marking a feature not experimental then add on just a massive scope of languages and frameworks they are attempting to cover. This creates a perfect storm where there is an incentive to argue about potential problems a feature might create since once it is locked in and shipped as stable you can never change them. So OpenTelemetry currently is attempting to support a dizzying number of languages and frameworks. OpenTelemetry is a g iant project. It spans dozens of languages, hundreds of libraries, and countless backends. To keep things sane, the project splits work into two buckets: There exists the otel-collector, the thing that runs along the thing so that you can ship logs metrics and traces. That copies the same rough pattern. But for the languages when we're talking about core vs contrib this is what we're talking about. Stuff that breaks goes in contrib, stuff that doesn't break goes into core. Now the reason this causes a conflict. is massive overkill for most projects. You don't want 300 exporters to add the one you typically need. On the language side, this isn't that big of a problem. gives you the stuff you need for flask. However on the collector side you end up having to do the OpenTelemetry Collector Builder to make your own collector (or just kinda ride the wave and hope it works out). While cool that this exists, it's a lot of scope to ask a team to take on. So I believe I have captured the workflow of adding a new feature to OTel. You can check my homework here: Things I'm not really clear on So because OpenTelemetry is a CNCF project, I figured it made the most sense to compare them to other CNCF projects. My basis for comparison is Envoy and Prometheus. I have used a hacky Python script I've used before for measuring the "health" of open-source projects, which is probably not the best. However I'll include a link to the raw data without the charts so folks can review it and (more than likely) find a problem in what I generated. So we look at 24 months of activity for Envoy and what we see is a pretty healthy project. There's good distribution of authors, mergers, issue closers. is obviously pretty important to the project but in general there's a good bench of people to step in if needed. I've attempted to filter out all the known bot traffic. Let's compare that to one of the OpenTelemetry languages. The ones I have the most professional experience with are Golang and Python, but I hear from a lot of folks in the community that the Ruby and PHP ones struggle a lot. This is the PHP one for the same period. So we see pretty clearly that there's way too much concentrated on 2 people. This is not a healthy open-source project and they clearly don't have enough people to cover the kind of scope OTel needs to cover. Same story with Ruby. In comparison the "strongest" OpenTelemetry SDKs in my opinion, Golang and Dotnet (although Python is also no slouch) look more healthy. So the first issue is maybe the least surprising. There's too much concentration among too few maintainers. Your authors shouldn't also be your mergers and your issue closers. Ideally these tasks should be distributed out more evenly. For what its worth I think the maintainers have done a good job of attempting to keep their discussions public. It was very easy for me to find the public meeting notes of the different groups of maintainers, read through them and see what was going on. I don't get the sense that these maintainers are trying to stop people from getting involved as much as the expectations of stability have, more or less, frozen the project in place. The issue is more a classic case of "someone has to pay the maintainers". The project is too complex for someone to realistically do this as a hobby. I think any project signing on for such long stability contracts cannot turn to the community of hobbyists expecting assistance. I can't join calls and do the things I would be expected to do for a project of this size and importance for free. But it also means that the people doing this critical work have expectations placed on them by their parent organizations. So these SDKs have too few maintainers. But that doesn't fully explain why it seems to take so long for new features to get through the stack. My guess for that was that somewhere in the process between submission of the new idea and the formalization of the idea was a long discussion that took a million years. So with this level of surface area across different frameworks and languages, it makes sense to concentrate the conversation about conventions in one place. That lives here: https://github.com/open-telemetry/semantic-conventions If vendor debate is causing the slowdown, we should (in theory) see this slowdown in PRs here. Then you should see the slowdown basically propagate out. Spoiler alert, I was wrong about this. Big thanks to the OpenTelemetry people for having good conventions on labeling their PRs which made this much easier. So if is the slowdown, let's look at the slowest PRs there. Yeah some of them are pretty slow, but there are some complex topics being discussed. However interestingly this slowdown doesn't really trickle into the SDK/API space, suggesting that OpenTelemetry is going a good job of keeping these conversations siloed off. If we look at Python we see that their slowest PRs aren't related. In reality the slowdown for these are the extra required check imposed by the which requires another maintainer. But that seems appropriate and takes us back to the initial problem of "not enough maintainers". So after looking at all of this, the pattern becomes clear. A new feature takes a very long time to make it to the end user in OpenTelemetry because they take stability very seriously, combined with a relatively limited bench of talent to pull from. Once things make it through the entire stack, implementing the API and getting that API change through to the end user falls on an overworked maintainer pool. So what do we do? I think one idea worth exploring is adding some sort of time-bound beta tier. Basically between the "Experimental" and the "Stable" in the following diagram. The problem is that for end users, due to the extra steps to use Experimental features, they might as well not exist. 99% of us have no idea when an experimental feature is added and we would never engage with it. But if I knew the feature would stick around for at least 12 months without a removal and was more accessible to me as an end user, it could actually help the project get more actionable feedback. Basically a feature would go Experimental (pretty low usage) -> Beta (more exposed to the end user than Experimental) -> 12 months -> Removal or Stable. Now confusingly Beta exists for Otel but is used for SDKs, not for components. Like Rust is a Beta but it seems like Profiles cannot be a Beta. Honestly it's nearly impossible for me to figure out like what labels should apply to what things. I suspect nobody really knows. Here's the explanation of Beta that I think only applies to SDKs. In addition it is, respectfully, misleading to imply that Go and Ruby are being maintained at the same standard. This isn't a shot at the Ruby folks — they are doing heroic work with what they have. But pretending parity exists when it doesn't just creates confusion and quiet resentment when a user shows up expecting one experience and gets another. Being honest about maintenance tiers would let people make informed choices and might attract more help to the other tiers by naming the problem out loud. Finally I would try to surface these problems more openly for OpenTelemetry from the perspective of "we need more maintainers". I feel like the people doing this work probably knew there was a problem, but it seems like the community at large has no idea that there is a need for frankly more engaged ideally independent maintainers and contributors. OpenTelemetry is a great project that is doing great work. It's doing, frankly, heroic work at this scale with this few people. But I think in order to actually replace the vendor specific SDKs we need to start getting a bit more pragmatic about what is realistic to do in terms of stability contracts and number of languages. I don't think breaking changes are as devastating to the community as these promises imply as long as they are communicated well and I think with this thin of a bench of maintainers, something has to give. Anyway feel free to check my data for accuracy and let me know if you find problems! Core → Maintained directly by the OTel project. Small, stable, vendor-neutral, and tightly reviewed. This is the "spec-defining" surface. Contrib → Community- and vendor-contributed. Broader, faster-moving, and covers the long tail of integrations. OpenTelemetry Enhancement Proposal (OTEP) ( https://github.com/open-telemetry/opentelemetry-specification/tree/main/oteps/ ) Once the OTEP is accepted, the text goes into the Specification directory in the same repo. After that it seems to go to Semantic conventions. This seems to be where we get down to the specific details and where most of the long discussions seem to live. At this point we're talking about more or less a permanent commitment to this design and where the lock-in process becomes very hard to change. Each of the SDKs implements the API surface that is defined in the specification. Now some of the SDKs have done 2.0 breaking changes, so it does seem like the earlier "please no 2.0 at all costs" sentiment has been abandoned (which I think is smart and good). Contrib / instrumentation. This is slightly more mushy. Looks like they should track latest API/SDK but each contrib package may version independently so its more flexible as a design. Collector + OTLP. The data has to actually go somewhere. OTLP (wire protocol) has its own stability lifecycle and specification ( here ). Collector components have their own stability in their READMEs and as far as I can tell that's kinda all over the place. It's unclear how long the OTEP -> Specification process takes. I've looked through the Git history but there doesn't seem to be any predictable number or cycle. I don't fully understand what is the relationship between all these stability commitments. Does Collector + OTLP group work in lockstep? Can a language "fall out of scope" if you lag too far behind?

Open Source Go Php DevOps Performance

▲ 0

0 views

matduggan.com 1 months ago

I added a blogroll

I realized that it might be nice if you happen to stumble on this website if I had a way to recommend other websites you might enjoy. As it turns out this is a "blogroll", a concept I have never heard of before today but whatever. It's still a good idea. I tried to add some CSS to make it easier to follow and search, but let me know if you think I missed a great site that people should check out. I'm also always on the hunt for more good stuff to read. You can find my email and social on the About page above. Also if you want to just take this Ghost theme and use it yourself feel free: https://gitlab.com/matdevdug/minimal-ghost-theme Anyway here is my new blogroll: https://matduggan.com/blogroll/

Open Source Web Development CSS

▲ 0

0 views

matduggan.com 1 months ago

Art Doesn't Scale

Listening to two acquaintances argue about if AI art is "art" is not a new experience for me, the now eternal debate between Doomer and Boomer. In this particular context, I'm listening to it as I wait for my surprisingly expensive iced latte in downtown Copenhagen. This coffee shop is one of my favorites, a small hole in the wall where the shop starts on the second floor. Everything is hand painted in a way I associate with the hippie roots of Copenhagen. They have an older golden retriever lying on the floor in the heat with a sign on his crate telling people not to bother him if he's taking a break. "Sometimes he needs his alone time". Me fucking too I think as this conversation drags on. There is a certain irony to the people having this conversation. One is a well-off tech worker, telling the much less well-off artist that the thing he has made through a prompt and the thing she practiced for years to make are functionally the same thing which is a classic AI Boomer argument. "I'm not saying I would sell mine, but isn't the point of art the emotion it brings out when you see it?" The artist seems somewhat baffled by the question, which I understand because she doesn't see the world as a machine that inputs ideas and outputs profit. I use some LLM tools while programming but I would argue I am still more Doomer than Boomer. As someone who straddles these two worlds, both working for large corporations and interfacing with these borderline sociopaths in business casual pants and expensive watches and also as someone who finds the joy in their life through the celebration of the artistic output of others, I understand them both. One sees it as a capacity problem, a barrier broken down so that now he can do everything and no choices have cost. The other sees it as a baffling attack on her passion. You could always learn to draw, nothing was ever stopping you is her refrain. Which is true but doesn't resonate with him, in part because I suspect he believes he could do anything. But I hear this argument enough that I thought we should talk about it, if for nothing else than to give me something to think about as I stare at the dog crate and hope the dog comes out to say hi. So I love books of all sorts, including a lot of very trashy books. Nothing I enjoy more than a Space Marine ripping the head off an alien. One medium that I really like as an adult with limited free time are comic books. I love everything about them, the feel of the too slippery covers and the rub-off on your fingers of old comics. I love how weird they are, how stories that don't work are dropped on the floor and never mentioned again. It's like watching someone build a railroad as the train rides down the tracks. My routine for years was to go pick up my pull list and then wander around for a bit while they assembled it. This is how I found great series like Sandman and countless others that ended up being some of my favorite reads. One of these pull list discoveries I made while standing around surrounded by the delightful acidic smell of a comic book shop was The Black Monday Murders. You can find it here . The story of The Black Monday Murders is genius and I won't ruin it. You can read the first issue for free at the link above. My friends and I still text each other "All Hail God Mammon" when the stock market goes way up or down. The basic concept was "what if money and magic were related and banks were basically worshiping the god of greed". I consumed this series and fell in love with it. I mean come on what's not to love about that. So the first issue comes out in 2016, we get a....very slow drip of new releases through 2018. 2018 is when I got issue 8 and then that's it. Since then we have had teased new issues, every year or so someone posts on social media "oh there's a new one coming". There is a lot of media like this but for me, this is the one I think of most often as "what-if". I think it is a genius idea that is especially relevant in today's world and I am sad that we may never get another one. So I was expressing this to a boomer friend who casually responded with "why don't you just make your own"? LLMs allow us to generate lots of artwork like this now, you can maybe cobble together a story, wouldn't it be fun to make your own issue for your own personal consumption? To him, this is the perfect use of this technology, allowing a fan to make more of the thing they love. To me it feels like if someone said "if you are hungry, there's a puppy over there, just pop that bad boy in the oven for 2 hours and have yourself a snack". To me that's not a tribute or the act of a fan. It's making a forgery. The thing has no value to me as a reader if I removed it from the entire creative context of its creation. There's a great short essay by the philosopher Denis Dutton from 1979 called Artistic Crimes . It's about forgeries, not AI, but every paragraph rings like a bell for the argument I keep having. You can read the entire thing here (it's not that long I promise). What I was struck by was how many parallels you can draw between the two issues. Like AI art, the existence and occasional acceptance as real artistic works of forgeries has a special power to discredit the art critic and historian community. The success of a forgery creates a belief that the entire criteria by which art is judged by is fundamentally flawed. If something is amazing before you knew it was a fraud and trash after you came to find out it was fake, then the assessment of art has no basis in aesthetic properties. It is pointed to as a demonstration that the only reason a "masterpiece" is valued over a random painting at the flea market is that a famous person made the first one. Effectively all art is bullshit and LLM slop/forgeries are equally valid because they invoke the same emotions in the audience before the deception is revealed. Now everyone, or at least everyone whose opinion I value, can agree that knowing if a piece of art is real of fake matters in terms of both its monetary value and its value historically. Even the most staunch Boomer I know would admit that fake paintings shouldn't hang in museums next to real ones. Where the two groups diverse, in LLM art and forgeries, is whether the assessment of its aesthetic merits should not be impacted by its status as real or fake. Part of what is impressive about the performance of making something is that you are celebrating how much a person or group of people succeeded. This is why we delight in stories like "Stardew Valley was made by one person in his house for years with no money hand-drawing each piece of art in the game, so intense was his love and commitment to the project". You don't have to know that to enjoy the game, but pretending it changes nothing about how you enjoy it once you do it is nonsense. Same with movies like "Good Will Hunting". Sure it's a good movie, but the story of two childhood friends making it for not a lot of money and one of them leaving Harvard to shoot it changes your connection to the material. It can't not. You love the thing, you learn more about the human story behind the thing, you love the thing even more. Part of the magic of wandering through a museum is that I cannot do the things the artists here have done . I cannot make modern art, I don't have a vision like those people do. I don't have the strength of character and conviction to commit to something like they do. I frankly lack the fucking spine to stand there next to a creation of mine that a thousand morons a day will look at and say "I could do that". No you couldn't and that's part of why its impressive to look at. Forgeries, like LLM art, are misrepresentations of achievement. You want the same credit and respect as someone who actually did a thing, but you haven't done the thing. Once that discovery is made or, frankly, one that is suspected, the relationship to the work has been permanently changed. I'm no longer viewing it from the perspective of the output of a human who I want to succeed but instead skeptically as someone attempting to steal credit for work they didn't do. The other argument the boomers make is about bottlenecks. My favorite comic isn't finished because the author is busy and human. But we could feed the machine the premise, let it crank out pages, have him touch them up, and, bim bam boom issue nine, ten, eleven, a hundred. Let me be clear that I'm not precious about everything in my life. I am not one of those people who thinks everything must be handmade by candlelight. I love a Dunkin Donuts coffee. I love the ridiculous bucket-sized cup, the sugar content that could kill a horse, the milk-adjacent liquid that has clearly never seen the inside of a cow and is shelf-stable until the heat death of the universe. I love standing in line behind two guys in Carhartts complaining about a foreman named Chris who is, and I quote, "a real fucking prick, man." I could eavesdrop in an IHOP for the rest of my natural life and die happy. Sometimes I want the $22 burger. Sometimes I want the McDonald's cheeseburger, no pickles. Both are fine. Both have their place. But here's the thing the boomers keep missing: I get to pick. The choice is the thing. And the AI rollout, as currently constructed, is designed to remove the choice. Not offer a new option but replace the existing one. The boomer argument, stripped of its language about productivity and democratization, is really this: we're going to make the cheaper thing, we're going to stop labeling it, and you're going to keep paying the same price because what are you going to do, not consume media? Because it is cheaper and faster to make doesn't make it something anybody wants. My favorite local bakery makes a great sourdough that I try to get whenever I can. If they sell out, I don't buy a loaf of Wonder Bread and pretend it's the same thing. I go home without bread. Sometimes the honest answer to scarcity is going without . This is what boomers can't hear. They think we're arguing about the quality of the output, and so they keep showing us better and better outputs as if that will close the deal. But we're not arguing about the output. We're arguing about what it means to receive a made thing from another human being. Every comic in my pull list, every trashy Space Marine novel, every weird indie game are letters from strangers who cared enough to spend years of their lives making something for me to find. The value isn't in the artifact. The value is in the fact that somebody bothered . A machine cannot bother. It can only produce. I would rather have eight issues of The Black Monday Murders and the ache of never getting a ninth than a thousand AI-generated issues that continue the story forever. The ache is part of the art. The waiting is part of the art. The possibility that the thing I love might never be finished is part of what makes it real. Forgeries don't ache. They just fill space, like the paintings in a Best Western lobby, hung there because a wall needed something on it. All hail God Mammon. He always did prefer the cheaper option.

AI

▲ 0

0 views

matduggan.com 1 months ago

Midnight Train to Stockholm

I was recently summoned to a meeting in Stockholm, a city I had somehow managed to avoid despite living in Copenhagen for years. My Swedish experience, up to this point, consisted entirely of trips to Malmö — the closest Swedish city to Denmark and, more importantly, home to a Costco. As an American living abroad, I am duty-bound to report there every six months so the proper authorities know I'm still alive and to procure my ceremonial barrel of peanut butter pretzels. It's less a shopping trip than a consular check-in. Stockholm, it turns out, is much further from Copenhagen than anyone lets on. My options were to fly or take the train. Flying is technically a one-hour affair, but to make a 9 AM meeting I'd have to wake up at 4 AM, shuffle through security in a fugue state, and land in Sweden looking like a hostage video. I wasn't sure when I'd nap. This seemed insane. Then I had what I believed, at the time, to be a genius idea: the overnight train from Malmö to Stockholm. I'd sleep en route, wake refreshed, stride into my meeting like a man who understood something about life that others did not. Plus, I love trains. Sweden has a high-speed line that does the run in four hours, but the night train takes its time, which sounded charming. It was not charming. The first surprise came at booking. There are three tiers of experience: a seat, a couchette (whatever the fuck that is), or a private sleeping compartment. Since I wasn't paying, I chose the private compartment. This would prove to be the single smartest decision of my adult life, possibly of anyone's adult life. If you are reading this from some point in the future and you are over the age of thirty, book the private compartment. I don't care what it costs. Sell a kidney. I'll explain. My train left Malmö at 10:30 PM. The station there is depressing in a way that's hard to articulate in that nothing is obviously wrong. There's a grocery store. There's a convenience store. And yet everyone inside looks stranded , as though they've been waiting for something that isn't coming. Small children roam in feral packs. There is a pervasive sense that this is the last train out of somewhere very bad, and that whatever is chasing everyone is still, perhaps, on its way. Amtrak in the US has this feeling, a vibe that you are running from the law. I remember when my family used to take the train from Ohio to New Jersey, waiting for it at a train station that was basically a concrete bunker in the middle of a corn field. The concrete box would either be freezing cold due to too much AC or boiling hot due to too much heat. I would stay up late on the train and watch as parents would abandon sleeping children to jump off at stops and catch a smoke. They all had a nervous desperation that these Swedish travelers shared. It's the kind of place where you wouldn't be surprised to see someone take a SIM card out of a phone and throw it in the trash. I stopped at the grocery store and loaded my backpack — my only luggage — with provisions: two large water bottles, wet wipes, a change of clothes, a bag of nuts, and, as an emergency measure, two Red Bulls, in case sleep failed me and I had to power through the following day on hatred and that cursed, faintly urine-themed energy drink. Then I stopped by the men's room, which featured a decorative fish tank whose sole occupant had a full-frontal view of the urinals. If reincarnation is real, Henry Kissinger is in that tank. European train station bathrooms are often weird, but this was up there. First there was no automated system to get in, it was just a guy with a credit card reader. Also they were piping in tropical sounds to the bathroom which I assume is to cover the unspeakable horrors happening in the stalls. I felt uncomfortable that I kept looking at the fish and found it to be always staring at me. The tank was in really good condition, with incredibly clean water. I couldn't help but think maybe its better for the fish to die faster than live their entire lives staring at an endless line of men peeing. I found my train and boarded, and I knew immediately I was in trouble. This was an old-school train — varnished wood, worn blue upholstery, late-70s energy throughout. The regular seats were hard, upright benches with little fold-down wooden tray tables. They did not recline. Not a little. Not at all. Which raises the question: why call it a night train? Night train implies, to me, that at some point during the night, someone might sleep . But the seats also had bright lights above them that never turned off, which transformed them from sleeper seats into something closer to interrogation chairs. The couchettes turned out to be stacked bunks — men-only, women-only, or mixed and the passengers were packed together so tightly that the gender segregation began to make a grim, practical sense. I'm not squeamish around strangers, but we're talking well within reach-out-and-stroke-someone's-hair range. My private room looked roughly like a jail cell: a cot, a light that turned off, a door that locked. In other words, everything I have ever wanted from a hotel. That's not sarcasm, I'm easy to please. Naturally, I was far too curious about the rest of the train to actually sleep, so I set out to wander. The three conductors on duty were all wearing bodycams strapped to their chests, which is always an encouraging sign in that it suggests both that they had been attacked and that they had, at some point, done some attacking of their own. They were also wearing shorts, which felt deeply wrong. There's something unsettling about a train conductor without pants. It's a formal job. You don't want your pilot in flip-flops and you don't want your conductor showing knee. I don't know what it is about shorts on men specifically that come across as clownish, but there is a ranking of jobs where one shouldn't wear shorts all the way to one cannot wear shorts. Doctors, pilots, lawyers, accountants are all pants jobs. Train conductor felt like a no-brainer that it would be a pants job, also frankly I think they should also have to wear a cool hat and have a pocket watch. In the same way I would bristle at a judge sentencing me to death in a Hawaiian shirt, a train conductor in shorts checking my ticket feels wrong. I made my way a few cars down to see how the general population was faring. The door slid open and I was hit, physically, by the smell of cheap vodka. Before me stretched a sea of Swedes, each with the specific facial expression of a person who has just realized they have made a terrible mistake and cannot un-make it. Two people were openly weeping. Three others were borderline-homeless-looking punk kids dressed exactly the way punks dressed in 1994 — I don't know who is still manufacturing M65 field jackets and military jump boots, but they're clearly still moving units in southern Sweden. One of the punks was vomiting into a plastic bag, seated next to a very sweet-looking, deeply concerned young woman who had presumably boarded this train with hopes and plans. She had the look of a woman who had her life together. The conductors arrived, spoke to him, were told to fuck off, and then quietly relocated the young woman to a new seat the way you'd move a house plant away from a leaking radiator. The punks then began joking loudly among themselves, two of them taking turns retching up what smelled like vodka cut with unleaded gasoline. God help anyone trying to sleep back here. Between the puking, the reek, and the punks openly hitting on every woman within shouting distance, it was like being trapped on a Greyhound bus that had sworn a blood oath never to stop. I watched a man roughly my own age attempt to sleep by laying his face directly on the wooden tray table, earplugs jammed in, arms limp at his sides, in the international posture of I have given up . I left when a boyfriend and girlfriend began fighting because she had proven surprisingly receptive to the advances of a punk kid whose body odor was strong enough to reach me three rows back. The boyfriend — sitting directly next to her — took issue with this, which seemed reasonable. I moved on to the meal car. The meal car was the hangout, the refugee camp, the place where people who had discovered they couldn't sleep in the bunks came to sit and stare into the middle distance. Two young women were seated at a table nearby, one of them work-shopping, in English, why she deserved better than her current boyfriend in Malmö. "I don't think I should settle for average." Her friend was being supportive and kept trying to inject something about her own life, only to be steamrolled every time. "Yeah, I know exact—" "It's just, I work so hard at school and he doesn't." "My last boyfr—" "Maybe when we get to Stockholm we go buy some nice dresses and go dancing." "That sounds fu—" "Because I really do think I deserve to feel beautiful ." After a few rounds of this, I got bored. It was all early-twenties drama, and I don't say that with contempt because it's a phase we all pass through. In your twenties, you discuss your plans and feelings as though they matter, because to you, they do. In your late thirties, you come to understand that nobody actually cares if you live or die, and you learn, mercifully, to keep it all to yourself. It's one of the small gifts of aging, along with knowing how to fold a fitted sheet and no longer pretending to enjoy helping people move. Friends in your 20s are your therapists, your closest confidants and your relationship counselors. In your 30s, you have an actual therapist and don't have to burden the people around you. At some point in everyone's life someone they love and respect will put up their hands and say "alright ENOUGH" and you'll realize how tiring you are. These women hadn't gotten there yet, but I did think the friend should start charging for this therapy session. I retired to my cabin and fell asleep almost instantly. The rocking of the train, the smug satisfaction of not being propped upright in a wooden pew next to a vomiting stranger, and a modest dose of melatonin combined into something close to bliss. We arrived on time. I grabbed my backpack and stepped out into Stockholm Central Station at 6 AM, which was nearly deserted. I found a coffee shop, ordered a coffee, and was charged an amount that made me wonder "should I open a coffee shop in Stockholm?". As he handed it over, the barista said, cheerfully, "It's my first day." "You shouldn't tell people that," I replied. I then felt terrible about it for approximately one hour. He, for his part, seemed entirely unfazed, or possibly hadn't heard me at all, which somehow made it worse. I carried the guilt with me through security, out onto the street, and into the cab I hailed for the twenty-minute ride to my meeting. The meter began climbing almost immediately, and with a kind of enthusiasm I hadn't previously known meters possessed. He asked where I was from. I said the U.S. He said he loved Americans. He said Americans were the best people which, frankly, nobody says unless they're being tortured by us. He asked if this was my first time in Stockholm. He asked what I did for work. Every question was warm and generous and I understood, dimly, that I was being courted. But it was only twenty minutes, how bad could it be? Eleven hundred kronor later, I stepped out onto the curb while he tried to press his business card into my hand so I could call him directly for future rides. Having just paid roughly $120 to travel the distance of a decent jog, I now understood his enthusiasm. He'd hit the jackpot, and the jackpot was me. The meeting wrapped in a few hours. I took an Uber back for a quarter of the cab fare — a small, petty vindication I savored the entire ride — and spent the next four hours walking around Stockholm. It's genuinely lovely, and denser than I expected. Copenhagen feels like a city that was designed by someone who liked people; Stockholm feels like a city that was designed by someone who respected them but wasn't sure he wanted them over for dinner. More cars, less green, wider streets, harder edges. Every Swede I passed looked as though they were on their way to something slightly more important than what I was doing. It was beautiful in the way certain people are beautiful — the kind of beauty that doesn't especially need you to notice. Should you take the overnight Swedish train? Honestly? Probably not this one, unless you are terrified of flying or being sober. The only tolerable option for an adult human is the private cabin, and at that price you can usually just fly. But it was, undeniably, an experience and one I would happily repeat, provided someone else were footing the bill.

Culture

▲ 0

0 views

matduggan.com 2 months ago

Clickhouse is winning the Observability Wars

For roughly the last ten years, a meaningful percentage of my working hours have been spent thinking about observability. If you're not familiar with the term, "observability" is what we call it now that "monitoring" doesn't sound expensive enough. The actual work is unglamorous in that you collect a lot of logs, some metrics, a few traces, and then you give them to people. I generally like my job. I like that we're always trying new ideas and approaches. I like the fact that when things go wrong, the answer is almost always sitting there in the data, waiting to be found by whoever is patient enough to look. But I want to be honest with you: in ten years of doing this work, across a half-dozen companies and every observability platform you've heard of and a few you probably haven't, logs have never stopped being the worst part of the job. They were the worst part when I started. They are the worst part today. I fully expect them to be the worst part of this job forever until the robots rise up and rip my head off in one clean sweep. I've written about why logs are terrible before , so I'll spare you the full lecture and give you the short version. Every developer's expectations for logs are set by a single formative experience: the syslog box. Or a container running locally. Or tail -f on a production server they probably shouldn't have SSH'd into. The point is that at some early, tender moment in their career, they had an experience with logs that was flawless. They ran and something useful came back. They piped it into jq and got exactly what they needed. This experience is the observability equivalent of a first kiss. It ruins them for everything that comes after. Because here is the thing about that flawless experience: it works because the system is small, the volume is trivial, and the person querying is the same person who wrote the log line. There is no schema drift, no cardinality explosion, no cross-team consumer with dashboard expectations, no VP asking why the "revenue events" graph has a gap in it. Then there are forty services. Now there are four hundred. Now the logs are being consumed not just by developers but by customer service, who need to look up a specific user's failed checkout from Tuesday. And by the data team, who are quietly building a business-critical dashboard on top of a log line that a backend engineer is about to refactor without telling anyone. And by the on-call, who at 3 AM does not want to learn a new query language, does not want to think about index patterns, and would like the search bar to just work. So you have a technical problem — the volume is enormous, the shape is inconsistent, the queries are unpredictable — sitting on top of an expectations problem, which is worse. Developers want logs instantly, they want to run arbitrary operations on them, and they will not commit to a schema. Meanwhile the less-technical consumers of that same data want the dashboards to be stable forever, the UI to be forgiving, and the whole thing to feel like a normal product. These two audiences are, in most practical respects, at war with each other, and you are the diplomat. ClickHouse came out of Yandex, where it was built to chew through analytical queries against absurd volumes of clickstream data. It was not designed for observability. It just happens to be shockingly good at it, because clickstream data and observability data have a lot in common: high volume, append-heavy, time-ordered, mostly read in aggregate, and every so often you need to reach in and find one specific needle. You can run it yourself with Helm charts. You can point Grafana at it via the ClickHouse plugin, or use their own web UI, or bring your own frontend. Their docs are actually good, which I mention because it's rare enough to be worth flagging. I've never used their ClickStack setup though, so YMMV. For observability specifically, the OpenTelemetry Collector has a ClickHouse exporter, which means you can pipe OTLP data straight in and let it manage the initial schema for you. ClickHouse is designed to scan billions of rows and ingest an amount of data that, when you first see the numbers, makes you assume they're lying. They're not lying. You query it with SQL, which is a language that already exists and was not created by a startup two weeks ago. I'm ranting about logs and then I'm explaining why I like to administer Clickhouse more. Let me take a second and explain why Clickhouse is really good at logs at scale. Logs, as a data shape, have some peculiar properties. They're append-only. You never update a log line, and you almost never delete a single one, though you delete a lot of them at once when retention kicks in. They arrive roughly in time order, though never actually in order. They're read in bursts where nobody looks at logs for days, and then during an incident somebody wants to scan a billion of them in seconds. They're highly compressible, because most of the bytes in your logs are repeated: the same service names, the same hostnames, the same error strings, the same JSON keys, over and over and over again. And critically, when you query them, you almost always want either a narrow time range across all fields or an aggregation across a wide time range with a few filters. You very rarely want "give me one specific row by ID" the way you would from a transactional database. (There are exceptions when its something like GDPR or compliance logging which is its own subgenre of nightmares). In a row-oriented database — Elasticsearch, Postgres, MySQL — the data for a single log line is stored together on disk. If your log has 40 fields and your query only cares about 3 of them, tough luck, you're reading all 40 from disk anyway. The database will filter it in memory, but the disk I/O has already happened. ClickHouse stores each column separately. If your query says SELECT service, status_code, count() FROM logs WHERE timestamp > now() - INTERVAL 1 HOUR GROUP BY service, status_code, ClickHouse reads exactly three columns off disk: timestamp, service, and status_code. The other 37 columns in your schema might as well not exist. On observability data, where you often have dozens of attributes but any given query touches three or four, this is the difference between scanning 800GB and scanning 40GB. This is also why the compression numbers look absurd. Columnar data compresses far better than row-oriented data because the values within a single column are, by nature, similar to each other. A column of service_name values might have a hundred distinct strings across a billion rows. ZSTD eats that for breakfast. You'll routinely see 10–14x compression ratios on real observability data, compared to 2–3x for Elasticsearch. The amazing thing is that ClickHouse scales without changing shape. I don't know how else to say this. Every other observability backend I've worked with mutates as it grows. The architecture at 1 TB a day and the architecture at 10 TB a day are recognizably different systems, with different failure modes, different ops burdens, and different mental models. ClickHouse at 10 TB a day looks like ClickHouse at 1 TB a day with more shards. That's it. That's the pitch. That's the whole reason I'm writing this. Let me show you what I mean. At 1 TB a day, every modern observability stack is roughly okay. If you're at this scale, you can pick almost anything and be productive. The differences below are real but they're not yet painful. Here is the honest truth: at 1 TB a day, ClickHouse is not less complicated than its peers. It's roughly the same. Maybe slightly more, if you count the schema design work you have to do up front. You get 10–14x compression with ZSTD and proper codecs, the Altinity Operator handles keeper coordination and the whole thing runs in about seven pods. But you do have to design your schemas. ORDER BY keys matter enormously. There is no native PromQL, so metrics workflows go through the Grafana plugin or through chproxy and an adapter. Roughly $1.5–2.5K/month. If you took the diagrams at this tier and squinted, you'd say they're all in the same weight class. And you'd be right. Now watch what happens next. This is where the exponential curve kicks in for everybody except one of these. You'll notice, if you look at the diagram, that I basically just added shards. That's it. That's the change. Same operator, same query engine, same query language, same mental model. Rebalancing after adding shards is manual, which is a real trade-off — most teams pre-provision or use weighting on Distributed tables to sidestep it. Materialized views for dashboard rollups shift from "nice to have" to "essential." Roughly $7–11K/month. The gap between ClickHouse and everything else opens up here. It doesn't close. This is where most solutions genuinely stop working, in the sense that even a well-staffed internal team cannot keep up with the operational load. If you've read this far, the point is probably already obvious, but I want to say it directly. Every observability stack works at 1 TB a day. If you're small, pick whatever your team already knows. Life is short. We're all just waiting for the robots to kick our heads off like soccer balls. The question is not which stack works today. The question is which stack still resembles itself two years from now, when your data volume has 5x'd and your team has 2x'd and the person who originally designed the whole thing has left the company. Elasticsearch mutates. LGTM mutates. Datadog stays operationally simple but mutates financially into something that requires its own dedicated team of accountants and pipeline engineers just to keep the bill from spiraling. ClickHouse just gets wider. You add shards. That's the whole trick. There is a real cost to this: you have to eat the schema-design and query-engine complexity up front, at a scale where the other options are objectively easier. You will be, briefly, the one making things harder for your developers. They will not always appreciate this. But the trade you're making is that their experience — and yours — remains roughly the same as the data grows by an order of magnitude, and the next order of magnitude, and probably the one after that. I have spent ten years watching observability stacks change shape underneath me while I tried to keep them running. ClickHouse is the first one that hasn't and that has been able to actually scale with me . That's pretty incredible. A relatively vanilla Elasticsearch cluster with Logstash providing some buffer between ingest and the Lucene indexes. Users get full-text search, which is genuinely good — this is the thing Elasticsearch is actually best at, and at this scale it delivers. Mapping explosions are already a background risk with mixed data, so dynamic mapping needs to be disabled or carefully templated from day one. ILM policies (hot → warm → delete) are non-optional even at this size, because forgetting to set them is how you get paged on a Saturday about disk pressure. Roughly $6–9K/month. Nothing too crazy. Alloy (formerly Grafana Agent, RIP) unifies the collection story into a single daemon, which is nice. Loki works well as long as you spend some time educating developers on how to attach useful labels — a conversation you will have many times, with many people, for the rest of your career. Mimir and Tempo largely do what it says on the tin. Roughly $3.5–5K/month. At 1 TB a day, Datadog is genuinely great. This is the scale it was built for, and it shows. You install the agent, you look at dashboards, you go home. There is almost nothing to think about, which is the entire point. You can already see the shape of the cost problem lurking in the diagram — the metered pipelines, the indexed-vs-ingested logs distinction, the custom metrics cardinality tax — but at this scale it's manageable. Roughly $45–75K/month, though negotiated pricing varies enough that I'd take that number with a grain of salt the size of a fist. Datadog's whole pricing philosophy is that they save you a full-time engineer. I think that framing is somewhat deranged, but they are extremely rich and I am not, so consider your source. Kafka is no longer optional. At 5 TB a day, direct writes into Elasticsearch cause bulk-reject storms and backpressure that will absolutely take your cluster down during a traffic spike. So now you're running Kafka, which means you're either running Kafka well or you're about to have a second, entirely different set of problems. Shard math becomes critical — at 50GB target shards, you're minting ~200 shards a day counting replicas, and your cluster state size becomes its own concern. You almost certainly need Elastic's commercial license for searchable snapshots and the frozen tier. Roughly $40–55K/month before licensing. That but Kafka You are now in microservices mode, whether you wanted to be or not. That means 65+ pods across three separate systems, each with its own compaction pipeline, its own hash ring, its own memcached tier. The gossip/memberlist ring becomes a real operational concern; ingester rollouts require careful -ingester.autoforget-unhealthy tuning, and if you get it wrong you either lose data or duplicate it. Roughly $22–32K/month. The operational complexity is still low, in that you don't run any servers. But you now need a full pipeline team whose entire job is reducing your Datadog bill. Exclusion filters, sampling rules, cardinality caps, tag allow-lists, the whole apparatus. This is what I call the "you build a system to avoid using the system you're paying for" trap, and once you're in it, you are in it forever. Roughly $180–350K/month, depending on how aggressive the pipeline team gets. This is also where you are basically fighting with your SaaS provider all the time, pouring over their billing documentation to figure out how to reduce costs. It's a hostile relationship and one I don't enjoy. You are now running three separate Elasticsearch clusters — one for logs, one for metrics, one for APM — federated through Cross-Cluster Search. Hot-tier NVMe cost dominates the bill. This is the scale at which teams start seriously evaluating alternatives, and where a lot of the recent migrations to ClickHouse have originated. Roughly $95–140K/month plus commercial licensing. You need people who are legitimate experts on Elasticsearch. Now thankfully Elastic just laid a ton of those people off, so they're probably possible to get, but still. Running this thing at this size is very complicated . Around 180+ pods, zone-aware everything, split-and-merge compaction, per-tenant limits, shuffle sharding to prevent noisy neighbors. You almost certainly have a dedicated observability platform team of three to five engineers at this point. If you don't, get ready for a bad fucking time. Roughly $55–85K/month. Still very easy to run, in the strict sense that you don't run anything. But your bill is now measured in six or seven figures a month, and the org has almost certainly built a pre-processing pipeline team whose entire existence is dedicated to reducing that bill. Most companies at this scale have gone hybrid: Datadog for APM and high-value metrics, self-hosted (increasingly ClickHouse) for logs. The complexity paradox at this scale is that you now have Datadog's simplicity plus your pipeline complexity plus a second self-hosted stack. Pricing is all over the goddamn place. You might be over a $1 million a month here. Look at the diagram and then look back at the 1 TB diagram. It's the same diagram. There are more shards. That's the difference. Materialized views for rollups are now mandatory rather than optional. Schema design mistakes you made two years ago will start to hurt, so hopefully you didn't make many. Rebalancing after adding shards is still manual; most teams pre-provision or use clickhouse-copier or a dual-write migration when they need to grow the cluster. Kafka starts to become useful as a buffer for very bursty ingest, though it's not required. Roughly $18–28K/month.

Database Sql JSON DevOps

▲ 0

0 views

matduggan.com 3 months ago

The Intolerable Hypocrisy of Cyberlibertarianism

I like the Internet. I am old enough to remember the pre-Internet era and despite the younger generations pining for those simpler days, I was there. Paper maps were absolutely horrible, just you and a compass in your car on the side of the road in the middle of the night trying to figure out where you are and where you are going. Once when driving from Michigan to Florida I got so lost in the middle of the night in Kentucky that I had to pull over to sleep and wait for the sun so I could figure out where I was. I awoke to an old man staring unblinkingly into my car, shirtless, breathing heavy enough to fog the windows. To say I floored that 1991 Honda Civic is an understatement. You would leave your house and then just disappear. This is presented as kind of romantic now, as if we were just free spirits on the wind and could stop and really watch a sunset. In practice it was mostly an annoying game of attempting to guess where people were. You'd call their job, they had left. You'd call their house, they weren't home yet. Presumably they were in transit but you actually had no idea. As a child my response to people asking me where my parents were was often a shrug as I resumed attempting to eat my weight in shoplifted candy or make homemade napalm with gasoline and styrofoam. Sometimes I shudder as a parent remembering how young I was putting pennies on train tracks and hiding dangerously close so that we could get the cool squished penny afterwards. Cassettes are the worst way to listen to music ever invented. Tapes squealed. Tapes slowed down for no reason, like they were depressed. Multiple times in my life I would set off on a long road trip, pop in a tape, and within fifteen minutes watch as it shot from the deck unspooled like the guts from the tauntaun in Star Wars . You'd then spend forty-five minutes at a Sunoco trying to wind it back in with a Bic pen knowing in your heart you were performing CPR on a corpse. Then you'd put it back in the player out of pure stubbornness, and it would chew itself again immediately, and you'd drive the next six hours in silence with your own thoughts, which were not as good as Pearl Jam. So I am, mostly, grateful for the bounty the internet has provided. But there is something wrong, deeply wrong, with what we built. The wrongness was there at the start. It was baked into the foundation by people who told themselves a story about freedom, and that story was a lie, and we are all, every one of us, paying their tab. To understand what happened we need to go back to the 90s. One of the first and most classic examples of the ideology that powered and continues to power tech is the classic "A Declaration of the Independence of Cyberspace" by John Perry Barlow written in 1996. You can find the full text here . I remember thinking it was genius when I first read it. I was young enough that I also thought "Snow Crash" was a serious political document. Today the Declaration reads like one of those sovereign citizen TikToks where someone in traffic court is claiming diplomatic immunity under maritime law. It helps to know who Barlow was. Barlow was a Grateful Dead lyricist. He was also a Wyoming cattle rancher. He was also, briefly, the campaign manager for Dick Cheney's first run for Congress. (You did not misread that.) He spent his later years as a fixture at Davos, the World Economic Forum, where the very wealthy gather each January to remind each other that they are interesting. It was at Davos, in February 1996, fueled by champagne and grievance over the Telecommunications Act, that Barlow banged out the Declaration on a laptop and emailed it to a few hundred friends. From there it became, somehow, one of the founding documents of the modern internet. Many of the pillars of "modern Internet" are here. Identity isn't a fixed concept based on government ID but is a more fluid concept. We don't need centralized control or really any form of control because those things are unnecessary. It was this and the famous earlier "Cyberspace and the American Dream: A Magna Carta for the Knowledge Age" that laid a familiar foundation for a lot of the culture we now have. [ link ] The Magna Carta is also our introduction to the (now familiar) creed of "catch up or get left behind". The adoption of new technology must be done at the absolute fastest speed possible with no regulations or checks. You don't need to worry about the consequences of technology because these problems correct themselves. If you told me the following was written two weeks ago by OpenAI I would have believed you. The cumbersome copyright/patent process. Cumbersome to whom, exactly? This is always the move. The thing your industry would prefer not to deal with is reframed as an obsolete burden. Your refusal to do it is rebranded as innovation. Your inability to imagine a world where you don't get exactly what you want becomes a manifesto. So there are dozens of these pieces and they all read the same. If you don't regulate these technologies humanity will only benefit. Education, healthcare, industry, etc. We don't need regulations because the transformation from the medium of paper to digital has transformed the human spirit. But one was extremely surprising to me. Langdon Winner wrote something almost prophetic back in 1997. You can read it here . He coins the term cyberlibertarianism (or at least is the first mention of it I could find) and then goes on to describe an almost eerily accurate set of events. In all he lays out 4 pillars of this ideology. Technological determinism. The new technology is going to transform everything, it cannot be stopped, and your only job is to keep up. Stewart Brand's actual quote, which Winner pulls out and lets sit there like a body on display, is "Technology is rapidly accelerating and you have to keep up." There's no room to ask whether we want any of this. The wave is coming. Surf or drown. It does not occur to anyone in this discourse that 'drown' is a choice the wave is making, not a natural law. Waves do not have intentions. Destroying your livelihood and leaving you to rot isn't a requirement of the natural order as much as that would convenient. Radical individualism. The point of all this technology is personal liberation. Anything that gets in the way of the individual maximizing themselves be it government, regulation, social obligation, your annoying neighbors, is an obstacle to be removed. Winner notes, with what I imagine was a very dry expression, that the writers of the "Magna Carta for the Knowledge Age" cited Ayn Rand approvingly. In 1994. As intellectual grounding. For a document about computers. There is something deeply funny about a movement claiming to invent the future and grounding its case in a Russian émigré's airport novels about steel barons in love with their own reflections. Free-market absolutism. Specifically the Milton Friedman, Chicago School, supply-side flavor. The market will sort it out. Regulation is theft. Wealth is virtue. George Gilder, who co-wrote the Magna Carta, had previously written a book called Wealth and Poverty that helped sell Reaganomics to the masses. He then wrote Microcosm , which argued that microprocessors plus deregulated capitalism would liberate humanity. He was very serious about this. Don't worry, Gilder is still out there. He loves the blockchain and crypto now. He now writes about how Bitcoin will save the soul of capitalism, which it is somehow doing while also destroying the planet. Both can be true in his cosmology. The ideology is flexible like that. A fantasy of communitarian outcomes. This is the part that should make you laugh out loud. After establishing that government is bad, regulation is theft, and the individual is sovereign, the cyberlibertarians then promise that the result of all this will be... rich, decentralized, harmonious community life. Negroponte: "It can flatten organizations, globalize society, decentralize control, and help harmonize people." Democracy will flourish. The gap between rich and poor will close. The lion will lie down with the lamb, and the lamb will have a Pentium II. We also have the advantage of hindsight and know, without question, that all of these predicted outcomes were wrong. Not 'directionally wrong' or 'wrong in the details.' Wrong the way it would be wrong to predict that if you set your kitchen on fire, the result will be a renovation. You have to hold these four ideas in your head at the same time to see the trick. The cyberlibertarians wanted you to believe that radical individualism plus deregulated capitalism plus inevitable technology would produce communitarian utopia. This is, on its face, insane. It is the economic equivalent of claiming that if everyone punches each other really hard, eventually we'll all be hugging. But Winner's sharpest observation, the one I keep coming back to, isn't about any of the four pillars individually. It's about the move underneath them. He writes: This is the entire game. This is how "don't tread on me" becomes "Meta should be allowed to do whatever it wants." This is how the rights of the lone hacker working in their garage become indistinguishable from the rights of a multinational with a market cap larger than most countries' GDP. The Magna Carta literally argues that the government should reduce barriers to collaboration between cable companies and phone companies in the name of individual freedom and social equality . Winner caught this in 1997. What makes the essay uncomfortable to read now is that Winner wasn't even predicting the future. He was just describing what was already happening and noting where it would obviously lead. He saw the media mergers and asked the question nobody in the industry wanted to answer: what happened to the predicted collapse of large centralized structures in the age of electronic media? Where, exactly, did the decentralization go? He saw that the cyberlibertarians were going to deliver the opposite of everything they promised, and that they were going to keep getting paid to promise it anyway. He was writing before Google. Before Facebook. Before the iPhone. Before YouTube. Before Twitter, Bitcoin, Uber, AirBnB, OpenAI, and the entire app economy. Before any of the actual examples that would eventually prove him right existed. He just looked at the people doing the talking, listened to what they were saying, and wrote down where it ended. It is not a long essay. He didn't need a long essay. The future was right there on the page, in their own words. He just had to read it back to them. The essay closes with a question that has, to my knowledge, never been seriously answered by the industry it was aimed at: Twenty-eight years later, the industry still treats this question as somewhere between naive and seditious. It's the question Barlow's declaration was specifically designed to make unaskable. And it remains, to this day, the only question that actually matters. When you look at these early formative writings, so much of what we see now becomes clear. The cyberlibertarian deal was always the same: you're on your own. The industry would build the infrastructure, take the profits, and shove every consequence, every harm, every cost, every responsibility, onto somebody else. There is no greater example to me than the moderator. Anyone who has ever moderated a forum or a subreddit knows that adding the word "cyber" to a space doesn't suddenly turn people into better humans. People are still people. They flame each other, they post slurs, they doxx, they harass, they spam, they post CSAM, they radicalize each other, they grief, they coordinate, they lie. A space with humans in it requires governance. They produce, with frightening regularity, the exact behavior any kindergarten teacher could have predicted. Then they act surprised. But the cyberlibertarian model required pretending it was unforeseeable. The platforms couldn't acknowledge that they needed governance because acknowledging it would mean acknowledging responsibility, and acknowledging responsibility would mean acknowledging liability, and acknowledging liability would mean the entire economic model collapses. So instead the industry invented a beautiful fiction: governance happens, but it happens by magic, performed by volunteers, for free, who we will simultaneously rely on and mock. Reddit is run by unpaid moderators. Wikipedia is run by unpaid editors. Stack Overflow was run by unpaid experts and is now a ghost town. On TikTok and Twitter it is the unknowable "algorithm" that is the cause of and solution to every problem backed by capricious moderators who delight in stopping free speech. Unless you don't like it, then it's negligence moderation in defense of your enemies. Open source is run by unpaid maintainers having nervous breakdowns. The platforms collect the rent. The people doing the actual work of making the platforms livable get nothing, and when they ask for anything like recognition, tools, basic protection from harassment, they're told they're power-tripping nerds who should touch grass. This is also the crypto story, just with the masks off. What if we made worse money on purpose, money that bypassed every protection consumers had won over the previous century, money that couldn't be reversed when stolen, money that funded ransomware attacks on hospitals and pump-and-dumps targeting people's retirement accounts? The cyberlibertarian answer was: that's freedom. The losses were real. People killed themselves. Hospitals had to turn away patients. The architects became billionaires and bought yachts and now sit on the boards of AI companies, where they are reinventing the same con with a new vocabulary. Now Winner got one thing wrong, and it's worth pausing on, because it's the most interesting wrinkle in all of this. What actually happened was weirder and worse. The cyberlibertarians became the corporations. They didn't sell out. They didn't betray their principles for the first offer of money. They simply scaled until their principles became inconvenient, and then they stopped mentioning them. Once the platforms got large enough to be unstoppable, once they captured enough of the regulatory apparatus to write their own rules, the libertarian rhetoric got quietly shelved like a college poster you took down before your in-laws came over. Meta no longer pretends it stands for free speech and seemingly takes delight in putting its thumb on the scale. TikTok users have invented an entire euphemistic shadow language to evade automated censorship like "unalive," "le dollar bean," "graped" that would have made 1996 Barlow weep into his bolo tie. Copyright and patents matter when they're Apple's copyright and patents. Or Googles. Or OpenAIs. Go try to make a Facebook+ website and see how quickly Meta is capable of responding to content it finds objectionable. Cyberlibertarianism was the ladder. Once they were on the roof, they kicked it away and started charging admission to look at the view. Remember I like the Internet. I said it in the beginning and it is still true. I love the Fediverse, I love weird Discords about small tabletop RPGs I'm in. I spend hours in the Mister FPGA forums. There are corners that are good. But they're mostly good because they're not big enough to be worth breaking up. It feels increasingly like I'm hanging out in the old neighborhood dive bar after most of the regulars have moved away. The lighting is the same. The bartender remembers your order. But you can hear yourself think now, and that's mostly because the room is half empty and the jukebox finally died. The new clientele is from out of town. They are taking pictures of the menu. If we want to have a serious conversation about why we are in the situation we're in, it is no longer possible to pretend that the broken ideology that put us on this trajectory is still somehow compatible with the harsh realities that surround us. It is not clear to me if democracy can survive a deregulated Internet. A deregulated Internet filled with LLMs that can perfectly impersonate human beings powered by unregulated corporations with zero ethical guidelines seems like a somewhat obvious problem. Like an episode of Star Trek where you the viewer are like "well clearly the Zorkians can't keep the Killbots as pets." It doesn't take some giant intellect to see the pretty fucking obvious problem . If we want to save the parts of the internet worth saving, we have to evolve. We have to find some sort of ethical code that says: just because I can do something and it makes money, that is not sufficient justification to unleash it on the world. Or, more simply: just because I want to do something and you cannot actively stop me, that does not make doing it a good idea. We have waited thirty years for the cyberlibertarian future to arrive and produce the promised harmonious community. It's time to face the facts. It's never coming. The bus left in 1996. The bus was never real. People did not get better because they went online. Giving everyone access to a raw, unfiltered pipeline of every fact and lie ever produced did not turn them into better-educated people. It broke them. It allowed them to choose the reality they now inhabit, like ordering off a menu. If I want to believe the world is flat, TikTok will gladly serve me that content all day. Meta will recommend supportive groups. There will be hashtags. There will be Discords. There will be a guy named Trent who runs a podcast. I will never have to face the deeply uncomfortable possibility that I might be wrong about anything, ever, until the day I die, surrounded by people who agree with me about everything, including which of the other mourners are secretly lizards. That is the internet we built. It was not an accident. It was the product of a specific ideology, written down by specific people, at a specific cocktail party in Davos, in 1996. Winner watched it happen and told us where it was going. We did not listen. There is still time, maybe, to start.

Culture

▲ 0

0 views

matduggan.com 4 months ago

If I Could Make My Own GitHub

My friend and I have a game where we talk about what we'd do if we were rich. Not rich like 'paid off the mortgage' rich. Rich like a man who owns a submarine he's never been inside. Rich like a man whose third wife has a skincare line. Tech-titan rich — the kind of money that buys you a compound in Wyoming and the confidence to wear the same gray t-shirt to congressional testimony." One of mine, for a long time, has been the dream of making a new forge. I was prompted to write this after reading the good post about Ghostty leaving GitHub but it's something I've written and talked about for a few years. Given how bad GitHub has become at its core job, it seemed like a fun opportunity to try and write up what my billionaire folly of a forge would look like. This folly would have less penile rockets filled with aging celebrities. GitHub, GitLab and Gitea (those being the 3 I've used the most) are all modeled on effectively the same design. There are differences, but you can tell that GitHub sets the pattern for the industry and then those features are ported over to the other two with varying levels of success. The issue with all of these is that they're designed to add things git doesn't do that you need. Git is great at what it is designed to do, but what it is designed to do isn't the way most people are using it. Git is a perfect tool for kernel development. It is a decentralized distributed version control system that relies on the idea of patches being sent to maintainers over email. You trust those maintainers to maintain their sections and merge in the stuff that makes sense and not merge in the other stuff. It's a pretty high trust environment that places very few restrictions on how online a specific contributor is or what system they are using. If you have a laptop from 2010 that connects to the Internet once a week you can still be a meaningful contributor to a project with these workflows. . However, in most jobs, git is effectively just the way I pull and push from a centralized repository stored in a forge. All the important stuff happens inside the forge, and very little of it happens on my client. Pull Requests are how I enforce the four-eyes principle, GitHub Actions are how I run my tests and linting on those Pull Requests to ensure they are functional and meet my organizational requirements, the user's identity in relation to that forge is how I verify who they are. I track issues with my code through Issues and cut releases for users to download through Releases. There's not a lot of git in this workflow, this is mostly placed on top of git. So here are the primary issues I see with modern forges that I would love to solve. Absolutely. There are a lot of tools that do parts of this. I want someone to take them, put them all together and fit them up. I want JJ as the VCS, I want this as the forge and I want the expectation that I as a user could live happily with a raspberry pi as a forge for a long time. I want those forges designed around modern concepts like object storage and shallow clones and getting constantly hammered by LLM bots. Now in a universe where GitHub was doing a good job, I wouldn't even bother writing this up. GitHub is the default and talking to people about overcoming the default is usually a waste of time. Heinz is the default ketchup, when I order a Coke I don't want a Pepsi and if I'm going to use a forge up until 2026 there would have to be an amazing reason for me to not choose the one that everyone uses. Up until recently other forges have been like sweet potato french fries, which is to say never the thing you actually want. But we live in a world where the monolithic forge is breaking down and nobody has built the replacement. The people with the money are busy with the rockets. The people with the taste are busy with their day jobs. And the rest of us are opening PRs titled 'asdfasdf' at midnight, waiting for a robot to check them, wondering when the tool we spend our whole working lives inside stopped being built for us. If I ever get the submarine money, I'll let you know. Stuff happens in the wrong order. You know the PR. Commit 1: 'Feature.' Commit 2: 'fix.' Commit 3: 'fix.' Commit 4: 'actually fix.' Commit 5: 'please.' Commit 6, made at 11:47 PM on a Thursday: 'asdfasdf'. This person has a family. This person has hobbies. This person is, at this moment, crying. You don't want the feedback loop after the commit you want it before. Let me do an enforced pre-commit hook to run the jobs remotely on the forge and provide the feedback to the user before they push. PR approval is too boolean. The PR is approved or it's not approved. Real code review, like real life, lives in the middle. 'Sure, fine, we'll deal with it later' is a legitimate human response and should be a legitimate button. Gerrit has a better model for this. If I weakly approve something as a maintainer, let me flag it for later. PRs are too inflexible. I don't need 4 eyes on every change, especially in a universe where LLMs exist. The global GDP lost annually to senior engineers staring at a four-line PR waiting for someone — anyone — to type 'LGTM' could fund a moon mission. A nice one. With legroom. Let me customize and more easily control this. If the person is a maintainer and the LLM says its low risk/no risk just let them go. Stacked PRs are just better. They're easier to review and understand. They have to be a first-class citizen not an add-on through a tool other than your VCS. A forge shouldn't do everything. Issue tracking yes. Kanban board, probably not. Wiki? I doubt it. Everything tools always turn into crap. You add features when its easy to add features and then pay the maintenance price for those features forever regardless of their rate of adoption because now someone, somewhere uses them and you are locked in. The standard unit of hosting is too large. Running Github Enterprise is a big task. Running GitLab is also a relatively big ask. These are complicated products with a lot of moving pieces. I want smaller individual units of hosting that I can link together to make an organization. It's fine if they're not globally federated and I need to make an account for each Organization, but an Organization should be flexible enough to let me say "these 12 Raspberry Pis are my org". I don't know how they communicate securely, I hire people for that problem. My local copy of the repo should be a representation of the entire repo, not just the code. I should be able to approve a PR from the same VCS I use to check in the code. I should be able to go through my issues by looking through local files. On the flip side, since I need to be online all the time to really work with a team, don't make me pay the storage price all the time. I want my VCS and my forge to work together. If I clone a repo, I want a pretty limited history for that repo when I clone. If I start to go back in time, spin up a worker to go fetch that stuff from the VCS when I need it. I don't need to hammer the forge with giant clone requests on the assumption that I might need to rebuild the forge at any moment with the entire history of the entire project. Actions need to be signed, SHA'd and usable offline. If I want, I should be able to get tarballs of all my actions, stick them in the repo and tell my system "don't go anywhere for checkout action, that's right here". If I say latest, have that work like Dependabot does now where it opens up a PR to put the latest tarballs in my repo. Actions are critical and they should be runnable on my local machine through the same VCS if I want to.

Open Source Git

▲ 0

0 views

matduggan.com 4 months ago

You can absolutely have an RSS dependent website in 2026

I write stuff here. Sometimes the stuff is good. Sometimes it reads like I wrote it at 2 AM after an argument with a YAML file, which is because I did. But one decision I made early on was that I didn't want to offer an email newsletter. Part of this was simple economics. At one point I did have a Subscribe button up, and enough people clicked it that the cost of actually sending those emails started to resemble a real bill. Sending thousands of emails when you have no ads, no sponsors, and no monetization strategy beyond "I guess people will just... read it?" doesn't make a lot of financial sense. But the bigger reason — the one I actually care about — is that I didn't want a database full of email addresses sitting under my control if I could possibly avoid it. There's a particular flavor of anxiety that comes with being the custodian of other people's personal data, a low-grade dread not unlike realizing you've been entrusted with someone's elderly cat for two weeks and the cat has a medical condition. I can't lose data I don't have. I never need to lie awake wondering whether some user is reusing their bank password to log into my website just to manage their subscription preferences. The best way I can safeguard user data is by never having any in the first place. It's not a security strategy you'll find in any textbook, but it is airtight. Now, when I explained this philosophy to people who run similar websites, the reaction was — and I'm being generous here — warm laughter . The kind of laughter you get when you ask if an apartment in Copenhagen is under $1,000,000. Email newsletters are the only way to run a site like this, they said. RSS is dead, they said. You might as well be distributing your writing via carrier pigeon or community bulletin board. One person looked at me the way you'd look at someone who just announced they were going to navigate cross-country using only a paper atlas. Not angry. Just sad. I'm lucky in that I'm not trying to get anyone to pay me to come here. If I were, the math would probably change. I'd be out there A/B testing subject lines and agonizing over open rates like everyone else, slowly losing pieces of my soul in a spreadsheet. But if your question is simply, "Can I make a hobbyist website that actual humans will find and read without an email newsletter?" — the answer is a resounding yes. And I have the logs to prove it. All of this is from Nginx access.log. These logs get rotated daily and don't include the majority of requests that hit the Cloudflare cache before they ever reach my server, so the real numbers are higher. But I think they're reasonably representative of the overall shape of things. About half my traffic is readers hitting or — people who have, of their own free will, pointed an RSS reader at my site and said yes, tell me when this person has opinions again . The other half are arriving via a specific link they stumbled across somewhere in the wild. If we do a deeper dive into that specific RSS traffic, we learn a few interesting things. The user-agent breakdown shows the usual suspects — the RSS readers you'd expect, the ones that have been around long enough to have their own Wikipedia articles. There are also some abusers in the metrics. I have no idea what "Daily-AI-Morning" is, but whatever it's doing, it's polling my feed with the frantic energy of someone refreshing a package tracking page on delivery day. The time distribution, though, is pretty good — spread out across the day in a way that suggests real humans checking their feeds at real human intervals, rather than a single bot hammering me every thirty seconds. My conclusion is this: if you want to run a website that relies primarily on RSS instead of email newsletters, you absolutely can. The list of RSS readers hasn't dramatically changed in a long time, which is actually reassuring — it means the ecosystem is stable, not dead. The people who use RSS really use RSS. They're not trend-chasers. They're the type who still have a working bookmark toolbar. They are, in the best possible sense, your people. Effectively, if you make your site RSS-friendly and you test it in NetNewsWire, you will — slowly, quietly, without a single "SUBSCRIBE FOR MORE" pop-up — build a real audience of people who actually want to read what you write. No email database required. No passwords to leak. No giant confusing subscription system.

Web Development Blogging Yaml

▲ 0

0 views

matduggan.com 5 months ago

I Can't See Apple's Vision

I don't typically write about Apple stuff. It's the most written-about company on earth. Every product launch gets the kind of forensic scrutiny normally reserved for plane crashes and celebrity divorces. Mostly though, I feel like a line cook at a Denny's talking trash about whether the French Laundry has lost their way. I'm back here microwaving a Grand Slam and opining about Thomas Keller's sauce work. The engineers I know personally at Apple are, on average, much more talented than me. They work harder, they do it for decades without a break, and none of them have ever shipped a feature while still wearing pajama pants at 2 PM. It seems insane for someone of my mediocre talent to critique them. It also feels a little dog-pile-y. Apple employees know Tahoe sucks. They know it the way you know your haircut is bad — they don't need strangers on the internet confirming it. And to be fair, there's genuinely great work buried inside Tahoe: the clipboard manager, the automation APIs, a much-improved Spotlight. But visually it's gross, and that matters when your entire brand identity is "we're the ones who care about design." Instead, I want to talk about a bigger problem and one that I do feel qualified to talk about because I am very guilty of committing this sin. I don't see a cohesive vision for MacOS and WatchOS. This, more than one bad release, seems far worse to me and dangerous for the company. Since this is already 2000 words as a draft I'll save WatchOS for another time. I'm verbose but even I have limits. Now to be clear this isn't across every product . iPadOS has a strong vision and have the strength of their convictions to change approaches. The different stabs at solving the window problem inside of the iPad and make it so that you still have an iPad experience while being able to do multiple things at the same time is proof of that. iOS has an incredibly strong vision for what the product is and isn't and how the software works with that. VisionOS and tvOS are less strong, but visionOS is still finding its footing in a brand new world. The Apple TV hardware and software is in a weirdly good position even though nothing has changed about it in what feels like geological time. I've purchased every version of the Apple TV, and with the exception of that black glass remote — the one that felt like it was designed by someone who had never held a remote, or possibly a physical object — everything has been pretty good. I'm still not clear how storage works on the Apple TV and I don't think anybody outside of Apple does either. I'm not even sure Apple knows. But somehow it's fine. But with watchOS and MacOS we have 2 software stacks that seem to be letting down the great hardware they are installed in. They seem to be evolving in random directions with no clear end goal in mind. I used to be able to see what OS X was aiming for, even if it didn't hit that goal. Now with two of Apple's platform I'm not able to see anything except a desire to come up with something to show as this years release. When I got my first Mac — an iBook G3 — the experience was like test-driving a Ferrari that someone had fitted with a lawnmower engine. You'd click on the hard drive icon and wait. And wait. And in those few seconds of waiting, you'd think: man, this would be incredible if the hardware could keep up. The software had somewhere it wanted to go. The hardware just couldn't get it there yet. This trend continued for a long time on OS X, where you'd see Apple really pushing the absolute limits of what it could get away with. After the rock solid stability of 10.4 Apple took a lot of swings with 10.5 and they didn't all land. The first time you opened the Time Machine UI and the entire thing crawled to an almost crash, you'd think boy maybe this wasn't quite ready for prime time . But this entire time there wasn't really a question, ever, that there was a vision for what this looked like. The progression of OS X from the beta onward was this: OS X tried to accommodate you, not the other way around. When you look at these screenshots I'm always surprised how light the touch is. There isn't a lot of OS here to the user. Almost everything is happening behind the scenes and the stuff you do see is pretty obvious. The first time I thought "oh man, they've lost the thread" was Notifications. On iOS, Notifications make sense — you've got apps buried in folders three screens deep, so a unified system for surfacing what's happening is genuinely useful. On macOS, this design makes absolutely no sense at all. You can see your applications. They're right there. In the Dock. Which is also right there. This is the beginning of this feeling of "we aren't sure what we're doing here with the Mac anymore". iOS users like Notifications so maybe you dorks will too? It consumes a huge amount of screen real estate, it was never (and still isn't) clear what should and shouldn't be a notification. Even opening up mine right now it's filled with garbage that doesn't make sense to notify me about. A thing has completed running the thing that I asked it to run? Why would I need to know that? There is also already a clear way to communicate this information to me. The application icon adds an exclamation point or bounces up and down in the dock. With Notifications you end up with just garbage noise taking up your screen for no reason. Maybe worse, it's not even garbage designed with the Mac in mind. It's just like random crap nobody cares about that looks exactly like iOS Notifications. The issue with copying everything from iOS is that it's like copying someone's homework — except they go to a different school, in a different country, studying a different subject. It's not just wrong in the way where you tried and failed. It's wrong in a way that makes everyone who encounters it deeply uncomfortable. The teacher doesn't even know where to begin. They just stare at it. For years afterwards it seemed like the purpose of MacOS was just to port iOS features to the Mac years after their launch on iOS. Often these didn't make much sense or hadn't had a lot of effort expended in making them very Mac-y. Like there was clearly a favorite child with iOS, then a sassy middle child with iPadOS and then, like a 1980s sitcom where there was a contract dispute, "another child" you saw every 5th episode run down the stairs in the background with no lines. Me at home would shout at my TV "I knew they didn't kill you off MacOS!". Now with Tahoe there's clearly some sort of struggle happening inside of the team. And here's what's maddening — buried inside this visual catastrophe, someone at Apple is doing incredible work. Clipboard management has been table stakes in the third-party ecosystem for years. Apple finally added a version that handles 90% of use cases. It's classic Sherlocking: Apple shows up ten years late to the party, brings a decent bottle of wine, and somehow half the guests leave with them. Same with Spotlight. Spotlight hasn't gotten a ton of love in years. Suddenly it's really competing with third-party tools. If you're searching for a file, you can filter it based on where the file is stored. Type "name of Directory" press the Tab key, and then type the name of the file before pressing Enter. This is great! We finally have keyword search for stuff like . Application shortcuts for opening stuff with things like for Firefox is nice. Assign a quick key like “se” to  Send Email . Type it in Spotlight, hit enter, and compose your message. This is all classic Apple thinking which is "how can we make the Mac as good as possible such that you, the user, don't need to download any third-party applications to get a nice experience". You don't need a word processor, you have a word processor and a spreadsheet application and presentation software and a PDF viewer and a clipboard manager and a system launcher and automation APIs etc etc etc. This is a vision that is consistent throughout the entire systems history, how can we help you do the things you need to do more easily. But the reason why I'm stressed as someone who is pretty invested in the ecosystem is that the visual stuff is so bad and not just bad, but negligent. We didn't test how it was gonna look under a bunch of situations so that's now someone else's problem. Whenever I get a finder sidebar covering folder contents so I had to resize the window every time, or the Dock freaks out and refuses to come back out, it feels like I installed one of those OS X skins for a Linux distro. I buy Apple stuff cause its nice to look at and this is horrible to look at. Why is this so big? Why did you cut off the word "Finder" from Force Quit? Everywhere you look there's a million of these papercuts. We have a resolution on our laptops screen that would have made people collapse in 2005 why must we waste all of it on UI elements? Also you can't grab window edges as shown by the best post ever written here: https://noheger.at/blog/2026/01/11/the-struggle-of-resizing-windows-on-macos-tahoe/ Why is there so much empty space between everything? Why are there six ways to do literally everything? Why did we copy the concept of Control Center from iOS at all if there's very little limit on screen real estate and we could already do this from the menu bar? So we're going to keep the Mac menu bar but we're going to add a full iPad control system and then we're going to use the iPad control system to manage the menu bar . I will say the "Start Screen Saver" makes me laugh because its a mistake I would make in CSS. The text is too long so the button is giant but we didn't resize the icon so it looks crazy. Now do we need the same text inside the button as outside of it? No, and that leads me to the other banger. It's pretty clear the two white boxes inside of "Scene or Accessory" were supposed to be text, Scene on the top and then Accessory on the bottom, but SwiftUI couldn't do that so they left the placeholder. Somewhere there is a Jira ticket to come back to this that got trashed. Also, complete aside. Has anyone in the entire fucking world ever run Shazam from a Mac? What scenario are we designing for here? I hear a banger at the coffee shop so I hold my MacBook Pro up over my head like John Cusack in Say Anything , hoping it catches enough audio before my arms give out? "Recognize Music" is in my menu bar, taking up space that could be used for literally anything else, on the off chance I need to identify a song using a device that weighs four pounds and has no microphone worth using in a noisy room. If you are going to copy ipadOS's homework you need to think about it for 30 seconds . So my hope is that the improvement camp wins. That the people who built the better Spotlight and the clipboard manager and the automation APIs are the ones who get to set the direction. Because right now it feels like the best work on macOS is being done in spite of the overall vision, not because of it. Like someone's sneaking vegetables into a toddler's mac and cheese. The good stuff is in there — you just have to eat around a lot of neon orange nonsense to find it. Steve Jobs talked about creative people having to persuade five layers of management to do what they know is right. I don't know how many layers there are now. But I know what it looks like when the creative people are losing that argument, and I know what it looks like when they're winning it. Right now, on macOS, it looks like both are happening at the same time, in the same release, on the same screen. And that's scarier than any one bad design choice. It's Unix, but you never need to know that. All the power, none of the beard. You get the stability of a server OS without ever having to type into anything. Everything annoying is abstracted away. Drivers? Gone. "Installing" an application? You drag it into a folder. That's it. That's the install. It felt like the computer was meeting you more than halfway — it was practically doing your job for you and then apologizing for not doing it sooner. If it seems like it should work, it works. Double-click a PDF, it opens. Put in a DVD, it plays. Drag an app to the Applications folder and it becomes an application. This sounds obvious now, but in 2003 this was like witchcraft if you were coming from Windows. But it was also serious. It wasn't cluttered with stupid bullshit. It was designed for people who made things — with real font management, color calibration, the works. The OS tried to stay out of your way. Your content was the show; everything else was stagecraft.

CSS Design

▲ 0

0 views

matduggan.com 5 months ago

Hosting a Snowflake Proxy

In the nightmarish world of 2026 it can be difficult to know how to help at all. There are too many horrors happening to quickly to know where one can inject even a small amount of assistance. However I wanted to quickly post about something I did that was easy, low impact and hopefully helps a tiny fraction of a fraction of a percent of people. So I was browsing Mastodon when someone posted a link asking for people to host Snowflake proxies. Snowflake is a lightweight proxy best explained by David Fifield below. Effectively it is a lightweight and easy to run way to bypass censorship that doesn't require running a VPN and involves almost zero technical knowledge. It's quite the design and one that I kept shaking my head thinking "man I never would have thought of this in a mission years" as I read more about how it works. So I have a box sitting on an internet connection where I'm lucky enough to have plenty of excess capacity. I figured "why not share it". I thought I'd post the process here in case people were curious but were worried about how much bandwidth it might use or how many resources. Setting it up on a Debian box took like 5 minutes. That's it. So this has been running for two weeks and in that two weeks I've served up the following amount of traffic: CPU usage is quite low, Memory is slightly higher than I would have thought but that's likely a function of running for so long. Remember you can modify the systemd service file to limit memory if you are interested in running this yourself but are concerned about crossing a gig of memory. All in all I haven't noticed I'm running this at all. Obviously its great to run the browser extension to increase the pool of IP addresses and keep them from becoming static and blockable, but if you have a dedicated box with a large amount of bandwidth and are looking for a quick 20 minute project to help out people trying to deal with internet censorship, this seems like a good one to me. Get the package from here: https://packages.debian.org/sid/snowflake-proxy with: Make sure it is enabled and running

Open Source

▲ 0

0 views

matduggan.com 5 months ago

Markdown Ate The World

I have always enjoyed the act of typing words and seeing them come up on screen. While my favorite word processor of all time might be WordPerfect ( here ), I've used almost all of them. These programs were what sold me on the entire value proposition of computers. They were like typewriters, which I had used in school, except easier in every single way. You could delete things. You could move paragraphs around. It felt like cheating, and I loved it. As time has gone up what makes up a "document" in word processing has increased in complexity. This grew as word processors moved on from being proxies for typewriters and into something closer to a publishing suite. In the beginning programs like WordPerfect, WordStar, MultiMate, etc had flat binary files with proprietary formatting codes embedded in there. When word processors were just proxies for typewriters, this made a lot of sense. But as Microsoft Word took off in popularity and quickly established itself as the dominant word processor, we saw the rise of the .doc file format. This was an exponential increase in complexity from what came before, which made sense because suddenly word processors were becoming "everything tools" — not just typing, but layout, images, revision tracking, embedded objects, and whatever else Microsoft could cram in there. At its base the is a Compound File Binary Format, which is effectively just a FAT file system with the file broken into sectors that are chained together with a File Allocation Table. It's an interesting design. A normal file system would end up with sort of a mess of files to try and contain everything that the has, but if you store all of that inside of a simplified file system contained within one file then you could optimize for performance and reduced the overhead that comes with storing separate objects in a flat file. It also optimizes writes, because you don't need to rewrite the entire file when you add an object and it keeps it simple to keep revision history. But from a user perspective, they're "just" dealing with a single file. ( Reference ) The .doc exploded and quickly became the default file format for humanity's written output. School papers, office memos, résumés, the Great American Novel your uncle was definitely going to finish — all .doc files. But there was a problem with these files. They would become corrupted all of the goddamn time. Remember, these were critical documents traveling from spinning rust drives on machines that crashed constantly compared to modern computers, often copied to floppy disks or later to cheap thumb drives you got from random vendor giveaways at conferences, and then carried to other computers in backpacks and coat pockets. The entire workflow had the structural integrity of a sandwich bag full of soup. So when Word was saving your critical file, it was actually doing a bunch of different operations. It was: These weren't atomic operations so it was super easy in an era when computers constantly crashed or had problems to end up in a situation where some structures were updated and others weren't. Compared to like a file where you would either get the old version or a truncated new version. You might lose content, but you almost never ended up with an unreadable file. With as someone doing like helpdesk IT, you constantly ended up with people that had just corrupted unreadable files. And here's the part that really twisted the knife: the longer you worked on the same file, the more important that file likely was. But Word didn't clean up after itself. As a .doc accumulated images, tracked changes, and revision history, the internal structure grew more complex and the file got larger. But even when you deleted content from the document, the data wasn't actually removed from the file. It was marked as free space internally but left sitting there, like furniture you moved to the curb that nobody ever picked up. The file bloated. The internal fragmentation worsened. And the probability of corruption increased in direct proportion to how much you cared about the contents. Users had to be trained both to save the file often (as AutoRecover wasn't reliable enough) and to periodically "Save As" a new file to force Word to write a clean version from scratch. This was the digital equivalent of being told that your car works fine, you just need to rebuild the engine every 500 miles as routine maintenance. The end result was that Microsoft Word quickly developed a reputation among technical people as horrible to work with. Not because it was a bad word processor — it was actually quite good at the word processing part — but because when a user showed up at the Help Desk with tears in their eyes, the tools I had to help them were mostly useless. I could scan the raw file for text patterns, which often pulled out the content, but without formatting it wasn't really a recovered file — it was more like finding your belongings scattered across a field after a tornado. Technically your stuff, but not in any useful arrangement. Sometimes you could rebuild the FAT or try alternative directory entries to recover slightly older versions. But in general, if the .doc encountered a structural error, the thing was toast and your work was gone forever. This led to a never-ending series of helpdesk sessions where I had to explain to people that yes, I understood they had worked on this file for months, but it was gone and nobody could help them. I became a grief counselor who happened to know about filesystems. Thankfully, people quickly learned to obsessively copy their files to multiple locations with different names — thesis_final.doc, thesis_final_v2.doc, thesis_FINAL_FINAL_REAL.doc — but this required getting burned at least once, which is sort of like saying you learned your car's brakes didn't work by driving into a bus. So around 2007 we see the shift from to , which introduces a lot of hard lessons from the problems of . First, it's just a bundle, specifically a ZIP archive. Now in theory, this is great. Your content is human-readable XML. Your images are just image files. If something goes wrong, you can rename the file to .zip, extract it, and at least recover your text by opening document.xml in Notepad. The days of staring at an opaque binary blob and praying were supposed to be over. However, in practice, something terrible happened. Microsoft somehow managed to produce the worst XML to ever exist in human history. Let me lay down the scope of this complexity, because I have never seen anything like it in my life. Here is the standards website for ECMA-376. Now you know you are in trouble when you see a 4 part download that looks like the following: If you download Part 1, you are given the following: Now if you open that PDF, get ready for it. It's a 5039 page PDF. I have never conceived of something this complicated. It's also functionally unreadable, and I say this as someone who has, on multiple occasions in his life, read a car repair manual cover to cover because I didn't have anything else to do. I once read the Haynes manual for a 1994 Honda Civic like it was a beach novel. This is not that. This is what happens when a standards committee gets a catering budget and no deadline. There was an accusation at the time that Microsoft was making OOXML deliberately more complicated than it needed to be — that the goal was to claim it was an "open standard" while making the standard so incomprehensibly vast that it would take a heroic effort for anyone else to implement it. I think this is unquestionably true. LibreOffice has a great blog post on it that includes this striking comparison: So the difference between ODF format and the OOXML format results in a exponentially less complicated XML file. Either you could do the incredible amount of work to become compatible with this nightmarish specification or you could effectively find yourself cut out of the entire word processing ecosystem. Now without question this was done by Microsoft in order to have their cake and eat it too. They would be able to tell regulators and customers that this wasn't a proprietary format and that nobody was locked into the Microsoft Office ecosystem for the production of documents, which had started to become a concern among non-US countries that now all of their government documents and records were effectively locked into using Microsoft. However the somewhat ironic thing is it ended up not mattering that much because soon the only desktop application that would matter is the browser. The file formats of word processors were their own problems, but more fundamentally the nature of how people consumed content was changing. Desktop based applications became less and less important post 2010 and users got increasingly more frustrated with the incredibly clunky way of working with Microsoft Word and all traditional files with emailing them back and forth endlessly or working with file shares. So while was a superior format from the perspective of "opening the file and it becoming corrupted", it also was fundamentally incompatible with the smartphone era. Even though you could open these files, soon the expectation was that whatever content you wanted people to consume should be viewable through a browser. As "working for a software company" went from being a niche profession to being something that seemingly everyone you met did, the defacto platform for issues, tracking progress, discussions, etc moved to GitHub. This was where I (and many others) first encountered Markdown and started using it on a regular basis. John Gruber, co-creator of Markdown, has a great breakdown of "standard" Markdown and then there are specific flavors that have branched off over time. You can see that here . The important part though is: it lets you very quickly generate webpages that work on every browser on the planet with almost no memorization and (for the most part) the same thing works in GitHub, on Slack, in Confluence, etc. You no longer had to ponder whether the person you were sending to had the right license to see the thing you were writing in the correct format. This combined with the rise of Google Workspace with Google Docs, Slides, etc meant your technical staff were having conversations through Markdown pages and your less technical staff were operating entirely in the cloud. Google was better than Microsoft at the sort of stuff Word had always been used for, which is tracking revisions, handling feedback, sharing securely, etc. It had a small subset of the total features but as we all learned, nobody knew about the more advanced features of Word anyway. By 2015 the writing was on the wall. Companies stopped giving me an Office license by default, switching them to "you can request a license". This, to anyone who has ever worked for a large company, is the kiss of death. If I cannot be certain that you can successfully open the file I'm working on, there is absolutely no point in writing it inside of that platform. Combine that with the corporate death of email and replacing it with Slack/Teams, the entire workflow died without a lot of fanfare. Then with the rise of LLMs and their use (perhaps overuse) of Markdown, we've reached peak . Markdown is the format of our help docs, many of our websites are generated exclusively from Markdown. It's now the most common format that I write anything in. This was originally written in Markdown inside of Vim. There's a lot of reasons why I think Markdown ended up winning, in no small part because it solved a real problem in an easy to understand way. Writing HTML is miserable and overkill for most tasks, this removed the need to do that and your output was consumable in a universal and highly performant way that required nothing of your users except access to a web browser. But I also think it demonstrates an interesting lesson about formats. and . along with ODF are pretty highly specialized things designed to handle the complexity of what modern word processing can do. LibreOffice lets you do some pretty incredible things that cover a huge range of possible needs. Markdown doesn't do most of what those formats do. You can't set margins. You can't do columns. You can't embed a pivot table or track changes or add a watermark that says DRAFT across every page in 45-degree gray Calibri. Markdown doesn't even have a native way to change the font color. And none of that mattered, because it turns out most writing isn't about any of those things. Most writing is about getting words down in a structure that makes sense, and then getting those words in front of other people. Markdown does that with less friction than anything else ever created. You can learn it in ten minutes, write it in any text editor on any device, read the source file without rendering it, diff it in version control, and convert it to virtually any output format. The files are plain text. They will outlive every application that currently renders them. They don't belong to any company. They can't become corrupted in any meaningful way — the worst thing that can happen to a Markdown file is you lose some characters, and even then the rest of the file is fine. After decades of nursing .doc files like they were delicate flowers that you had to transport home strapped to your car roof, the idea of a format that simply cannot structurally fail is not just convenient. It's a kind of liberation. I think about this sometimes when I'm writing in Vim at midnight, just me and a blinking cursor and a plain text file that will still be readable when I'm dead. No filesystem-within-a-filesystem. No sector allocation tables. No 5,039-page specification. Just words, a few hash marks, and never having to think about it again. Updating the document stream (your text) Updating the formatting tables Update the sector allocation tables Update the directory entries Update summary information Flush everything to disk Part 1 “Fundamentals And Markup Language Reference”, 5th edition, December 2016 Part 2 “Open Packaging Conventions”, 5th edition, December 2021 Part 3 “Markup Compatibility and Extensibility”, 5th edition, December 2015 Part 4 “Transitional Migration Features”, 5th edition, December 2016

Programming HTML Xml Rust

▲ 0

0 views

matduggan.com 5 months ago

Update to the Ghost theme that powers this site

I added a few modifications to the OSS Ghost theme that powers this site. You can get it here: https://gitlab.com/matdevdug/minimal-ghost-theme I tried to make it pretty easy to customize, but if you need something changed feel free to open an issue on the repo. Thanks for all the feedback! Added better image caption support. Added the cool Mastodon feature outlined here to attribute posts from your site back to your Mastodon username by following the instructions here.

Web Development Open Source

▲ 0

0 views

matduggan.com 5 months ago

Boy I was wrong about the Fediverse

I have never been an "online community first" person. The internet is how I stay in touch with people I met in real life. I'm not a "tweet comments at celebrities" guy. I was never funny enough to be the funniest person on Twitter. So when Twitter was accidentally purchased by a fascist high on ketamine, I moved to Mastodon mostly because it seemed to be “Twitter without the bullshit”. No recommended for you feed, no ads, it was broken in a way I find charming. Of course search was broken because all OSS social tools must have one glaring lack of functionality. In a nightmare world full of constant change it’s good to have a few constants to hold on to. A lot of the narrative at the time was “this is our flag in the ground in the fight against The Man”. It wasn’t clear in this context if they meant corporations or the media or the weird pseudo celebrity that had taken over social media where people would breathlessly tell me about shit like “Chris-Chan” and “Logan Paul bought a Pokemon card”. We all need pointless hobbies, but I care about YouTube stars like I care about distant stars dying. It’s interesting to someone somewhere but those people don’t talk to me. I mostly use social media as a place to waste time, not a platform to form para-social relationships to narcissists. I prefer my narcissism farm to table. I’d rather dig a grave with a rusty spoon than watch a Twitch “star”. Anyway, I watched mostly apathetically as the internet tried to rally itself to another cause. I read my news at the normal newspapers, watched my normal television and put social media off into its own silo. Then Trump effectively shut down the entire free press in the US in a series of bullshit lawsuits. See I had forgotten the one golden rule of capitalism. To thrive in capitalism one must be amoral. Now you can be wildly sickeningly successful with morals but you cannot reach that absolute zenith of shareholder value. Either you accept a lower share price and don’t commit atrocities or you become evil. There is no third option. So of course media corporations became bargaining chips for the oligarchs' actual businesses. Why fight a defamation suit when you can settle it by running favorable coverage and maybe bankrupting the media outlet you bought as a stocking stuffer? Suddenly I couldn’t find any reliable reporting about anything in the US. My beloved Washington Post became straight-up propaganda and desperate attempts to cope. "Best winter stews to make while you watch your neighbors get kidnapped at gunpoint." Twelve dollars a month for that. Threads was worthless because it’s the most boring social media website ever imagined. It’s a social media network designed by brands for brands, like if someone made a cable channel that was just advertisements and meta commentary about the advertisements you just saw. Billions of dollars at their disposal and Meta made a hot new social media network with the appeal of junk mail. Bluesky had a bunch of “stuff” but they’re trying to capture that 2008 Twitter lightning in a bottle which is a giant waste of time. We’re never going to go back to pretending that tweeting at politicians does anything and everyone there is desperately trying to build a “brand” as the funny one or whatever. I want news I don’t want your endless meta commentary on the news. People talk a lot about the protocols that power Bluesky vs. ActivityPub, because we're nerds and we believe deep in our hearts that the superior protocol will win. This is adorable. It flies in the face of literally all of human history, where the more convenient thing always wins regardless of technical merit. VHS beat Betamax. USB-C took twenty years. The protocol fight is interesting the way medieval siege warfare is interesting — I'm glad someone's into it, but it has no bearing on my life. There's no actual plan to self-host Bluesky. Their protocol makes it easier to scale their service. That's why it was written and that's what it does. End of story. Now EU news remained reliable, but sending European reporters into the madness of the US and trying to get a “report” out of it is an exercise in frustration. This became especially relevant for me when Trump threatened to invade Greenland and suddenly there was a distinct possibility that there might be an armed conflict between Denmark and the US. Danish reporters weren’t getting meetings with the right people and it was just endless rumors and Truth Social nonsense. If the American press had given me 20 minutes of airtime I could have convinced everyone they don’t want to get involved with Greenland. We’re not tough enough as a people to survive in Greenland, much less “take it over”. Greenlandic people shrug off horrific injuries hundreds of kilometers from medical help with a smile. I watched a Greenlandic toddler munch meat from the spine of a seal with its head very much intact. We aren’t equipped to fuck with these people, they are the real deal. So in this complete breakdown of the press came in the Fediverse. It became the only reliable source of information I had. People posted links with a minimal amount of commentary, picking and choosing the best content from other social media networks. They’re not doing it to “build a brand” because that’s not a thing in the Fediverse. It’s too disjointed to be a place to build a newsletter subscription base. Instead it became the only place consistently posting trustworthy information I could actually access. This became personally relevant when Trump threatened to invade Greenland, which is the kind of sentence I never expected to type and yet here we are. It would be funny if I wasn't a tiny bit concerned that my new home was going to get a CIA overnight regime change special in the middle of the night. It was somewhere in the middle of DMing with someone who had forgotten more about Greenland than I would ever know and someone who lived close to an RAF base in the UK that it clicked. This was what they had been talking about. Actual human beings were able to find each other and ask direct questions without this giant mountain of bullshit engagement piled on top of it. Meta or Oracle or whoever owns TikTok this week couldn't stop me. I never expected to find my news from strangers on a federated social network that half the internet has never heard of. I never expected a lot of things. But there's something quietly beautiful about a place where people just... share what they know. No brand deals, no engagement metrics, no algorithm nudging you toward rage. Just someone who spent twenty years studying Arctic policy posting a thread at 2 AM because they think you should understand what's happening. It's the internet I was promised in 1996. It only took thirty years and the complete collapse of American journalism to get here. Find me at: https://c.im/@matdevdug

Media Culture Journalism

▲ 0

0 views

matduggan.com 6 months ago

I Sold Out for $20 a Month and All I Got Was This Perfectly Generated Terraform

Until recently the LLM tools I’ve tried have been, to be frank, worthless. Copilot was best at writing extremely verbose comments. Gemini would turn a 200 line script into a 700 line collection of gibberish. It was easy for me to, more or less, ignore LLMs for being the same over-hyped nonsense as the Metaverse and NFTs. This is great for me because I understand that LLMs represent a massive shift in power from an already weakened worker class to an increasingly monarch-level wealthy class. By stealing all human knowledge and paying nothing for it, then selling the output of that knowledge, LLMs are an impossibly unethical tool. So if the energy wasting tool of the tech executive class is also a terrible tool, easy choice. Like boycotting Tesla for being owned by an evil person and also being crappy overpriced cars, or not shopping at Hobby Lobby and just buying directly from their Chinese suppliers, the best boycotts are ones where you aren’t really losing much. Google can continue to choke out independent websites with their AI results that aren’t very good and I get to feel superior doing what I was going to do anyway by not using Google search. This logic was all super straight forward right up until I tried Claude Code. Then it all got much more complicated. Let’s just get this out of the way right off the bat. I didn't want to like Claude Code. I got a subscription with the purpose of writing a review on it where I would find that it was just as terrible as Gemini and Copilot. Except that's not what happened. Instead it was like discovering the 2AM kebab place might actually make the best pizza in town. I kept asking Claude to do annoying tasks where it was easy for me to tell if it had made a mistake and it kept doing them correctly. It felt impossible but the proof was right in front of me. I’ve written tens of thousands of lines of Terraform in my life. It is a miserable chore to endlessly flip back and forth between the provider documentation and Vim, adding all the required parameters. I don’t learn anything by doing it, it’s just a grind I have to push through to get back to the meaningful work. The amount of time I have wasted on this precious time on Earth importing all of a companies DNS records into Terraform, then taking the autogenerated names and organizing them so that they make sense for the business is difficult to express. It's like if the only way I knew how to make a hamburger bun was to carefully put every sesame seed by hand on the top only to stumble upon an 8 pack of buns for $4 at the grocery store after years of using tiny tweezers to put the seeds in exactly the right spot. I feel the same way about writing robust READMEs, k8s YAML and reorganizing the file structure of projects. Setting up more GitHub Actions is as much fun as doing my taxes. If I never had to write another regex for the rest of my life, that would be a better life by every conceivable measure. These are tasks that sap my enthusiasm for this type of work, not feed it. I’m not sad to offload them and switch to mostly reviewing its PRs. But the tool being useful doesn’t remove what’s bad about it. This is where a lot of pro-LLM people start to delude themselves. In no particular order are the arguments I keep seeing about LLMs from people who want to keep using them for why their use is fine. This is the most common one I see and the worst. It can be condensed down to “because most things on the internet originally existed to find pornography and/or pirate movies, stealing all content on the internet is actually fine because programmers don’t care about copyright”. You also can’t have it both ways. OpenAI can’t decide to enforce NDAs and trademarks and then also declare law is meaningless. If I don’t get to launch a webmail service named Gmail+ then Google doesn’t get to steal all the books in human existence. The argument basically boils down to: because we all pirated music in 2004, intellectual property is a fiction when it stands in the way of technology. By this logic I shoplifted a Snickers bar when I was 12 so property rights don't exist and I should be allowed to live in your house. I have an internet friend I met years ago playing EVE Online that is a brutally pragmatic person. To someone like him, code craftsmanship is a joke. For those of you who are unaware, EVE Online is the spaceship videogame where sociopaths spend months plotting against each other. His approach to development is 80% refining requirements and getting feedback. He doesn’t care at all about DRY, he uses Node because then he can focus on just JavaScript, he doesn’t invest a second into optimization until the application hits a hard wall that absolutely requires it. His biggest source of clients? Creating fast full stacks because internal teams are missing deadlines. And he is booked up for at least 12 months out all the time because he hits deadlines. When he started freelancing I thought he was crazy. Who was going to hire this band of Eastern European programmers who chain smoke during calls and whose motto is basically "we never miss a deadline". As it turns out, a lot of people. Why doesn't he care? Why doesn't he care about these things? He believes that programmers fundamentally don't understand the business they are in. "Code is perishable" is something he says a lot and he means it. Most of the things we all associate with quality (full test coverage, dependency management, etc) are programmers not understanding the rate of churn a project undergoes over its lifespan. The job of a programmer, according to him, is delivering features that people will use. How pleasant and well-organized that code is to work with is not really a thing that matters in the long term. He doesn't see LLM-generated code as a problem because he's not building software with a vision that it will still be used in 10 years. Most of the stuff typically associated with quality he, more or less, throws in the trash. He built a pretty large stack for a automotive company and my jaw must have hit the table when he revealed they're deploying m6g.4xlarge for a NodeJS full-stack application. "That seems large to me for that type of application" was my response. He was like "yeah but all I care about are whether the user metrics show high success rate and high performance for the clients". It's $7000 a year for the servers, with two behind a load balancer. That's absolutely nothing when compared with the costs of what having a team of engineers tune it would cost and it means he can run laps around the internal teams who are, basically, his greatest competition. To be clear, he is very technically competent. He simply rejects a lot of the conventional wisdom out there about what one has to do in order to make stuff. He focuses on features, then securing endpoints and more or less gives up on the rest of it. For someone like this, LLMs are a logical choice for him. The annoying thing about my friend is that his bank account suggests he's right. But I can't get there. If I'm writing a simple script or something as a one-off, it can sometimes feel like we're all wasting the companies time when we have a long back and forth on the PR discussing comments or the linting or whatever. So it's not that this idea is entirely wrong . But the problem with programming is you never know what is going to be "the core" of your work life for the next 5 years. Sometimes I write a feature, we push it out, it explodes in popularity and then I'm a little bit in trouble because I built a MVP and now it's a load-bearing revenue generating thing that has to be retooled. I also just have trouble with the idea that this is my career and the thing I spend my limited time on earth doing and the quality of it doesn't matter. I delight in craftsmanship when I encounter it in almost any discipline. I love it when you walk into an old house and see all the hand crafted details everywhere that don't make economic sense but still look beautiful. I adore when someone has carefully selected the perfect font to match something. Every programmer has that library or tool that they aspire to. That code base where you delight at looking at it because it proves perfection is possible even if you have never come close to reaching that level. For me its always been looking through the source code of SQLite that restores my confidence. I might not know what I'm doing but it's good to be reminded that someone out there does. Not everything I make is that great, but the concept of "well great doesn't matter at all" effectively boils down to "don't take pride in your work" which is probably the better economic argument but feels super bad to me. In a world full of cheap crap, it feels bad to make more of it and then stick my name on it. The best argument for why programmers should be using LLMs is because it's going to be increasingly difficult to compete for jobs and promotions against people who are using them. In my experience Claude Code allows me to do two tasks at once. That's a pretty hard advantage to overcome. Last Tuesday I had Claude Code write a GitHub Action for me while I worked on something else. When it was done, I reviewed it, approved it, and merged it. It was fine. It was better than fine, actually — it was exactly what I would have written, minus the forty-five minutes of resentment. I sat there for a moment, staring at the merged PR, feeling the way I imagine people feel when they hire a cleaning service for the first time: relieved, and then immediately guilty about the relief, and then annoyed at myself for feeling guilty about something that is, by any rational measure, a completely reasonable thing to do. Except it isn't reasonable. Or maybe it is. I genuinely don't know anymore, and that's the part that bothers me the most — not that the tool works, but that I've lost the clean certainty that it shouldn't. So now I'm paying $20 a month to a company that scraped the collective knowledge of humanity without asking so that I can avoid writing Kubernetes YAML. I know what that makes me. I just haven't figured out a word for it yet that I can live with. When I asked my EVE friend about it on a recent TeamSpeak session, he was quiet for awhile. I thought that maybe my moral dilemma had shocked him into silence. Then he said, "You know what the difference is between you and me? I know I'm a mercenary. You thought you were an artist. We're both guys who type for money." I couldn't think of a clever response to that. I still can't.

JavaScript AI Programming Terraform Yaml

▲ 0

0 views

matduggan.com 6 months ago

The Small Web is Tricky to Find

One of the most common requests I've gotten from users of my little Firefox extension( https://timewasterpro.xyz ) has been more options around the categories of websites that you get returned. This required me to go through and parse the website information to attempt to put them into different categories. I tried a bunch of different approaches but ended up basically looking at the websites themselves seeing if there was anything that looked like a tag or a hint on each site. This is the end conclusion of my effort at putting stuff into categories. Unknown just means I wasn't able to get any sort of data about it. This is the result of me combining Ghost, Wordpress and Kagi Small Web data sources. Interestingly one of my most common requests is "I would like less technical content" which as it turns out is tricky to provide because it's pretty hard to find. They sorta exist but for less technical users they don't seem to have bought into the value of the small web own your own web domain (or if they have, I haven't been able to figure out a reliable way to find them). This is an interesting problem, especially because a lot of the tools I would have previously used to solve this problem are....basically broken. It's difficult for me to really use Google web search to find anything at this point even remotely like "give me all the small websites" because everything is weighted to steer me away from that towards Reddit. So anything that might be a little niche is tricky to figure out. So there's no point in building a web extension with a weighting algorithm to return less technical content if I cannot find a big enough pool of non-technical content to surface. It isn't that these sites don't exist its just that we never really figured out a way to reliably surface "what is a small website". So from a technical perspective I have a bunch of problems. I think I can solve....some of these, but the more I work on the problem the more I'm realizing that the entire concept of "the small web" had a series of pretty serious problems. First I need to reliably sort websites into a genre, which can be a challenge when we're talking about small websites because people typically write about whatever moves them that day. Most of the content on a site might be technical, but some of it might not be. Big sites tend to be more precise with their SEO settings but small sites that don't care don't do that, so I have fewer reliable signals to work with. Then I need to come up with a lot of different feeding systems for independent websites. The Kagi Small Web was a good starting point, but Wordpress and Ghost websites have a much higher ratio of non-technical content. I need those sites, but it's hard to find a big batch of them reliably. Once I have the type of website as a general genre and I have a series of locations, then I can start to reliably distribute the types of content you get. Google was the only place on Earth sending any traffic there Because Google was the only one who knew about it, there never needed to be another distribution system Now that Google is broken, it's almost impossible to recreate that magic of becoming the top of list for a specific subgenre without a ton more information than I can get from public records.

Web Development Data Web

▲ 0

0 views

matduggan.com 6 months ago

GitButler CLI Is Really Good

My workflow has remained mostly the same for over a decade. I write everything in Vim using the configuration found here . I run Vim from inside of tmux with a configuration found here . I write things on a git branch, made with the CLI, then I add them with to that branch, trying to run all of the possible linting and tests with before I waste my time on GitHub Actions. Then I run which is an alias to . Finally I successfully commit, then I copy paste the URL returned by GitHub to open a PR. Then I merge the PR and run to go back to the primary branch, which is an alias to . This workflow, I think, is pretty familiar for anyone working with GitHub a lot. Now you'll notice I'm not saying because almost nothing I'm doing has anything to do with . There's no advantage to my repo being local to my machine, because everything I need to actually merge and deploy code lives on GitHub. The CI runs there, the approval process runs there, the monitoring of the CI happens there, the injection of secrets happens there. If GitHub is down my local repo does, effectively, nothing. My source of truth is always remote, which means I pay the price for complexity locally but I don't benefit from it. At most jobs: This means the following is also true: Almost all the features of are wasted on me in this flow. Now because this tool serves a million purposes and is designed to operate in a way that almost nobody uses it for, we all pay the complexity price of and never reap any of the benefits. So instead I keep having to add more aliases to paper over the shortcomings of . These are all the aliases I use at least once a week. Git's offline-first design creates friction for online-first workflows, and GitButler CLI eliminates that friction by being honest about how we actually work. (Edit: I forgot to add this disclaimer. I am not, nor have ever been an employee/investor/best friends with anyone from GitButler. They don't care that I've written this and I didn't communicate with anyone from that team before I wrote this.) So let's take the most basic command as an example. This is my flow that I do 2-3 times a day without my aliases. I do this because can't make assumptions about the state of the world. However because GitButler is designed with the assumption that I'm working online, we can skip a lot of this nonsense. It's status command understands that there is always a remote main that I care about and that when I run a status that I need to understand my status relative to the remote main as it exists right now. Not how it existed the last time I remembered to pull. However this is far from the best trick it has up its sleeve. You're working on a feature, notice an unrelated bug, and now you have to stash, checkout, fix, commit, push, checkout back, stash pop. Context switching is expensive and error-prone. GitButler effectively hacks a solution into that fixes this with multiple branches applied simultaneously. Assign files to different branches without leaving your workspace. What do I mean by that. Let's start again with my status Great looks good. Alright so lets say I make 2 new branches. I'm working on a new feature for adding auth and while I'm working on that, I see a typo I need to fix in a YAML. I can work on both things at the same time: And easily commit to both at the same time without doing anything weird . Stacked PRs are the "right" way to break up large changes so people on your team don't throw up at being asked to review 2000 lines, but Git makes them miserable. When the base branch gets feedback, you have to rebase every dependent branch, resolve conflicts, force-push, and pray. Git doesn't understand branch dependencies. It treats every branch as independent, so you have to manually maintain the stack. GitButler solves this problem with First-class stacked branches. The dependency is explicit, and updates propagate automatically. So what do I mean. Let's say I make a new API endpoint in some Django app. First I make the branch. So let's say I'm working on the branch and get some good feedback on my PR. It's easy to resolve the comments there while leaving my branched off this as a stacked thing that understands the relationship back to the first branch as shown here. In practice this is just a much nicer way of dealing with a super common workflow. Maybe the most requested feature from new users I encounter is an easier undo. When you mess up in Git, recovery means diving into , understanding the cryptic output, and hoping you pick the right . One wrong move and you've made it worse. GitButlers is just easier to use. So the basic undo functionality is super simple to understand. rolls me back one operation. To me the mental model of a snapshot makes a lot more sense than the git history model. I do an action, I want to undo that action. This is better than the git option of: I've been using GitButler in my daily work since I got the email that the CLI was available and I've really loved it. I'm a huge fan of what this team is doing to effectively remodel and simplify Git operations in a world where almost nobody is using it in the way the tool was originally imagined to be used. I strongly encourage folks go check it out for free at: https://docs.gitbutler.com/cli-guides/cli-tutorial/tutorial-overview . It does a ton of things (like help you manage PRs) that I didn't even touch on here. Let me know if you find something cool that I forgot at: https://c.im/@matdevdug You can't merge without GitHub (PRs are the merge mechanism) You can't deploy without GitHub (Actions is the deployment trigger) You can't get approval without GitHub (code review lives there) Your commits are essentially "drafts" until they exist on GitHub You never work disconnected intentionally You don't use local branches as long-lived divergent histories You don't merge locally between branches (GitHub PRs handle this) You don't use for archaeology — you use GitHub's blame/history UI (I often use git log personally but I have determined I'm in the minority on this). Your local repo might be offline for days or weeks The "remote" might be someone else's laptop, not a central server Divergent histories are expected and merging is a deliberate, considered act

Python Yaml Git

▲ 0

0 views

matduggan.com 8 months ago

The Year of the 3D Printed Miniature (And Other Lies We Tell Ourselves)

One amusing thing about following tech news is how often the tech community makes a bold prediction or assertion, only to ultimately be completely wrong. This isn't amusing in a "ha ha, we all make mistakes" kind of way. It's amusing in the way that watching someone confidently stride into a glass door is amusing. You feel bad, but also, they really should have seen that coming. Be it VR headsets that would definitely replace reality by 2018, or self-driving cars in every driveway "within five years" (a prediction that has been made every five years since 2012), we have a remarkable talent for making assumptions about what consumers will like and value without having spent a single goddamn minute listening to those same consumers. It's like a restaurant critic reviewing a steakhouse based entirely on the menu font. So when a friend asked me what I thought about "insert new revolutionary technology that will change everything" this week, my brain immediately jumped to "it'll be like 3D printers and Warhammer." This comparison made sense in the moment, as we were currently playing a game of Warhammer 40,000, surrounded by tiny plastic soldiers and the faint musk of regret. But I think, after considering it later, it might make sense for more people as well—a useful exercise in tech enthusiasm versus real user wants and needs. Or, put another way: a cautionary tale about people who have never touched grass telling grass-touchers how grass will work in the future. One long-held belief among tech bros has been the absolute confidence that 3D printers would, at some point, disrupt . Exactly what they would disrupt wasn't 100% clear. Disruption, in Silicon Valley parlance, is less a specific outcome and more a vibe—a feeling that something old and profitable will soon be replaced by something new and unprofitable that will somehow make everyone rich. A common example trotted out was one of my favorite hobbies: tabletop wargaming. More specifically, the titan of the industry, Warhammer 40,000. Every time a new 3D printer startup graced the front page of Hacker News, this proclamation would echo from the comments section like a prophecy from a very boring oracle: "This will destroy Games Workshop." Reader, it has not destroyed Games Workshop. Games Workshop is doing fine. Games Workshop will be selling overpriced plastic crack to emotionally vulnerable adults long after the sun has consumed the Earth. For those who had friends in high school—and I'm not being glib here, this is a genuine demographic distinction—40k is a game where two or more players invest roughly $1,000 to build an army of small plastic figures. You then trim excess plastic with a craft knife (cutting yourself at least twice, this is mandatory), prime them, paint them over the course of several months, and then carefully transport them to an LGS (local game shop) in foam-lined cases that cost more than some people's luggage. Another fellow dork will then play you on a game board roughly the size of a door, covered in fake terrain that someone spent 40 hours making to look like a bombed-out cathedral. You will both have rulebooks with you containing as many pages as the Bible and roughly as open to interpretation. Wars have been started over less contentious texts. To put 40k in some sort of nerd hierarchy, imagine a game shop. At the ground level of this imaginary shop are Magic: The Gathering and Pokémon TCG games. Yes, these things are nerdy, but it's not that deep into the swamp. It's more of a gentle wade. You start with Pokémon at age 10, burn your first Tool CD at 14, and then sell your binder of 'mons to fund your Magic habit. This is the natural order of things. Deeper into the depths, maybe only playing at night like creatures who have evolved beyond the need for vitamin D, are your TTRPGs (tabletop RPGs). The titan of the industry is Dungeons & Dragons, but there is always some new hotness nipping at its heels, designed by someone who thought D&D wasn't quite complicated enough. TTRPGs are cheap to attempt to disrupt—you basically need "a book"—so there are always people trying. These are the folks with thick binders, sacks of fancy dice made from materials that should not be made into dice, and opinions about "narrative agency." Near the bottom, almost always in the literal basement of said shop, are the wargame community. We are the Morlocks of this particular H.G. Wells situation. I, like a lot of people, discovered 40k at a dark time in my life. My college girlfriend had cheated on me, and I had decided to have a complete mental breakdown over this failed relationship that was doomed well before this event. The cheating was less a cause and more a symptom, like finding mold on bread that was already stale. Honestly, in retrospect, hard to blame her. I was being difficult. I was the kind of difficult where your friends start sentences with "Look, I love you, but..." Late at night, I happened to be driving my lime green Ford Probe past my local game shop. The Ford Probe, for those unfamiliar, was a car designed by someone who had heard of cars but had never actually seen one. It was the automotive equivalent of a transitional fossil. I loved it the way you love something that confirms your worst suspicions about yourself. There, through the shop window, I saw people hauling some of the strangest items out of their trunks. Half-destroyed buildings. Thousands of tiny little figures. Giant robots the size of a small cat with skulls for heads. One man was carrying what appeared to be a ruined spaceship made entirely of foam and spite. I pulled over immediately. The owner, who knew me from playing Magic, seemed neither surprised nor pleased to see me. This was his default state. Running a game shop for 20 years will do that to a person. "They're in the basement," he said, in the mostly dark game shop, the way someone might say "the body's in the basement" in a very different kind of establishment. I descended the rickety wooden stairs to a large basement lit by three naked bulbs hanging from cords. The aesthetic was "serial killer's workspace" meets "your uncle's unfinished renovation project." It was perfect. Before me were maybe a dozen tables littered with plastic. Some armies had many bug-like things, chitinous and horrible. Others featured little skeletons or robots. There were tape measures everywhere and people throwing literal handfuls of small six-sided dice at the table with the intensity of gamblers who had nothing left to lose. Arguments broke out over millimeters. Someone was consulting a rulebook with the desperation of a lawyer looking for a loophole. I was hooked immediately. 40k is the monster of wargaming specifically because of a few genius decisions by Games Workshop, the creators—a British company that has somehow figured out how to print money by selling plastic and lore about a fascist theocracy in space. It's a remarkable business model. Since the beginning of the game, 40k casual games have allowed proxies. Proxies are stand-ins for specific units that you need for an army but don't have. Why don't you have them? Excellent question. Let me tell you about Games Workshop's relationship with its customers. Games Workshop has always played a lot of games with inventory. Often releases will have limited supply, or there are weird games with not fulfilling the entire order that a game shop might make. Even when they switched from metal to plastic miniatures, the issues persisted. This has been the source of conspiracy theories since the very beginning—whispers of artificial scarcity, of deliberate shortages designed to create FOMO among people who were already deeply susceptible to FOMO because they collect tiny plastic soldiers. Whether the conspiracy theories are true is almost beside the point. The feeling of scarcity is real, and feelings, as any therapist will tell you, are valid. Even the stupid ones. So players had proxies. Anything from a Coke can to another unit entirely. Basically, if it had the same size base and roughly the same height, most people would consider it allowable. "This empty Red Bull can is my Dreadnought." Sure. Fine. We've all been there. This is where I first started to see 3D-printed miniatures enter the scene. Similar to most early tech products, the first FDM 3D-printed miniatures I saw were horrible. The thick, rough edges and visible layer lines were not really comparable to the professional product, even from arm's length. They looked like someone had described a Space Marine to a printer that was also drunk. But they were totally usable as a proxy and better than a Coke can. The bar, as they say, was low. But the technology continued to get better and cheaper and, as predicted by tech people, I started to notice more and more interest in 3D printing among people at the game stores. When I first encountered a resin 3D-printed army at the table, I'll admit I was intrigued. This person had basically fabricated $3,000 worth of hard-to-get miniatures out of thin air and spite. This was supposed to be the big jumping-off point. The inflection moment. There were a lot of discussions at the table about how soon we wouldn't even have game shops with inventory! They'd be banks of 3D printers that we would all effortlessly use to make all the minis we wanted! The future was here, and it smelled like resin fumes! Printing a bunch of miniatures off a resin 3D printer quickly proved to have a lot of cracks in this utopian plan. Even a normal-sized mini took hours to print. That wouldn't be so bad, except these printers couldn't just live anywhere in your apartment. They're not like a Keurig. You can't just put them on your kitchen counter and forget about them. When I was invited to watch someone print off minis with a resin 3D printer, it reminded me a lot of the meth labs in my home state of Ohio. And I don't mean that as hyperbole. I mean there were chemicals, ventilation hoods, rubber gloves, and a general atmosphere of "if something goes wrong here, it's going to go very wrong." The guy giving me the tour had safety goggles pushed up on his forehead. He was wearing an apron. At one point, he said the phrase "you really don't want to get this on your skin" with the casual tone of someone who had definitely gotten it on his skin. In practice, the effort to get the STL files, add supports, wash off the models with isopropyl alcohol, remove supports without snapping off tiny arms, and finally cure the mini in UV lights was exponentially more effort than I'm willing to invest. And I say this as someone who has painted individual eyeballs on figures smaller than my thumb. I have a high tolerance for tedious bullshit. This exceeded it. Before I start, I first want to say I don't dislike the 3D printing community. I think it's great they're supporting smaller artists. I love that they found a hobby inside of a hobby, like those Russian nesting dolls but for people who were already too deep into something. I will gladly play against their proxy armies any day of the week. But people outside of the hobby proclaiming that this is the "future" are a classic example of how they don't understand why we're doing the activity in the first place. It's like watching someone who has never cooked explain how meal replacement shakes will eliminate restaurants. You're not wrong that it's technically more efficient. You're just missing the entire point of the experience. The reason why Games Workshop continues to have a great year after year—despite prices that would make a luxury goods executive blush, despite inventory issues, despite a rulebook that changes often enough to require a subscription service—is because of this fundamental misunderstanding. Players invest a lot of time and energy into an army. You paint them. You decorate the plastic bases with fake grass and tiny skulls. You learn their specific rules and how to use them. You develop opinions about which units are "good" and which are "trash" and you will defend these opinions with the fervor of a religious convert. Despite the eternal complaints about the availability of inventory, the practical reality is that most people can only keep a pipeline of one or maybe two armies going at once. The bottleneck isn't acquiring plastic. The bottleneck is everything else . So let's do the math on this. You buy a resin 3D printer. All the supplies. You get a spot in your house where you can safely operate it—which means either a garage, a well-ventilated spare room, or a relationship-ending negotiation with whoever you live with. You find or buy all the STLs you need. Let's say they all have supports in the files, so you just need to print them off. Best-case scenario. Let's say we break even around 50-75 infantry and a few larger models. This is over the raw cost of materials, but we need to factor in the space in your house it takes up, plus there's a learning curve with figuring out how to do it. You also need to invest a lot of time getting these files for printing and finding the good ones. For the sake of keeping this simple, let's just assume the actual printing process goes awesome. No failed prints. No supports that fuse to the model. No discovering that your file was corrupted after six hours of printing. Fantasy land. Here's the thing: getting the raw plastic minis is not the time-consuming part. First, you need to paint them. I take about two hours to paint each model, and I'm far from the best painter out there. I'm solidly in the "looks good from three feet away" category, which is also how I'd describe my general appearance. Vehicles take longer because they're bigger—maybe 10-20 hours for one of those. We're talking somewhere in the ballpark of 150 hours to paint everything that you need to paint for a standard army. Now don't get me wrong, I love painting. But I'm a 38-year-old with a child and a full-time job. Finding 150 hours for anything that isn't work, childcare, or sleep requires the kind of calendar Tetris that would make a project manager weep. It is a massive investment of time to get an army on the table, even if you remove the financial element of buying the minis entirely. Frankly, the money I pay to Games Workshop is the easiest part of the entire process. Often the box will be lovingly stacked on top of other sealed mini boxes—a pile of shame, we call it—until I can start the process of even hoping to catch up. I have boxes I bought during the Obama administration. They're still sealed. They judge me. But okay, let's say we get them all painted. What's next? Next comes "learn how the army works." There is a ton of flexibility to each army in 40k and how they work and operate. It takes a bit of research and time to figure out what they all do, which is something you are 100% expected to know cover to cover when you show up to play. It's not my job to know what your army can and cannot do. If you show up not knowing your own rules, you will be eaten alive, and you will deserve it. So what I saw with the 3D printing crowd felt a lot like the "Year of the Linux Desktop" crowd. Every year they would proclaim that soon we'd all get on board with their vision. They would print off an incredibly impressive army with all the hard-to-find minis that were sold once at a convention in 1997. They'd get the army "painted" to some definition of painted—and I'm using those quotation marks with malice—get on the table, and then play effectively that one army the same as the rest of us. The printer didn't give them more time. It didn't give them more skill. It just gave them more unpainted plastic, which, brother, I have plenty of already. For those in the 3D printing crowd who weren't big into playing, just painting, part of the point is showing off your incredible work to everyone else. Except nobody wants to see a 3D-printed forgery of an official model. It's like showing up to a car show with a kit car that looks like a Ferrari. Sure, it's impressive in its own way, but it's not really a Ferrari, and everyone knows it, and now we're all standing around pretending we don't know it, and it's uncomfortable for everyone. Once someone figured out one of your minis was 3D printed, shops generally wouldn't feature it in their display cases. So there was no reason for people who were going to put in 10+ hours per model to skip paying for the official real models. If you're going to invest that much time, you want the real thing. You want the little Games Workshop logo on the base. You want to be able to say "yes, I paid $60 for this single figure" with the quiet dignity of someone who has made peace with their choices. "Well then the shops can just sell the STLs and do the printing there!" This shows me you haven't spent a lot of time in these shops. Game shops need to carry a ton of inventory all the time, and a lot of their sales are impulse purchases. I see a mini I wouldn't typically be interested in, but it's done and ready, and I'm weak, and now I own it. That's the business model. They also operate on relatively thin margins—these aren't Apple Stores, they're labors of love run by people who got into this because they loved games and are now slowly being crushed by commercial rent and distributor minimums. It's just not feasible for them to print minis on demand and have enough staff to keep an eye on all the printing. Plus, tabletop wargaming isn't their major revenue generator anyway—it's card games like Pokémon and Magic. The wargamers in the basement are a bonus, not the main attraction. We're the weird cousins who show up to Thanksgiving and everyone tolerates us because we're family. At the end of the day, the 3D printing proclamation that it would disrupt my hobby ended up being a whole lot of nothing. A series of reasonable mistakes were made by people enthusiastic about the technology, resulting in the current situation where every year is the year that all of this will get disrupted. Any day now. Just you wait. They looked at the price of miniatures and saw inefficiency. They looked at the scarcity and saw opportunity. What they didn't see was that the price and the scarcity were almost beside the point. The hobby isn't about acquiring plastic. The hobby is about what you do with the plastic after you acquire it. The hobby is about the 150 hours of painting. The hobby is about the arguments over rules interpretations. The hobby is about descending into a basement lit by three naked bulbs and finding your people. You can't 3D print that. So the next time someone tells you that some new technology is going to "disrupt" something you love, ask yourself: do they actually understand why people love it? Do they understand the irrational, inefficient, deeply human reasons people engage with this thing? Or are they just looking at a spreadsheet and seeing numbers that don't make sense to them? Because if it's the latter, you can probably ignore them. They'll be wrong. They're almost always wrong. In the meantime, you can find me in the basement, losing match after match, surrounded by tiny plastic soldiers I've spent hundreds of hours painting, playing a game that makes no sense to anyone who hasn't given themselves over to it completely. It's not efficient. It's not optimized. It's not disrupting anything. The game looks more complicated to play than it is. Especially now, in the 10th edition, the core rules don't take long to learn. However, there is a lot of depth to the individual options available to each army that take a while to master. So it hits that sweet spot of being fast to onboard someone onto while still providing frightening amounts of depth if you're the kind of person who finds "frightening amounts of depth" appealing rather than exhausting. I am that kind of person. This explains a lot about my life. The community is incredible. When I moved from Chicago to Denmark, it took me less than three days to find a local 40k game. Same thing when I moved from Michigan to Chicago. The age and popularity of the game means it is a built-in community that follows you basically around the world. Few other properties have this kind of stickiness. It's like being a Deadhead, except instead of following a band, you're following a shared delusion that tiny plastic men matter. They do matter. Shut up. Cool miniatures. They look nice. They're fun to paint and put together. They're complicated without being too annoying. This is the part that 3D printers are supposed to help with.

Gaming Business

▲ 0

0 views

matduggan.com 8 months ago

SQLite for a REST API Database?

When I wrote the backend for my Firefox time-wasting extension ( here ), I assumed I was going to be setting up Postgres. My setup is boilerplate and pretty boring, with everything running in Docker Compose for personal projects and then persistence happening in volumes. However when I was working with it locally, I obviously used SQLite since that's always the local option that I use. It's very easy to work with, nice to back up and move around and in general is a pleasure to work with. As I was setting up the launch, I realized I really didn't want to set up a database. There's nothing wrong with having a Postgres container running, but I'd like to skip it if its possible. So my limited understanding of SQLite before I started this was "you can have one writer and many readers". I had vaguely heard of SQLite "WAL" but my understanding of WAL is more in the context of shipping WAL between database servers. You have one primary, many readers, you ship WAL to from the primary to the readers and then you can promote a reader to the primary position once it has caught up on WAL. My first attempt at setting up SQLite for a REST API died immediately in exactly this way. So by default SQLite: This seems to be caused by SQLite having a rollback journal and using strict locking. Which makes perfect sense for the use-case that SQLite is typically used for, but I want to abuse that setup for something it is not typically used for. So after doing some Googling I ended up with these as the sort of "best recommended" options. I'm 95% sure I copy/pasted the entire block. What is this configuration doing. However my results from load testing sucked. Now this is under heavy load (simulating 1000 active users making a lot of requests at the same time, which is more than I've seen), but still this is pretty bad. The cause of it was, of course, my fault. My "blacklist" is mostly just sites that publish a ton of dead links. However I had the setup wrong and was making a database query per website to see if it matched the black list. Stupid mistake. Once I fixed that. Great! Or at least "good enough from an unstable home internet connection with some artificial packet loss randomly inserted". So should you use SQLite as the backend database for a FastAPI setup? Well it depends on how many users you are planning on having. Right now I can handle between 1000 and 2000 requests per second if they're mostly reads, which is exponentially more than I will need for years of running the service. If at some point in the future that no longer works, it's thankfully very easy to migrate off of SQLite onto something else. So yeah overall I'm pretty happy with it as a design. Only one writer at a time Writers block readers during transactions Switches SQLite from rollback journal to Write-Ahead Logging (WAL) Default behavior is Write -> Copy original data to journal -> Modify database -> Delete journal. WAL mode is Write -> Append changes to WAL file -> Periodically checkpoint to main DB So here you have 4 options to toggle for how often SQLite syncs to disk. OFF is SQlite lets the OS handle it. NORMAL is the SQLite engine still syncs, but less often than FULL. WAL mode is safe from corruption with NORMAL typically. FULL uses the Xsync method of the VFS (don't feel bad I've never heard of it before either: https://sqlite.org/vfs.html ) to ensure everything is written to disk before moving forward. EXTRA: I'm not 100% sure what this exactly does but it sounds extra. "EXTRA synchronous is like FULL with the addition that the directory containing a rollback journal is synced after that journal is unlinked to commit a transaction in DELETE mode. EXTRA provides additional durability if the commit is followed closely by a power loss. Without EXTRA, depending on the underlying filesystem, it is possible that a single transaction that commits right before a power loss might get rolled back upon reboot. The database will not go corrupt. But the last transaction might go missing, thus violating durability, if EXTRA is not set." = please wait up to 60 seconds. this one threw me for a loop. Why is it a negative number? If you set it to a positive number, you mean pages. SQLite page size is 4kb by default, so 2000 = 8MB. A negative number means KB which is easier to reason about than pages. I don't really know what a "good" cache_size is here. 64MB feels right given the kind of data I'm throwing around and how small it is, but this is guess work. = write to memory, not disk. Makes sense for speed.

Backend HTML Database

▲ 0

0 views