Taranis 2 weeks ago

Euravox is not an AI-first company.

But but but... Not jumping in with both feet to generative AI is heresy! In the current business climate, this is tantamount to sacrilege. How could any company, let alone a hopeful startup, take such a decision? I have my reasons, and they are difficult to refute. I'll take them in turn, then sum up at the end. This might be surprising, because the hype says that AI is supposed to be cutting everyone's costs, isn't it? Actually... not so much. Lets make the assumption that every word posted by users to Euravox is going to be passed through (at least) one LLM. Let's assume that we scale to about a million active users, and that each user on average posts about 100 words in total each day. This equates to 100 million words per day. Let's say 200 million tokens per day to take into account encoding inefficiencies and the need to add prompts, take output, etc. This equates to needing about 2250 tokens per second (without headroom to cope with bursts, and this will be bursty traffic). On OpenAI's pricing, that's roughly $350/day, or $128k/year. Not impossible, but not cheap. Gemini would be about $800/day, or a hair under $300k/year. Bearing in mind that OpenAI is almost certainly running at a loss here, this pricing isn't sustainable, and could increase by an order of magnitude. Running in-house hardware would, to get close to like for like, need to be running something like a Llama 3 405B model, which gets about 142 tokens/sec on an NVIDIA H200. It would require 16 to meet the average token rate, so probably 32 to give some headroom. If you can even get one of those things as a non-hyperscaler, this would be well over $1M factoring in the cost of the machines and other infra needed to run them. Economically, this would break even after about 3 years, so isn't a terrible investment if you have that money to burn up front. None of these things are happy-making. Electricity usage and environmental impact Based on a public estimate for OpenAI of 0.0001 kWh per token, processing 200 million tokens per day would use 20,000 kWh per day, equivalent to about 1,738 average households. I'm not going to try to estimate other environmental impacts, because there aren't good numbers available, but the idea of being responsible for more power usage by a factor of about 8 than the entire small town I live in doesn't sit well with me. Customers just don't want it Time and time again, talking to potential future users of the Euravox platform, the one question I've been consistently asked is, "will you fill the user interface with AI bullshit, or allow AI slop on the platform?" Though I'm also working toward many other differentiating features, this is by far the most commonly mentioned. The people I've talked to hate generative AI, and don't want it. They don't want to use it, and they don't want to have to wade through slop generated by it. Many people have told me that they don't want their content used to train gen AI, and are concerned about our platform's ethics as regards gen AI. They are currently stuck with legacy platforms whose ethics frankly disgust them. It's never a bad idea to listen to what your customers are telling you. Why would I use it, anyway? Certainly not for generating content or creating our own internal bots. It's hard to imagine anything that would annoy our users more, or that would be more likely to have them never sign up in the first place. The one – actually good – reason to use AI in social media is trust and safety. This encompasses moderation, but is a little more general than that in scope. Most social media platforms restrict content that would be damaging to users. In the EU, this is actually a legal requirement. Platforms define what they mean by harmful content differently – the US platforms are now mostly perfectly OK with fascistic content and even content that advocates for causing harm to certain minorities. We are not intending to allow such content. Almost all platforms do block CSAM (child sexual abuse material) – this isn't optional in most jurisdictions around the world, and rightly so. The difficult part is providing trust and safety at scale . Going back to the theoretical 1M users posting once a day, it would require 10,000 human moderators handling 100 messages a day each in order to keep up, and this is just practially and financially unviable. Moreover, it's a not talked about but actually widely known inside the industry problem that exposing human moderators to CSAM and other really bad material damages them psychologically. In practice, this means that moderators should only be allowed to do that work for a few months at most in a lifetime. The only practical alternative here is machine learning. Note that I used the term machine learning, not generative AI. I did that for a reason. Generative AI gets the news column inches, but it's only a latecomer in the machine learning world, and by no means the be all and end all. In fact, it's not actually particularly effective at trust and safety tasks – it tends to have a high failure rate, with too many false positives and false negatives, both for text and image classification, to be genuinely useful. There are older techniques that actually work better for this specific use case, not based on transformers, that are capable of getting a far lower error rate at a tiny fraction of the computational cost. These are, fortunately, small models that don't need huge, expensive NVIDIA datacenter-grade cards, and often get acceptable performance just runing on a CPU or (much smaller) integrated GPU. It's fair to point out that text classification is also extremely useful for helping to understand a user's interests. This tends to be necessary for a few reasons – showing them ads that are related to their interests is a big one, as is helping them to discover people, groups and posts (and, at some point, other media) that is likely to be interesting to them. This would be the biggest use case for gen AI (and was the basis of the numbers that I showed in the first section above), but again, practical experiments have shown it to not be particularly effective, with far lighter weight, lower CPU/GPU, lower power approaches with negligible environmental impact being far preferable. Query-by-Example – old-skool algorithmic magic to the rescue? I have a personal history of having worked on text retrieval, back in the day. Pre-Google Search, back at a time when the internet existed but the web had just been invented. Since this was right in the middle of the original AI winter, this was the way to do interesting things with natural language at scale. The system I developed, back in the mid-90s, was orginally aimed at being the backend for an email client that would give very fast search capabilities. Web search engines weren't a thing yet. The email client never got implemented, but the code did end up back-ending a real-time financial news and price distribution system. Frankly, it was way ahead of its time. Way too far ahead to actually be successful as a product. I still own the IP, I still have the code sitting in an archive somewhere. I'd imagine that it would still work fine if I hacked it to work on modern hardware. One of the things that system could do was query by example. This isn't a very well known technique these days, but in effect what it means is doing text retrieval by giving the search engine a piece of text, and having it return a number of similar texts, ranked for similarity. I've implemented basically the same algorithm in the Euravox back-end. Not exactly the same – it's decades later and I've learned a few things in the time since then – but I've built something very similar. It's much faster than the old system, which would take about 200ms to index a new message (admittedly on single processor Pentium 1 servers, but it was disk-, not CPU-bound even then). With careful attention to detail, I can currently index the entire text of Mary Shelley's Frankenstein (thank you Project Gutenberg for my test data!) in about 16ms on a single node. 200 words takes somewhere in the microseconds, so even a single node could potentially be close to handling 1M users. Of course, I'd never attempt that with a single node, but it's comforting to see those numbers all the same. And yes, the new implementation can also do QBE. This turns out to be surprisingly useful – rather than taking a modern approach of, for example, running all of our text through (computationally expensive) sentence transformers and using a vector database to do the matching (which is also computationally expensive, particularly at scale), I can just throw some text at the search engine and have it directly tell me where to look for similar text. This solves (most of) the text classification problem without needing machine learning at all. My (currently incomplete, lacking some caching, so slower than the production code will be) can do a QBE query typically in about 20ms. On one node. I suspect that this may go down to about 5ms or less with optimization. I'd probably run a query like this per user maybe once an hour or so. For 1M users, this equates to 136 QPS, or about 7.3ms per query. So we're in the ballpark of 1 to 2 nodes to handle that traffic. It remains to be seen whether QBE will be effective for harmful material detection, but I suspect that it might be. To compare like for like, let's assume we actually need 10 nodes (because reasons). Let's assume that these are fairly chunky Intel servers running close to flat out, using about 1kW each on average, so 240kWh per day. That's 83 times less power than a gen AI approach (1.2%), with no need to give money to Google or OpenAI. I am currently evaluating the possibility of using ARM-based hardware, which is far more power efficient than Intel, which can be about 5 - 10 times more power efficient again. I'll take it. Vibe Coding I wish I didn't have to talk about this, but this is an Oh Hell No for me. I have so many reasons, but: The Euravox code is old-fashioned, corn-fed, hand-crafted artisanal goodness. And probably a tenth the size it would have been had it been vibe coded. I'm managing to go quite fast enough, thanks, and I'm not piling on tech debt that I'll have to repay later. I'm building for massive scale from the outset. Conclusions My feelings toward gen AI are mostly that of disappointment. When I first played with a transformer-based chatbot, it was an unreleased, internal-only thing when I was working at Google. It was creepily good (and the same one I think that led to an engineer declaring it sentient, causing a bit of a ruckus at the time). It was also obviously not ready for prime time, because it would devolve into bizarre weirdness and (to be honest, like current publicly accessible models) could make some dangerous suggestions to a gullible user. To their credit, Google didn't release it. Then OpenAI threw caution to the wind, and the current madness started, dragging the rest of the world along with it. When people talk publicly and openly about a bubble in an industry, listen to them. I was around for the dot com crash. I was actually consulting for what I think was the first UK company to go bang ignominiously (another story for another time). It was bad then. An AI bubble popping will inevitably be much worse. I suspect that the entire tech industry will be hit with the backlash – economists are saying that the only reason the US economy isn't officially in recession is due to AI investment spending. This means an immediate US recession, and a hard one, because the AI bubble is said to be about 17 times bigger than the dot com bubble, and three times bigger than subprime. My prediction is that when it happens – not if – any business that has upended its internal processes to depend on gen AI will be in immediate and very deep shit. OpenAI is running at a huge loss, so they will be gone, Enron-style. Other AI companies who exist essentially as wrappers around OpenAI will evaporate overnight, as will any business plan valued on 'it'll be incredibly valuable eventually when blah blah AI blah blah' principles. Google will survive, though their share price will tank. They may end up with datacenter overcapacity, but that'll probably just mean that they will lay off a bunch of people to save money and then wait for organic growth to catch up. Companies who jumped on the AI bandwagon without a real plan for revenue will rip it out, and most likely go back to offshoring jobs and other similar tactics. NVIDIA won't go bankrupt, but its share price will tank. Oh and Elon, with the AI datacenters-in-space grift. I've written about this elsewhere, but at that time it wasn't clear why this thing was being talked about at all. Technically it's basically a completely stupid idea, practically it'll likely never fly at all. What it actually appears to be is a grift to pump SpaceX's share price at its forthcoming IPO by a factor of two. I was a little surprised to hear Google go along with this, because they should certainly know better, but when I found out later that they own 9% of SpaceX, and stand to make about an extra 30 billion dollars by overvaluing the company on the basis of that incredibly stupid idea, everything started to make sense. It's vapourware, and intended specifically to fleece investors. Needless to say, if the AI bubble pops before the IPO, there will be no IPO. If the AI bubble pops after the IPO, the project will be shelved with a 'too bad, so sad, market has gone away' excuse to the markets. Me? I will use some machine learning where appropriate. Likely non of it being generative. Image classification is one area where neural networks are the only game in town, and it's critically needed in order to detect CSAM. I probably won't use neural networks on text at all if QBE works as well as it's doing in experiments currently. But I am expecting the AI bubble to crash, hard, and am setting up Euravox specifically in such a way that it will survive and thrive through such a storm. This doesn't require rocket science – avoiding tech that will become difficult or unacceptably expensive, or unethical to access, whilst being very careful to maximize efficiency so we can run on minimal hardware, minimal electricity and with as small an environmental impact as possible just makes sense. I don't want code in my codebase written by someone I don't have access to, particularly early in the project I want to know for damned sure that I actually own all the necessary copyright, and that I haven't accidentally plagiarised something that could bite me later I've been coding for nearly 50 years at this point. I am better at it than the vast majority of people whose code was used to train gen AI. At best, it will generate code that's not as good as mine. Bugs cost exponentially more to handle, further down the pipe. If you can catch a bug in the design, it's 10 times cheaper than catching it in development, which is 10 times cheaper than catching it in production. Gen AI doesn't change this, and in fact goes directly to dumping all of your bugs into production. Gen AI generates bad code (boilerplate, not factored into libraries, often just plain weird ). I don't find it actually saves time. It might get you to 80% (of a far less computationally difficult project than Euravox) quickly, but that last 20% will bite your shiny metal ass. Hard.

Business AI

▲ 0

0 views

Taranis 1 months ago

Datacenters in space are a terrible, horrible, no good idea.

In the interests of clarity, I am a former NASA engineer/scientist with a PhD in space electronics. I also worked at Google for 10 years, in various parts of the company including YouTube and the bit of Cloud responsible for deploying AI capacity, so I'm quite well placed to have an opinion here. The short version: this is an absolutely terrible idea, and really makes zero sense whatsoever. There are multiple reasons for this, but they all amount to saying that the kind of electronics needed to make a datacenter work, particularly a datacenter deploying AI capacity in the form of GPUs and TPUs, is exactly the opposite of what works in space. If you've not worked specifically in this area before, I'll caution against making gut assumptions, because the reality of making space hardware actually function in space is not necessarily intuitively obvious. The first reason for doing this that seems to come up is abundant access to power in space. This really isn't the case. You basically have two options: solar and nuclear. Solar means deploying a solar array with photovoltaic cells – something essentially equivalent to what I have on the roof of my house here in Ireland, just in space. It works, but it isn't somehow magically better than installing solar panels on the ground – you don't lose that much power through the atmosphere, so intuition about the area needed transfers pretty well. The biggest solar array ever deployed in space is that of the International Space Station (ISS), which at peak can deliver a bit over 200kW of power. It is important to mention that it took several Shuttle flights and a lot of work to deploy this system – it measures about 2500 square metres, over half the size of an American football field. Taking the NVIDIA H200 as a reference, the per-GPU-device power requirements are on the order of 0.7kW per chip. These won't work on their own, and power conversion isn't 100% efficient, so in practice 1kW per GPU might be a better baseline. A huge, ISS-sized, array could therefore power roughly 200 GPUs. This sounds like a lot, but lets keep some perspective: OpenAI's upcoming Norway datacenter is intending to house 100,000 GPUs, probably each more power hungry than the H200. To equal this capacity, you'd need to launch 500 ISS-sized satellites. In contrast, a single server rack (as sold by NVIDIA preconfigured) will house 72 GPUs, so each monster satellite is only equivalent to roughly three racks. Nuclear won't help. We are not talking nuclear reactors here – we are talking about radioisotope thermal generators (RTGs) , which typically have a power output of about 50W - 150W. So not enough to even run a single GPU, even if you can persuade someone to give you a subcritical lump of plutonium and not mind you having hundreds of chances to scatter it across a wide area when your launch vehicle explosively self-disassembles. Thermal Regulation I've seen quite a few comments about this concept where people are saying things like, "Well, space is cold, so that will make cooling really easy, right?" Really, really no. Cooling on Earth is relatively straightforward. Air convection works pretty well – blow air across a surface, particularly one designed to have a large surface area to volume ratio like a heatsink, will transfer heat from the heatsink to the air quite effectively. If you need more power density than can be directly cooled in this way (and higher power GPUs are definitely in that category), you can use liquid cooling to transfer heat from the chip to a larger radiator/heatsink elsewhere. In datacenters on Earth, it is common to set up cooling loops where machines are cooled via chilled coolant (usually water) that is pumped around racks, with the heat extracted and cold coolant returned to the loop. Typically the coolant is cooled via convective cooling to the air, so one way or another this is how things work on Earth. In space, there is no air. The environment is close enough to a hard, total vacuum as makes no practical difference, so convection just doesn't happen. On the space engineering side, we typically think about thermal management , not just cooling. Thing is, space doesn't really have a temperature as-such. Only materials have a temperature. It may come as a surprise, but in the Earth-Moon system the average temperature of pretty much anything is basically the same as the average temperature of Earth, because this is why Earth has that particular temperature. If a satellite is rotating, a bit like a chicken on a rotisserie, it will tend toward having a consistent temperature that's roughly similar to that of the Earth surface. If it isn't rotating, the side pointing away from the sun will tend to get progressively colder, with a limit due to the cosmic microwave background, around 4 Kelvin, just a little bit above absolute zero. On the sunward side, things can get a bit cooked, hitting hundreds of centigrade. Thermal management therefore requires very careful design, making sure that heat is carefully directed where it needs to go. Because there is no convection in a vacuum, this can only be achieved by conduction, or via some kind of heat pump. I've designed space hardware that has flown in space. In one particular case, I designed a camera system that needed to be very small and lightweight, whilst still providing science-grade imaging capabilities. Thermal management was front and centre in the design process – it had to be, because power is scarce in small spacecraft, and thermal management has to be achieved whilst keeping mass to a minimum. So no heat pumps or fancy stuff for me – I went in the other direction, designing the system to draw a maximum of about 1 watt at peak, dropping to around 10% of that when the camera was idle. All this electrical power turns into heat, so if I can draw 1 watt only while capturing an image, then turn the image sensor off as soon as the data is in RAM, I can halve the consumption, then when the image has been downloaded to the flight computer I can turn the RAM off and drop the power down to a comparative trickle. The only thermal management needed was bolting the edge of the board to the chassis so the internal copper planes in the board could transfer any heat generated. Cooling even a single H200 will be an absolute nightmare. Clearly a heatsink and fan won't do anything at all, but there is a liquid cooled H200 variant. Let's say this was used. This heat would need to be transferred to a radiator panel – this isn't like the radiator in your car, no convection, remember? – which needs to radiate heat into space. Let's assume that we can point this away from the sun. The Active Thermal Control System (ATCS) on the ISS is an example of such a thermal control system. This is a very complex system, using an ammonia cooling loop and a large thermal radiator panel system. It has a dissipation limit of 16kW, so roughly 16 H200 GPUs, a bit over the equivalent to a quarter of a ground-based rack. The thermal radiator panel system measures 13.6m x 3.12 m, i.e., roughly 42.5 square metres. If we use 200kW as a baseline and assume all of that power will be fed to GPUs, we'd need a system 12.5 times bigger, i.e., roughly 531 square metres, or about 2.6 times the size of the relevant solar array. This is now going to be a very large satellite, dwarfing the ISS in area, all for the equivalent of three standard server racks on Earth. Radiation Tolerance This is getting into my PhD work now. Assuming you can both power and cool your electronics in space, you have the further problem of radiation tolerance. The first question is where in space? If you are in low Earth orbit (LEO), you are inside the inner radiation belt, where radiation dose is similar to that experienced by high altitude aircraft – more than an airliner, but not terrible. Further out, in mid Earth orbit (MEO), where the GPS satellites live, they are not protected by the Van Allen belts – worse, this orbit is literally inside them. Outside the belts, you are essentially in deep space (details vary with how close to the Sun you happen to be, but the principles are similar). There are two main sources of radiation in space – from our own star, the Sun, and from deep space. This basically involves charged particles moving at a substantial percentage of the speed of light, from electrons to the nuclei of atoms with masses up to roughly that of oxygen. These can cause direct damage, by smashing into the material from which chips are made, or indirectly, by travelling through the silicon die without hitting anything but still leaving a trail of charge behind them. The most common conseqence of this happening is a single-event upset (SEU), where a direct impact or (more commonly) a particle passing through a transistor briefly (approx 600 picoseconds) causes a pulse to happen where it shouldn't have. If this causes a bit to be flipped, we call this a SEU. Other than damage to data, they don't cause permanent damage. Worse is single-event latch-up. This happens when a pulse from a charged particle causes a voltage to go outside the power rails powering the chip, causing a transistor essentially to turn on and stay on indefinitely. I'll skip the semiconductor physics involved, but the short version is that if this happens in a bad way, you can get a pathway connected between the power rails that shouldn't be there, burning out a gate permanently. This may or may not destroy the chip, but without mitigation it can make it unusable. For longer duration missions, which would be the case with space based datacenters because they would be so expensive that they would have to fly for a long time in order to be economically viable, it's also necessary to consider total dose effects . Over time, the performance of chips in space degrades, because repeated particle impacts make the tiny field-effect transistors switch more slowly and turn on and off less completely. In practice, this causes maximum viable clock rates to decay over time, and for power consumption to increase. Though not the hardest issue to deal with, this must still be mitigated or you tend to run into a situation where a chip that was working fine at launch stops working because either the power supply or cooling has become inadequate, or the clock is running faster than the chip can cope with. It's therefore necessary to have a clock generator that can throttle down to a lower speed as needed – this can also be used to control power consumption, so rather than a chip ceasing to function it will just get slower. The next FAQ is, can't you just use shielding? No, not really, or maybe up to a point. Some kinds of shielding can make the problem worse – an impact to the shield can cause a shower of particles that then cause multiple impact at once, which is far harder to mitigate. The very strongest cosmic rays can go through an astonishing amount of solid lead – since mass is always at a premium, it's rarely possible to deploy significant amounts of shielding, so radiation tolerance must be built into the system (this is often described as Radiation Hardness By Design, RHBD). GPUs and TPUs and the high bandwidth RAM they depend on are absolutely worst case for radiation tolerance purposes. Small geometry transistors are inherently much more prone both to SEUs and latch-up. The very large silicon die area also makes the frequency of impacts higher, since that scales with area. Chips genuinely designed to work in space are taped out with different gate structures and much larger geometries. The processors that are typically used have the performance of roughly a 20-year-old PowerPC from 2005. Bigger geometries are inherently more tolerant, both to SEUs and total dose, and the different gate topologies are immune to latch up, whilst providing some degree of SEU mitigation via fine-grained redundancy at the circuit level. Taping out a GPU or TPU with this kind of approach is certainly possible, but the performance would be a tiny fraction of that of a current generation Earth-based GPU/TPU. There is a you-only-live-once (my terminology!) approach, where you launch the thing and hope for the best. This is commonplace in small cubesats, and also why small cubesats often fail after a few weeks on orbit. Caveat emptor! Communications Most satellites communicate with the ground via radio. It is difficult to get much more than about 1Gbps reliably. There is some interesting work using lasers to communicate with satellites, but this depends on good atmospheric conditions to be feasible. Contrasting this with a typical server rack on Earth, where 100Gbps rack-to-rack interconnect would be considered at the low end, and it's easy to see that this is also a significant gap. Conclusions I suppose this is just about possible if you really want to do it, but I think I've demonstrated above that it would firstly be extremely difficult to achieve, disproportionately costly in comparison with Earth-based datacenters, and offer mediocre performance at best. If you still think this is worth doing, good luck, space is hard. Myself, I think it's a catastrophically bad idea, but you do you.

Ruby Cloud Hardware

▲ 0

0 views

Taranis 1 months ago

Trans Masc Voice Training Tool

My husband is going through voice training at the moment, and was already using the original version of this tool. By request, I've created a modified version that, rather than beeping when you go below a target pitch, beeps when you go above it. In other respects, the tool is identical. You can download it here: Unless you already happen to have it installed, you also need to download the Pd (PureData) audio synthesizer engine, which is free. You can get in here: To set the pitch at which the tool beeps, the box marked 'Low Freq', which defaults to 150, can be changed. You can either click on it and edit the number, or click and drag up or down to tweak it as you use your mouse. If you want to save your changes for next time, Save As... from Pd's File menu will do that. As always, this is free to download and do whatever you like with. I hope it helps!

Programming AI Open Source

▲ 0

0 views

Taranis 1 months ago

So it's going to be about appearance?

After the utter debacle surrounding the UK EHRC's initial guidance, it seems that their revised guidance has leaked. The Times has seen it – here's a non-paywalled secondary source: This is a spectactularly bad idea. It gives free rein to anyone who wants to police other people's looks the go ahead to abuse people. And I do mean all people here, cis and trans, because they really can't 'always tell'. I'm a trans woman who tends toward the high-ish femme end of the spectrum by my own preference. I pass as cis face-to-face, and after a lot of voice training also do so on the phone. I don't tend to make photos of myself very public because I value my own safety, so you're going to have to just accept this as written. Trans people are a tiny percentage of the population. Some of us 'look' trans. Many don't. But the bigger problem here is that this also applies to cis people. A substantial proportion of cis women aren't anything like as femme-presenting as me, and as-such are way more likely to be excluded. Since cis women are roughly half the population, way more cis women than trans women will be affected by this idiotic legislation, just by sheer weight of numbers. The nonsensical nature of this argument can easily be laid bare: if trans women are such a danger to cis women (which is blatantly untrue anyway, but even letting that slide for a second), why are cis-passing trans women somehow given a pass, whilst a whole spectrum of cis women are excluded on the basis that they (checks notes) might be trans? What a load of absolute fucking bollocks. Let's just see this for what it is: an attempt to force women – all women, cis and trans – into rigid, narrow gender roles, on pain of exclusion from society. And let's also not miss the fact that this has considerable racist overtones, because the standards of femininity we're talking about here are those of idealised cis white high femme presenting women. Black and brown women have historically been seen as less femme, more masculine, so this will inevitably be seen here. Can we just cut to the chase and bury this total fucking stupidity, before another genration (and most likely the EHRC's anagramatic nemesis, the ECHR) has to fight to slap it down? For the record: this legislation would basically give me a free pass. I'm really very far from wanting to accept that privilege at the cost of the freedom of other women, both cis and trans.

Media Culture

▲ 0

0 views

Taranis 2 months ago

k-Hot Holographic Encoding

It's a fairly common thing in machine learning to need to deal with categorical data. Straightforwardly, if a model had to recognize inputs and classify them into lions, tigers, giraffes and scarecrows, we might assign lion=0, tiger=1, giraffe=2 and scarecrow=3. This is categorical data. A common way to encode this is with 1-hot encoding – this involves having a vector with a 1 representing a category and a 0 representing not-that-category. So lion would be encoded [1, 0, 0, 0], tiger would be encoded [0, 1, 0, 0], giraffe would be encoded [0, 0, 1, 0] and scarecrow [0, 0, 0, 1]. Things get more complicated when something might need to be represented by more categories than one. Let's say we had an animal that was a cross between a tiger and a lion – we might want to represent this as [1, 1, 0, 0]. Obviously this isn't one-hot encoding any more – we now have k -hot encoding, for some value of k. Neural networks can be trained straightforwardly both with one-hot and k -hot encoding – they don't seem to mind. However, things get tricky when you have to deal with a very large number of possible categories. A popular approach to text analysis is bag-of-words – this involves analyzing a large amount of text, extracting its vocabularly, and creating vectors that include a 1 where a word is present in a piece of text, or 0 otherwise. In many languages, English particularly, order suprisingly word meaning effect little has. (Did you see what I did there, Yoda?) Things get a bit tricky, though, because if you're doing text retrieval, it's often the rarer words that are the most useful. It's not uncommon to end up with very large vocabularies, bigger than 100000 words. Let's say we want to classify pieces of text taken from this vocabularly – if we wanted to train against, for example, Wikipedia's article classifications, there are also roughly that many of them. If we reject those that don't have many articles associated with them, we still have over 50000 of them if we insist on at least 1000 articles per category. Even a single dense layer gets painfully big at these dimensions – given 4 byte floats representing the weights, that's 20GB right there, without even considering the space needed to support backpropagation in training. Barely fitting a single layer into an NVIDIA A6000 means we need to do better. A common technique is to throw away less common words from the vocabulary, but it's already known from text retrieval (back from the pre-neural networks are everything days) that this isn't a good idea. Similarly, we don't really want to throw away any categories we don't absolutely have to. What we really want to be able to do is compress the words and categories into much shorter, more managable vectors, but without losing information or accuracy. Sounds impossible, but it isn't. Let's say we have 100000 categories representing a bag-of-words, with a phrase size limit of (say) 200. We'd like to represent this as a k- hot vector that's managebly small, say with a size of 1024. A simple way to attempt this would be to take the category ID and to wrap it around the vector however many times are necessary to fit everything in. So if c is the category ID, and we have a vector of size 1024, we could encode c as c % 1024 (where % is the integer modulo division operator). To a simplistic level, this actually does work, but it has the problem of aliasing (I'm borrowing the term here from digital signal processing). So, for example, if we want to encode the list of categories [1, 2, 95, 110, 250, 1000] and try to figure out from the encoding what categories we started with, we might get something like: [1, 2, 28, 62, 95, 110, 250, 553, 559, ... , 99264, 99312, 99820] This is actual output – the code uses another trick to improve the results rather than simple modulo arithmetic, but we'll get on to that later. The bad news is that this list has 587 elements, and any neural network using this data has absolutely no way to tell real encodings from aliases. The trick turns out to be to represent categories with more than one '1' in the vector. Specifically, each category is encoded by exactly k '1' elements that are randomly positioned within the vector. So if we take that same list and encode it with two '1' elements per category, we get [1, 2, 95, 110, 250, 1000, 3696, 14934, 26553, 51130, 53643, 55596, 55903, 72592, 80631, 86806, 89631, 92838] This is 18 elements, a huge improvement over 587, but we can do better. Setting three '1' values per category gives us [1, 2, 95, 110, 250, 1000, 60663] so just a single alias. Setting four '1's gives us [1, 2, 95, 110, 250, 1000] which is perfect reconstruction. These six categories are being encoded as 6 x 4 = 24 ones in the vector – still quite sparse. It can be instructive to try this experimentally, varying the number of categories being encoded: The graph shows what happens as you encode more and more categories into a fixed-size vector, here 1024 in length and assuming 100000 possible categories. The pink dashed line is what you'd get by encoding into a vector the size of the number of possible categories (generally impractical). The other lines showwhat happen as k varies. There is clearly no free lunch here, but even k = 2 vastly outperforms a naive wraparound modulo encoding. Zooming in on the bottom left is instructive: Up to 40 categories or so, it's clear that k = 6 does almost as well as ideal, which considering the 98:1 compression ratio is quite impressive. Of course, conventional data compression approaches will beat this ratio many times over, but the important factor here is that this encoding is very easy for subsequent neural network layers to extract useful information from. If a neural network needs to uniquely identify a category in order to improve its loss function (which is not actually always necessary), it needs to encode the equivalent of a logical AND function. This is very straightforward for a dense layer with a relu activation function, and will easily fall out of training. Recovering encoded categories I've talked a lot about how categories are encoded, so it's a good idea to also mention how they can be decoded. We've talked about using these encodings to drive the inputs of neural networks – they can also be used to train their outputs. I recommend that if you try this, you use an output layer with a sigmoid activation function (don't use softmax, they only work for 1-hot). You should also probably use binary-crossentropy as an error function. This will result in the outputs of the neural network roughly corresponding to probabilities. To unpack these probabilities, the following algorithm will achieve this: for i in [0 .. num_categories]: m = 1.0 for p in [0 .. k]: m = m * vec[ permutation[p, i] ] cat[i] = m ^ (1/k) where num_categories is the number of categories, vec is the encoded vector, cat is the recovered categories, and permutation[p, i] is an array of permutation matrices that map category numbers onto indices into vec, where each row of the array is a different set of permutations. If we start by assuming the values in vec are probabilities in the range [0 .. 1], we use multiplication like the probabilistic equivalent to logical AND. This passes through 0 and 1 unchanged, but values like 0.5 get squashed toward zero as k increases, so we take the k- th root of the result to undo the effects of the multiplications. This approach makes it possible to use an output vector for a classifier that has a size that is a small fraction of the number of possible categories. This works in neural network training because the elements that each category are mapped to are individually trained by backpropagation, and upwind dense layers can and will implement the necessary logical structures. Kind of seems holographic, doesn't it? It's a similar principle – encoding a large, sparse data set into a short vector by creating something a bit like different paths a photon can pass down, then allowing them to interact and entangle. I'm currently attempting to use this to build a very small, lightweight text classifier that is extremely fast, of the order of a millisecond or so to process a piece of text. I'm training a variant with about 42 million parameters that seems to work fairly well. This is a direct requirement for Euravox , because whilst there are probably way more reliable classifiers out there, we have a requirement to do as much as we can with as little hardware as possible, and as low a power footprint as possible. Whilst the current buzz would have us chuck huge numbers of big GPUs at the problem and use far larger models, even LLMs, this doesn't fit either our needs or ethos. I'm definitely not up for paying a third party for tokens. And in any case, LLMs royally suck at text classification, I know, I've tried. Open source implementation I have a simple Python implementation of the encoder/decoder that I'm happy to release. I've not done so yet, but I'll have some time waiting for training runs to complete in the next few days, so watch this space. I'll edit the web version of this post to include a link, and will probably write a new post announcing a repo. A personal note I've spent most of the last 20 years being unable to publish without jumping through excessively painful hoops. It is an absolute breath of fresh air to just be able to write anything I like now!

AI Machine Learning Python

▲ 0

0 views

Taranis 2 months ago

So what happens now?

I couldn't talk about this until I was no longer with my former employer, but I now am able to announce what I'm going to do next: Like many of us, I've been concerned for some time that social media has been sliding toward becoming a collection of unwelcoming, even tangibly dangerous, hellsites. As the Trump administration has pushed these companies to capitulate to his regime, we've seen both internal changes with them being forced to abandon internal DEI programmes, as well as adopting trust and safety policies that accommodate (or are at least less unfriendly to) fascist hate speech. This is disenfranchising a substantial proportion of the population, making it less and less safe for them to participate in public life. In extremis, if Project 2025 is fully deployed, many minorites, including but certainly not limited to queer and trans people, will be pushed entirely off the internet, and likely much worse. This also makes many allies extremely uncomfortable – no decent person likes to support fascism even tangentially, but many people, myself included, depend on social media for much of their social contact. It's not good enough to pick platforms that are less fascist. We all know which platforms are the worst offenders, I need not repeat them here, but the 'better' platforms many of us drift toward are still basically run by the same cohort of billionaire and billionaire-adjacent people, and are located and controlled in the US. We need and deserve something better than this. I'm going to build it. My vision for Euravox is that it will be a fully-featured social network platform, at least as capable as any legacy platform, so nobody will feel that they are missing something by moving to it. It will support posts and chat in a way that will be familiar to Facebook users, but with richer support for things like typography and inline images. It will (eventually) support other media types too, including music, podcasts, ebooks, audiobooks and of course video. I also want to enable it to support a creator economy, with a large proportion of income redistributed fairly to creators. There will be three kinds of account: free accounts that are ad-supported, low cost ad-free paid accounts, and (still pretty low cost) creator accounts whose content is monetized. Anyone will be able to sign up for any account tier – I won't make people jump through ridiculous hoops in order to become a creator The need to carry advertising is not something I can avoid. For the system to be self-funding, in order to have a user tier that is free at the point of use, ads are the only way to do this. When free tier users consume content, I need to be able to pay the creators, and that money has to come from somewhere. Most of it will in fact go to the creators, with the rest supporting the platform infrastructure, salaries, etc. Trust and safety will be central to the platform. Hate speech will absolutely not be tolerated. Though membership will be open to anyone, the focus is on people who have been displaced from legacy platforms, so this will be reflected in the trust and safety policies and terms of use. To be clear: Euravox will not tolerate transphobia, homophobia, racism, fascism or any other similar abuse. This is not negotiable and is an immutable founding principle. As hinted by the name, the company will be European, with our main operations in Europe. I see European privacy legislation as a selling point, not a threat to our business model. Our users will own their own data and are free to download and/or delete it at any time. Users will need to grant the bare minimum of (non-exclusive) rights in order to enable the platform to legally operate, and no more. They will own and retain their copyright. My intention is to primarily use hardware directly owned by the company and to run it in multiple, geographically distributed, colocation datacenters. This is the most cost-effective way to scale the service, and dramatically cheaper than, for example, deploying on AWS or Google Cloud. It also keeps our technology owned by us from the bare metal upwards, which is very important for data sovreignty – I can't trust cloud provider(s) not to hand over our user data to bad actors. Currently the business is just me. Bluntly speaking, I'll be coding my ass off for a while getting this thing running. I'm hoping to have a closed alpha by roughly the end of the year, and a more open beta in the early new year, followed by a (minimum viable product) public rollout probably toward the end of Q1 2026. Once there is revenue, I'll be looking to hire. You'll hear about it when I do, so there's no need to bug me about it just yet. It's a bit unclear exactly how and where I'll hire people, though where legally feasible, I'll help people move from dangerous locations to Europe. I'd like to keep most roles fully remote, but I'll probably need at least something somewhere, probably in mainland Europe, in order to stage hardware before moving it out to DCs. I'm not sure where the DCs will be yet, but very likely Germany, Spain and one of the Nordic countries (TBD). I've started a very basic corp website at corp.euravox.eu – there's a bit more there, with more details about the membership tiers, etc., as well as a manifesto that makes the motivation behind doing this stuff clearer. Note that the sign-up button is disabled right now, there being nothing to sign up to just yet! I'm not looking for business partners or investors at this time. I am not willing to dilute control over the business, because I see retaining that as absolutely vital in order to be able to deliver this to the people who need it the most. I've done some careful numbers, and it should be self-supporting fairly quickly. For this same reason, Euravox will be a company, not a nonprofit, because the latter would potentially be an attack vector for someone powerful enough who wanted to weaken our user protections. The fash will hate this, and IDGAF. It's not for them. They literally have the whole rest of the internet to spew their bile. I want somewhere that the rest of us can be safe. I want to found the kind of community that existed back in the early LiveJournal days, before algorithms driven by rage bait destroyed everything. I should probably talk a bit about The Algorithm. I am well aware that many seasoned social media users, particularly those of us who started in the LiveJournal world, Usenet and even BBSes, really appreciate a time ordered view of posts by people we care about. We're going to have that as a fully supported thing. We'll also have a discovery algorithm that can show you posts about things you commonly interact with, but this will be tuned specifically for that, not for user retention (this is what causes the rage bait on other platforms, and I just am not interested in that toxicity). I want people to use Euravox because it's a nice place to be, with nice people, not because they have been deliberately addicted to it by an algorithm that manipulates them into feeling constant fear, hooking them on dopamine. I also want to make Euravox a really good platform technically, which is why I want to support other media types like books, podcasts and audiobooks, things that have been entirely overlooked by legacy social media. And music, of course, is close to my heart because, being a musician myself, I am disgusted by the way that current internet streaming services pay almost nothing to creators. They will get a far better, fairer and more transparent deal on my platform, meaning both more money and a way to interact with fans that is completely missing from legacy streaming. Realistically, Euravox is not going to replace legacy social media, but if we can even take an appreciable fraction of their users, the project will be very successful. In order to compete with us effectively, legacy social media will be forced to shift its Overton window to the left – something that's currently implausible given their alignment with the Trump administration. So we'll just take our chunk of the market, and make a better world for ourselves in the process. Getting the thing to scale, and do so quickly, will be the biggest challenge. I have a plan for that. I'm not going to talk about it much, because for most people it's technical TMI, but I may post about it at a later date. Needless to say, this isn't going to be a LAMP stack. I know what works, and what doesn't, when scaling systems to enormous size, so I'm going to build this the right shape from the outset. I'll post updates as I'm getting closer to the alpha and then beta releases. I'm also very keen to hear from anyone who has strong feelings about how they would like social media to work – obviously I can't make any promises to implement anything specific, but I don't want to take decisions in a vacuum either. Some of you probably think I'm completely freaking insane trying to build something like this, essentially solo. You'd have a point! I definitely have WTF am I doing moments. However, I have all the technical knowledge and experience I need, and I'm certainly sufficiently determined. This is also far from being my first startup. I am intending to retain control as CEO, and if anyone suggests that I should hand that over to some 'more experienced' dude they will lose a testicle. This project is going to happen. Many people reading this will probably be wondering what they can do to help. I'm not looking to do a kickstarter/gofundme or anything like that. All I really ask is that, once the time comes, create an account and (hopefully!) enjoy helping each other create the community we all deserve. 😸

Business Media

▲ 0

0 views

Taranis 2 months ago

Today is my last day at Google

It's been a while. I joined Google in 2016 at the mothership in Mountain View, California. I'd just left 10 years at NASA, and was taking a leap into the unknown. I'd never worked for a large American company before, and Google seemed more than a little Bay Area culty at the time. I started as a Level 6 staff engineer, something I had absolutely no idea the definition for, and nobody really ever explained it. I stepped into a tech lead position in the Gin team – Google has lots of internal names for systems and products that are meaningless outside the company, so I should say that this has absolutely nothing to do with the Golang Gin web framework! It was actually an insider risk/counterintelligence auditing system inteded to do things like ferret out foreign spies, employees doing things they shouldn't, etc. It worked pretty well. A few months in, my manager suggested that maybe Gin could do well to become an external product. This led me to doing quite a bit of travel, visiting other Google sites in Chicago, London and Zurich, and many meetings with legal and VPs persuading them that this could work. At the time, Cloud and (what became known later as) Workspace were blocked on some very large deals because customers were concerned that Google employees might be able to access their data. Of course, if they did, Gin would most likely spot that, but those customers weren't happy to take that on trust. So I was pushed to move to Chicago, build out a team, and to externalize Gin. This wasn't straightforward – the data rate was scarily huge, and though our internal analysts could handle the data, it would be incomprehensible to outsiders. The solution became known as Access Transparency , which my team delivered in an astonishingly short time scale. I heard that it unlocked a lot of business once it rolled out. (Full disclosure: the Access Approval system came later, and wasn't my project) At that time, Trump's first term was happening, and as a queer immigrant I was getting more and more scared. I was checking the news more than once a day to see whether I needed to put an escape plan into effect. I had several, made a little simpler because Chicago is very close to the Canadian border, but I ended up taking a less dramatic approach than bailing unprepared. My mother passed away just before I moved to Chicago, and my dad's health was failing. They lived on the Spanish Balearic island Mallorca at the time, which was an easy flight from Zurich. I asked my upper management, and was able to relocate to Google Zurich. I liked Zurich a lot. I still do, to be honest, even though I left for Ireland a couple of years ago. Whilst there, I initially stayed in Security & Privacy, working in the Data Governance team for about 18 months. A big reorg killed my project, so I moved over to Cloud Site Reliability Engineering, and became a manager. A team had grown a bit too big, and was being split into two, so I ended up picking up half of the new team, which became known as the Zurich side of the Open Source Analytics SRE team. We ran a few systems, most notably Google's managed Hadoop-on-Cloud product, Dataproc . A year later, another reorg rolled our team's responsibilites back into the development team, and after a bit of finagling and wheeler dealing the majority of the old team went with me to YouTube, becoming the Zurich Trust & Safety SRE team. Then COVID happened. Things got weird, for everyone. I was doing whatever I could to support my team, even organizing some basic cooking lessons via Google Meet when it became apparent that some of them had never cooked for themselves and this was hard for them as they were having to depend on delivered groceries. While I was in Zurich I was diagnosed with a couple of autoimmune things, and was strongly advised to avoid catching viruses (not just COVID). Of course, I did manage to catch the damned thing just as everything shut down, before testing was widely available. Medical services were swamped, so people were being told not to show up unless they really were dying, so I just basically holed up for about 3 weeks. It sucked, but I was OK, thankfully no long-COVID symptoms. My spouse lost their genetic father to the virus soon afterwards. But it did seem to me that Google would probably return to office in a way that might be safe(ish) for most people, but not me. I asked to stay working from home, and was told I could, but I had to give up my management job. I was quite sad about that, but it was what it was and there wasn't much I could do about it. I brought on one of my team members as my replacement, and went back to a tech lead position. This lasted a few months. I took on an uber-tech-lead (UTL) position in YouTube SRE, but I really didn't get any traction. UTL is a difficult job – you are essentially tech leading tech leads, which critically depends on support from upper management. I really liked YouTube and the people, but I'd describe it as a bit of a supertanker – once it's going it's going, and changing direction even slightly is very difficult. An opportinity came up to move over to another team that was doing what these days would probably be called observability research – basically data science on logs, in order to detect and diagnose problems in large scale systems. This was a lot of fun – I got to throw everything from traditional statistical analysis to digital signal processing to various kinds of machine learning (genetic algorithms as well as neural networks) at huge amounts of log data. About half way through that project I moved to Ireland. My dad had passed away the previous year, I was starting to think forwards to post-Google life, and I realised that staying in Switzerland probably wasn't going to work. Though I loved the country, financially it didn't add up. Cost of living, particularly for property, was out of reach, even on a fat salary. Moving to Ireland resulted in a large pay cut because of the way Google operates payroll relative to local norms, but it worked out to be a wash. With even an apartment where I was living being the 1m+ CHF price bracket, I just couldn't see how I'd be able to retire once I hit retirement age. In Ireland, I was able to buy a house outright, free and clear, no mortgage, for what would have barely been a downpayment in Switzerland. I bought an old post office in a small village in the west of Ireland, still intact from the day it closed in 2009, with a 5 bedroom apartment above and several outbuildings and a fairly large garden. The research project came to an end a bit over a year ago (it had always been time-limited), so I moved over to the Turnups org (internally known as the Turnips, for probably obvious reasons). This is the bit of Cloud that 'turns up' clusters to cope with expansion, much of which around AI compute capability recently. About a year ago, I hit a point where I realised that there was some trans-related stuff that I'd been putting off for decades, and I really needed to deal with. With the wind blowing in a scary direction for trans people everywhere, passing has become a significant safety issue. I'd already had facial surgery in 2001 with its pioneer Dr Doug Ousterhout in San Francisco, but there were some less than ideal outcomes from the original surgery, and time and gravity had taken its toll. So early this year I took extended medical leave and went through several surgeries to deal with that, with Facialteam in Spain. I also started working with a voice therapist. It's been a long road, but I'm happy with the results. Being on extended medical leave gave me a lot of time to think, and I came to the realisation that staying long term at Google probably wasn't the right thing to do. The company doesn't feel like the company I joined – its values are different, and it seems increasingly likely to acquiesce to pressure from the Trump administration. I can easily see an executive order being issued forcing all companies with US government contracts (which of course includes Google, particularly Google Cloud) to fire trans workers worldwide. Even in Ireland, in the EU, where something like that would be flat out illegal, it felt like a career death sentence, so I started to look more seriously about what should happen next. Anyone who knows me knows that I'm anything but passive, so I wasn't keen on just waiting for the axe to fall, however good my salary might have been. So I decided to leave the company on my own terms. Yes, I have plans. But I'm not announcing them until I'm no longer an employee. Watch this space!

Security Go

▲ 0

0 views

Taranis 3 months ago

LLMs are a failure. A new AI winter is coming.

Like many people, I got pretty excited when it was discovered that the transformer neural network architecture appeared to break through many years of stagnation in AI research. Chatbots suddenly had emergent capabilities, derived almost entirely from unstructured, unsupervised learning, far surpassing older technologies. My first experiences were with unreleased models, pre-ChatGPT, and I was seriously impressed. Though these early, small, models would often mess up, even generating streams of garbage text, when they worked they worked. Spookily well. I completely understand why some people at the time thought they were sentient – this is a whole other discussion for another time. People were saying that this meant that the AI winter was over, and a new era was beginning. I should explain for anyone who hasn't heard that term before, that way back in the day, when early AI research was seemingly yielding significant results, there was much hope, as there is now, but ultimately the technology stagnated. First time around, AI was largely symbolic – this basically means that attempts to model natural language understanding and reasoning were based essentially on hard-coded rules. This worked, up to a point, but it was soon clear that it was simply impractical to build a true AI that way. Human language is too messy for mechanised parsing to work in a general way. Reasoning required far too much world knowledge for it to be practical to write the code by hand, and nobody knew how to extract that knowledge without human intervention. The other huge problem with traditional AI was that many of its algorithms were NP-complete, which meant that whilst a lot of the time you got a result, often you just didn't, with the algorithm taking an arbitrarily long time to terminate. I doubt anyone can prove this – I certainly wouldn't attempt it – but I strongly suspect that 'true AI', for useful definitions of that term, is at best NP-complete, possibly much worse. Though quantum computing in principle could give some leverage here, none of the technologies currently being built or that are being considered feasible are likely to be useful. Just not enough qubits to represent the kinds of data that would need to be processed – this is a way, way harder problem than trying to reverse encryption secured by the difficulty of prime factorization. So then came transformers. Seemingly capable of true AI, or, at least, scaling to being good enough to be called true AI, with astonishing capabilities. For the uninitiated, a transformer is basically a big pile of linear algebra that takes a sequence of tokens and computes the likeliest next token. More specifically, they are fed one token at a time, which builds an internal state that ultimately guides the generation of the next token. This sounds bizarre and probably impossible, but the huge research breakthrough was figuring out that, by starting with essentially random coefficients (weights and biases) in the linear algebra, and during training back-propagating errors, these weights and biases could eventually converge on something that worked. Exactly why this works is still somewhat mysterious, though progress has been made. Transformers aren't killed by the NP-completeness and scaling problems that caused the first AI winter. Technically, a single turn-of-the-handle, generating the next token from the previous token and some retained state, always takes the same amount of time. This inner loop isn't Turing-complete – a simple program with a while loop in it is computationally more powerful. If you allow a transformer to keep generating tokens indefinitely this is probably Turing-complete, though nobody actually does that because of the cost. Transformers also solved scaling, because their training can be unsupervised (though, practically they do often need supervised training in order to create guardrails against dangerous behaviour). It is now standard practice to train new models on just about every book ever written and everything that can be scraped from the internet. That's the good news. That was the good news. But we've gone past that point now, and we are now all up against the reality of widespread use of transformers. All transformers have a fundamental limitation, which can not be eliminated by scaling to larger models, more training data or better fine-tuning. It is fundamental to the way that they operate. On each turn of the handle, transformers emit one new token (a token is analogous to a word, but in practice may represest word parts or even complete commonly used small phrases – this is why chatbots don't know how to spell!). In practice, the transformer actually generates a number for every possible output token, with the highest number being chosen in order to determine the token. This token is then fed back, so that the model generates the next token in the sequence. The problem with this approach is that the model will always generate a token, regardless of whether the context has anything to do with its training data. Putting it another way, the model generates tokens on the basis of what 'looks most plausible' as a next token. If this is a bad choice, and gets fed back, the next token will be generated to match that bad choice. And as the handle keeps turning, the model will generate text that looks plausible. Models are very good at this, because this is what they are trained to do. Indeed, it's all they can do. This is the root of the hallucination problem in transformers, and is unsolveable because hallucinating is all that transformers can do. I would conjecture that this is another manifestation of the NP-completeness wall that slammed symbolic AI, causing the first AI winter. It's always possible to turn an NP-complete algorithm into one that runs quickly, if you don't mind that it fails to generate any output if you hit a timeout. The transformer equivalent of this is generating plausible, wrong, hallucinated output in cases where it can't pattern match a good result based on its training. The problem, though, is that with traditional AI algorithms you typically know if you've hit a timeout, or if none of your knowledge rules match. With transformers, generating wrong output looks exactly like generating correct output, and there is no way to know which is which. Practically, this manifests as transformers generating bad output a percentage of the time. Depending on the context, and how picky you need to be about recognizing good or bad output, this might be anywhere from a 60% to a 95% success rate, with the remaining 5%-40% being bad results. This just isn't good enough for most practical purposes. More concerning is the fact that larger transformer models produce extremely plausible bad output, that can only be identified as bad by genuine experts. The rumour mill has it that about 95% of generative AI projects in the corporate world are failures. This isn't really surprising to anyone who was around for the dot com bubble, where corporate executives all seemed to assume that just being online would somehow transform their businesses, and that new ventures only really needed user count and that the financials would sort themselves out later. The same thing is happening again with generative AI, though the numbers are far larger. It is absolutely inevitable that the bubble will burst, and fairly soon. Expect OpenAI to crash, hard, with investors losing their shirts. Expect AI infra spends to be cancelled and/or clawed back. Expect small AI startups that aren't revenue positive to vanish overnight. Expect use cases based on unrealistic expectations of LLM capabilites to crash the hardest. A good example is transformers used to assist in programming, or to generate code from scratch. This has convinced many non-programmers that they can program, but the results are consistently disastrous, because it still requires genuine expertise to spot the hallucinations. Plausible hallucinations in code often result in really horrible bugs, security holes, etc., and can be incredibly difficult to find and fix. My own suspicion is that this might get you close to what you think is finished, but actually getting over the line to real production code still requires real engineering, and it's a horrible liability to have to maintain a codebase that nobody on the team actually authored. Transformers must never be used for certain applications – their failure rate is unacceptable for anything that might directly or indirectly harm (or even significantly inconvenience) a human. This means that they should never be used in medicine, for evaluation in school or college, for law enforcement, for tax assessment, or a myriad of other similar cases. It is difficult to spot errors even when you are an expert, so nonexpert users have no chance whatsoever. The technology won't disappear – existing models, particularly in the open source domain, will still be available, and will still be used, but expect a few 'killer app' use cases to remain, with the rest falling away. We're probably stuck with spammy AI slop, and with high school kids using gen AI to skip their boring homework. We'll probably keep AI features in text editors, and a few other places. I know that this is a currently-unpopular opinion. It is based on solid science, however. For what it's worth, I founded a chatbot company back in the late 90s, based on symbolic AI technology, that went splat in the dot com crash. I've been around this block, and I've stayed up to date on the technology – I've built my own transformer from scratch, and have experimented quite a bit. My advice: unwind as much exposure as possible you might have to a forthcoming AI bubble crash. Winter is coming, and it's harsh on tulips.

AI Machine Learning

▲ 0

0 views

Taranis 7 months ago

Crawling the Queer Web

I've talked previously about transarchive.eu , an attempt to locate and archive information that is important to the LGBTQIA+ community, and to trans people in particular. When the project started, I thought it might have ended up with maybe a couple of hundred human-curated sites, but this vision didn't last long when it met reality squarely in the face. When I started digging into the problem, the numbers didn't add up. Firstly, supporting of the order of a couple of hundred sites wouldn't remotely scratch the surface. Secondly, the overhead of manual curation and moderation was, even at that scale, impractical. Though I might be OK with spending a few hours at a time going through URLs and writing listings for them, I timed myself and did the math. I also asked my long-suffering spouse to have a go too, and I timed them too. Though they were enthusiastic in principle, they lasted about 20 minutes and drifted off to do something else. I couldn't really blame them – it was hard work and not really fun, particularly when you end up having to read the kind of material that, as a trans person, is extremely corrosive to your wellbeing. Even then, with a modest scale-up to 1000 supported sites, I figured out that this would require hundreds of volunteers donating an hour or so each week even to make some kind of headway. The phenomenon that human moderation can be damaging is well known. The classic example, widely understood but not really talked about much in the industry, is that of material representing abuse – CSAM is the worst of this, but it's certainly not the only problematic case. Human moderators don't last long, and often end up with psychological damage. Though in our case we are less likely to encounter the more extreme material, since many (if not most?) trans people have suffered abuse themselves, exposing them (our pool of volunteer moderators) to anti-trans material would not be acceptable. It is common in the industry to use AI for this – indeed, this kind of approach was widely used long before LLMs appeared, and has been extremely successful. Though many of us have had posts rejected because of the occasional error, this is a very small price to pay relative to the real human cost that would have been the consequence of not doing this. After some experimentation, the TransArchive web crawler now uses three machine learning models: With a bit of experimentation, I managed to get this to work pretty well. The way the crawler works is it first pulls the contents of a URL, identifies its contents, and if it contains human-readable text with high probability, it analyzes the page to see whether it is of interest. If it is, the model is used a second time to generate the category, title and description. The results aren't perfect – they do need a bit of human fixup sometimes, and occasionally they mess up completely, but in a very high percentage of cases the results are directly usable. Running on a single NVIDIA RTX6000 GPU, this gives 2 or 3 new entries in the database per minute. Only pages that survive this process have their outgoing links added to the database – this hopefully guides the system toward only crawling parts of the web that are directly interesting. I started wondering whether this is actually sufficient. Will this actually crawl the entire queer web? Some very old mathematics can help answer this. Percolation theory deals with what happens when you have a large number of interconnected nodes that are connected to other nodes with a particular probability. As a very simplistic example, imagine an (arbitrarily large) square grid, with squares coloured black with a particular probability. In the illustration above, the left grid has black squares with a probability of 0.3 (i.e., on average 3 squares in 10 are black). If you pick a black square and see how far you can get by only stepping on adjacent black squares, you can't get very far before you run out of black squares. These disconnected groups are typically called closed clusters in the literature. Compare this to the grid on the right, where nearly all (95%) of squares are coloured black. Here, you can go more or less anywhere you like, and if the grid continued like that forever, you could most likely keep walking forever without running out of black squares. Such clusters are typically called open clusters . The one in the middle is the most interesting – it's right at the critical point between clusters being primarily open and primarily closed. Plotting probability against the likelihood of an open cluster being present looks like this: Something that is interesting about this is the slope of the curve is always pretty steep, and the point at which it passes 50% (the critical probability) is quite consistent for any underlying system with the same kind of connectivity. Taking this idea and dropping it in the middle of the need to crawl the web, it's pretty clear that web pages are way more interconnected than a 2d grid, even if we only expand nodes that are of direct interest. I don't have enough stats (yet?) to run a simulation, but it seems pretty likely that our slice of the web is indeed an open cluster in its own right. All this said, the web is big, and it will take a long time to crawl everything of interest. It may not even be possible to crawl everything without scaling up our infrastructure to an absurd level – we're not trying to be Google Search, Bing or Duck Duck Go. Therefore, it is still useful to have a suggest URL option on the web site, because even though the crawler will probably eventually hit every site of interest, it would be more useful to get to the best ones sooner. All this said, TransArchive is still an ongoing project. It will get more tweaks in the future, both to the UI and with respect to improving data quality, though it's now at a point where it won't be my 100% focus for a while. If anyone wants to contribute to cleaning up the data, ping me , and I can potentially add you as a (human) moderator. Eventually, when the database is big enough and enough cleanup has been done, I would like to have a go at fine-tuning the models so they do a slightly more consistent job, but this will take 50k - 100k examples, and we're nowhere close yet. A small, very fast, lightweight model whose purpose is, given a web page, to decide whether it contains human-readable text at all, and if so, in what language. Currently we don't do anything special with the language other than log it for later use, but eventually this will be useful for making TransArchive properly multilingual. A full-strength LLM configured to analyze the text on a page to decide whether it talks about something relevant to trans and/or LGBTQIA+ people. A second instance of that same model configured to summarize the page, generating a category, title and description for inclusion in the index.

AI

▲ 0

0 views

Taranis 8 months ago

Announcing transarchive.eu

I've briefly mentioned this in a few places, but I wanted to make a bigger announcement about https://transarchive.eu having gone live. It's still very much in alpha at the moment, but it's moving quickly toward being a real thing so it's time to talk about it. The rise of the extreme right wing – fascism by any other name – across the world is extremely concerning, particularly when you happen to be a member of the minority that they are most fond of harming. One of the scariest things for me is seeing information about us and created by us being systematically destroyed – this was a very bad sign in the 1930s, and it's a very bad sign now. The loss of data and scientific work is fundamental. Without this, it is harder to defend against entirely bogus appeals to 'common sense' or 'science' – if the contrary view is missing, it's easier to warp reality to support draconian policies. It's an old saw that reality has a left-wing bias – this is a simplification, of course, but travesties like the UK Cass Report and the US's more recent equivalent, based mostly on wishful thinking, bad science and made-up bullshit are difficult to unseat once they become entrenched. If the contrary (real, objective) data are simply deleted, we face decades of struggle. Secondly, and equally importantly, access to information and support will potentially get more difficult. We are already seeing large internet companies start to capitulate, so the information erasure may not be confined to governmental or government-funded sites, and may extend across social media. What happens if our support groups are shut down, or at least information about them is embargoed? What happens when popular and essential information sources online are similarly taken down? Access to gender-affirming care is becoming increasingly difficult in many countries (and is already nearly nonexistent in others, but that's another rant ), so this information is life or death for many of us. My solution isn't perfect, and honestly isn't a solution – only social change reversing the fascism would do that. But what I am attempting to do is archive as much information as possible, primarily trans-related but potentially also anything LGBTQ-related in the wider sense, as well as more mainstream resources that happen to be useful. I'm making this available via the web site at transarchive.eu . At the time of writing, this is all pretty alpha. This project is only something like 3 weeks old at this point, so having something – anything! – running is a lot better than nothing, but I really wanted to start the archival process as soon as humanly possible. Practically, the intention is that the site will serve partly as a useful index and jumping off point for all things trans-related, whilst also on a best efforts basis archiving the underlying data. Think of it as a trans-specific Internet Archive or Wayback Machine. I'm going to get a little technical here. No apologies – this might be useful to someone at some point. The site is split into a front end, which is the thing that appears in the browser, and a back end that runs on servers on the internet side. Most of the site's functionality – the things you click, scroll, interact with, etc., run entirely inside your browser or phone, which is why the site is relatively fast. The technology used here is Svelte 5, a modern Javascript framework that is relatively quick to develop but (importantly) runs very fast with relatively little server overhead. The code is written from scratch. Metadata – the list of sites, their categories, titles, descriptions, settings, the time they were last updated, etc., is all stored in a PostgreSQL database. The site data itself is stored as files in a filesystem. The server-side functionality is split into a few pieces: Site server. This is a Svelte 5/Sveltekit back end that serves the site seen in people's browsers. It also provides the API that gives the site access to data and its capability to make changes for moderation purposes. NAS. There are actually two NASes, a primary and backup. In this architecture, they store the archived site data and also database and system backups. nginx web server. This reverse-proxies the Sveltekit server and also serves the archived sites via an NFS share on the NAS, making them visible as a single domain name. This avoids the need for archive data to be routed via Sveltekit. PostgreSQL server. Nothing much to say here, it's pretty straightforward and does what you'd pretty much guess it does. Web Crawler. This is implemented in Python (my own code), using httrack to perform the actual mirroring. I may replace this later, but it was the quickest way to get something going on a quick-and-dirty-but-actually-working basis. Automod. The first iteration of the site's design omitted this piece because I naively thought that manual curation/moderation would be sufficient. I thought maybe we'd be mirroring a few hundred sites at most. I was off by a couple of orders of magnitude. Back-of-the-envelope calculations showed that I'd need to recruit literally hundreds of human moderators, each donating an hour or two of their time weekly, even to vaguely stand a chance of keeping up with the crawler. For a near-zero-budget volunteer effort that absolutely needs doing NOW, this wasn't going to work. I had to build automated sentiment/quality analysis to evaluate sites (most links are bad, broken, spammy, or even harmful). It also turned out to be very tedious and slow to have to manually set up categories, titles and descriptions, so automated text summarization was the only way I could make it work. None of this is perfect – only 3 weeks into the project, remember! – but it's working. It isn't a replacement for human moderation or curation, but it changes the job from having to do everything from scratch painfully to just going in and correcting mistakes. I'm intending writing more about how this works and the ethical issues involved, but that will be for a later post. Front end reverse proxy. This is kind-of like having my own Cloudflare-like capability, which means that the site looks like it's in East Germany, when it is actually elsewhere. I've written a bit more about this in another post . This isn't quite a traditional security and privacy problem. Normally, there is a wealth of user data that is inherently private that must be protected from external interference or eavesdropping. Here, the information we collect and disseminate is already in the public domain, in the sense that it is out there on the internet. Practically, we're not really doing anything much different to a search engine in this respect, so the site data and metadata isn't particularly sensitive. We do of course serve everything end-to-end via TLS with proper certificates. The API that the web site uses to talk to the server is protected with salted HMAC authentication with shared secrets, inside the encrypted TLS connections, but this is only really relevant if a human moderator is signed in. Normal users don't need to sign in at all. The protocol is fully stateless, so cookies are not used or needed and nothing needs to be stored in the browser. Yes, this goes a bit beyond standard Svelte/Sveltekit architecture, but I felt it was important. We don't have a need to track individual users or store information about them, so we simply just don't. The only thing we do keep (for a minimal period) is server-side logs for security and debugging purposes – this is necessary to protect against certain kinds of attack. From a user's point of view, we don't know much if anything about them, and we don't want to. From a moderator's point of view, we have their login details, but that's pretty much it. User data is therefore about as minimal as it could possibly be for a site that actually has any nontrivial functionality. That said, users' own browser histories, or any information on their communications traffic collected by their internet service provider, may still be dangerous if they are in a situation where being seen to access LGBTQ+ information might put them in danger. For these situations, we recommend viewing the site via Tor. It's not in place at the time of writing, but we intend to publish an onion endpoint so it's unnecessary for anything looking like our domain name to appear anywhere important. At the time of writing it is very early in the project's timeline, so things are admittedly rough around the edges. Not everything works perfectly. There is no N+1, let alone N+2. Data quality is a bit iffy in places – we have some duplicates, and sometimes things went really wonky with earlier versions of the crawler, and some of that data needs to be weeded out. It'll happen. The infrastructure isn't completely there yet – I need to add some hard drives to the NASes to cope with a likely large amount of incoming site data. Currently there's no GPU acceleration for the Automod subsystem, which is waiting on me getting back home a couple of weeks from now so I can upgrade the hypervisor on one of the servers so its GPU passthrough works (the VM is already set up, but currently stuck to using CPU, which means that the server is making a noise like a jet engine – the power is mostly solar so there's no particular problem with that, but it's slower than I'd like it to be right now). And the web front end 'worked fine on my machine' but now is showing its little flakies now I'm getting to prod it from other systems. I also need to put together a dev/staging/prod pipeline with proper rollouts and seamless deployment, or SREs will point and laugh. As ever, the last 20% of the effort always takes 142000% of the time. But so it goes! The biggest missing piece right now is a search engine. Since we will have a substantial collection of relevant information, it would be really nice if all of that was searchable. Of course, I'd like this to not suck, so I'd like to use modern approaches (text embeddings and vector search – the good part that came out of AI research around search – I have no particular appetite for LLM usage for that purpose). Though it's tempting to just use an off-the-shelf text search solution, I've got a fair bit of background in that area so I'm just basically unable to make myself not attempt to do it better! Everything we're using is open source, there are no paid licenses underlying anything, so it remains feasible to release our code at some point. However, it's currently a typical experimental hacky dog's breakfast, so I will want to do a lot of cleaning up and straightening out before anyone else has it inflicted on them. The plusses and minuses of open sourcing our code are tricky. I'd like to do so, even for no other reason than just getting help with the development side. I can see definite use cases outside the trans/broader queer community, e.g., to help prevent the erasure of BIPOC histories, data and mutual assistance. I'd like to see that happen, and may be willing to actually host an instance (though I'm very aware that I'm probably going up a couple of orders of magnitude in data size and bandwidth needs there). Yes, the chess piece in the top right is a queen.

Open Source Culture JavaScript Python Media

▲ 0

0 views

Taranis 8 months ago

All Roads Lead to Hellsites

I'm quite disillusioned with the way things went, all in all. I mean, back in the day, when people weren't really sure whether this newfangled thing was going to stay being called The Internet. The press seemed to like The Information Superhighway, but that was and is a stupid name. It seemed amazing back then. And it really was. Early adopters are a different crowd. It was a few years before the digital cognitive decline set in. The places we loved – LiveJournal, Usenet, even just the fact that people really used good oldfashioned email in long form – started to die as average attention spans shrank and UX became a game of how can we make this thing so braindead that a drunken sleep deprived toddler could use it at 3am. (No folks, don't feed alchohol to toddlers, even though my grandmother thought it was a neat idea) Yeahnope. That's not how it happened. It's easy to blame the dumbing down of the internet for the current state of things, but it's absolutely not the reason why hellsites happened. For this, the blame cannon needs to be pointed squarely at the dudebros (and, I think they pretty much were all bros, but do correct me if a sister missed out on the opprobrium) who wanted to get rich. VERY rich, off this Internet thing. Back in the late 90s Internet bubble, everything with a dot com up its jacksie was listing on stock markets and getting insane valuations. But the bubble popped, and market caps with lots of zeros ended up with hilariously few. There were, however, survivors. Most of the dot bombs were based on really quite idiotic business plans – no revenue, not even a plan for revenue. Bah, that was 20th Century thinking. In the new financial world, all you needed was a wing, a prayer, and sufficiently many funding rounds that hopefully nobody would notice the Ponzi scheme. I have stories, like most of us who were around the industry then. But, a few of the dudebros got really quite astonishingly lucky. They would like us to think that it was skill, but um... we're still watching you... and nope, the only plausible explanations are divine intervention from a universe with a bad sense of humour or a combination of psychopathic ruthelessness and blind luck. Business is harder than most people think. If you've not tried to actually run a business, and had yourself be where the buck stops, it's easy to be sucked into thinking that there is some magical property, CEOness, Chairmanity, call it what you will, that creates success. This is really absolutely untrue. The hardest fact to swallow is that you can't make a business successful – that's impossible. There's a saying in the music industry, "nobody ever knows what's going to hit." This holds true for business of all kinds. Of course, snatching defeat from the jaws of victory is a common skill, shared by most of us. You can defintely, without doubt make a business fail. When they do, it can almost always be traced back to a decision that someone made, or failed to make, or failed to make soon enough. These bad decisions have a few etiologies: inability (or lack of will) to predict the future, sheer bad luck, cowardice, hubris, inexperience, inadequate information, or quite commonly due to the consequences of some kind of major personality disorder. CEO is a shitty job. You'd have to be insane to want to do it. (Join the dots. I'll wait.) This is just scene-setting. What I really wanted to write here was why I think that almost every social media site ultimately ended up a hellsite. First, a definition: by hellsite, I mean a platform that is dominated by trolls, bigots, and other sundry scum of the earth who mostly live to cause harm to their perceived 'lib' enemies. Here's a bit of Karl Popper for you, from The Open Society and its Enemies: It's seductive to altruistically want to create a neutral platform for discussion, where the weight of the arguments hold sway, where people can talk things out and rightness will prevail. Unfortunately, this is like a candle in the wind against a deliberate aggressor who cares nothing of truth, who wants to flood the zone with their own bullshit, dis- and misinformation, and who will not stop. These aggressors are typically on the extreme political right, though not absolutely exclusively. They know how to play the game. Tolerance of their views is expected, even when the views are something along the lines of: Them: I think that all people like you should be genocided. Me: Um, no, and bugger off. Them: [waily waily pearly clutchy] See? See how meeeeeeeean they are to poor us? They are always staining our shiny jackboots with their facebloodses! So taken literally, I wasn't tolerant of their views. Boo hoo. But they certainly were not tolerant of mine, and moreover they wanted to cause me considerable direct harm. Not the same. It's easier to figure this out, as quite a few philosophers have pointed out, by regarding this as a social contract, not a mutually agreed tolerance policy. If the deal is, don't cause harm to other people, rather than agreeing to be blindly tolerant, the outcome is far clearer. Where nobody is allowed to harm others, and this is enforceable, this intersects with free speech in a way that some find uncomfortable. But, even in its most unprotected form, a right to free speech is not a right to be absolved of the consequences of speech. Hellsites are what happens, therefore, when free speech is accepted without consequences, and algorithms optimize for engagement rather than social good. This happens regardless of politics – the early slide of several of the larger platforms were examples of this. But the intersection with fascist extreme right wing ideology wasn't pretty, and right off the cliff they went. It's fair to say that the owners of our favourite hellsites were absolutely down with it, but I'll admit that they were also subject to a bit of the old Mafioso 'nice company you got there, shame if the DOJ happened to it...' implied or literal threat. Don't bend the knee by the approved angle in degrees, or fail to adequately kiss the sphincter with the expected enthusiasm and expect a trip directly to antitrustsville, population you. Hellsites have levels of hell, accelerated by thirst for money and political power, and we haven't seen the worst of them yet.

Media Business

▲ 0

0 views

Taranis 8 months ago

Irish trans healthcare is astonishingly bad

I've lived in Ireland for less than two years at this point, but I transitioned over 30 years ago. Moving here, I talked to my GP about continuity of care (mostly just HRT), and was flatly refused. Though she was personally sympathetic, she told me that she'd been threatened by the National Gender Service not to prescribe, diagnose or refer for anything trans-related. All she could do was put me on their waiting list, which it seems is currently 10 to 13 years. A break in care of that duration would be extremely inappropriate (to say the least) – whilst being traumatic, it would have significant health risks associated with it. I'm not a shrinking violet. If someone puts up a roadblock, I'll go around it if possible, or demolish it if necessary. I arranged private access to HRT outside Ireland pretty quickly, so I was fine. Paying out of pocket, of course, but fine. My GP will at least do the necessary blood testing, which helps a lot practically. I've heard, let's say, stories , about the state of trans healthcare in Ireland via the NGS. TheJournal has a current news story running about this (link below). It reminds me of the bad old days of the Charing Cross clinic in the UK in the 1990s. Possibly even worse. I had a friend back then who was denied GRS because one of the psychs found out that she was in a lesbian relationship. I heard from another who had been held in the system for over 20 years waiting for surgery, seemingly because of a power trip and her not quite being 50s ideal woman enough. I never went there myself, I went elsewhere, but I remember being yelled at because I'd been working in a datacenter at a bank all day, crawling around behind racks fixing a network problem, and hadn't had time to change so I showed up in – shock horror – trousers. Oh, the humanity. Won't someone think of the children? I'll let you read the article yourselves, but suffice it to say that it very much feels like the NGS is more someone's little kingdom and power trip than it is an honest intention to actually provide healthcare. I'm OK, I have provision set up via providers elsewhere in Europe. But it hurts to hear stories from people who tried to do the right thing, who were brought up to trust their doctors, and who never had access to the information (or, for that matter, any idea that they might need access) necessary to go around the system. There is a myth that this is inherently dangerous, and that DIYing trans healthcare is analogous to taking illegal substances – this is absolute nonsense. The provider I'm currently working with for HRT follows WPATH guidelines and requires regular blood testing – ironically, this is something that no doctor ever did for me when I was going the conventional route. As regards surgery, Ireland has no providers anyway. NGS will technically pay, or will sign off so insurance (e.g., VHI, if your plan covers it) will pay. But good luck with that. They make it impossible to get a referral elsewhere within Ireland, which blocks access to insurance. So, if you want or need surgery, you need to expect to pay. The better news is that this can be arranged outside Ireland by working directly with the providers. Many work on an informed consent model, and it is also possible to get a necessary diagnosis via overseas providers too. This can be put in place in weeks to months, so there is honestly zero requirement or need for years of torture and neglect. Yes, torture. For a trans person, deciding to transition is a pivotal moment, and for many of us it can be a dangerous time. We are often shunned by family and friends, and being denied access to care can result in spiralling into a bad depression, which in all to many of us can end our lives. So – just to be clear – this is the kind of shit we were up against before all the Trump/JKR/Tory/Starmer bullshit hit. I've heard life as a trans person being described as life on hard mode – cis people honestly have no idea. There is a myth that we have some kind of magical power that means we get stuff for free and get better treatment, DEI approved into jobs we aren't qualified for. This is such absolute garbage that it's hard to know where to even start. I've been denied access to healthcare, housing and education many times. I've been denied jobs and fired when people found out I'm trans. I've been made technically homeless twice (though was able to narrowly avoid ending up on the streets). I've been SA'd, raped, and survived an attempted murder. Now we have to contend with losing what tenuous rights we struggled to achieve over decades, just because some orange fuckhead decided to spend 200 million dollars on vilifying us because, like all fascists, he needs an internal enemy against which to rile his dumbass masses. It's hard, and sometimes I wonder if I can go on, but then I bounce back and continue via sheer determination and no small amount of spite.

Healthcare Journalism Social Issues

▲ 0

0 views

Taranis 8 months ago

Trusting Copilot Is Like Letting a Dog Drive Your Car

I was working on a personal project over the weekend. Mostly the reason for this was not having used any of the current web front end technologies, or frankly any much since GWT was a thing way back when. I've been teaching myself Svelte and Sveltekit, because reasons. I did my usual thing (these days) of setting up a Linux VM with Postgres, all the Svelte stuff, VSCODE and the usual dev tools and had at it. I've used a few AI code autocomplete tools in the past with mixed results – some are pretty good, I'm particularly impressed with the one built into Google's Colab environment. I used Tabnine a year or so ago on another personal project and quite liked it. So I turned on Copilot when the option came up installing VSCODE. There's something not right about the UX. It's just somehow way too aggressive – I've been trying to figure it out, and I think it's just basically too fast. The Colab equivalent has a delay of a few hundred milliseconds before making a suggestion, which is plenty of time to type something or move the cursor or whatever. But Copilot is ridiculously fast, responding in probably less than 100ms. It also tends to suggest much larger blocks of code in one go. I think maybe one in 10 of Copilot's suggestions were good. The rest were hallucinated nonsense, or misunderstandings of my code. Colab does a better job, I'm not sure what the percentage is, but it's certainly far higher, and the code it generates is often spookily good, and I found a useful speedup relative to my usual coding rate, mostly because I'm not looking up API docs so much. But not so much Copilot. I'm getting to the turning it off point. That's OK, I don't really mind, it was an experiment anyway. However, it did do one thing that messed me up. It managed to insert a line of code that looked completely plausible, and did it so fast that I didn't notice. I think I was cutting and pasting to move some code around, and it basically just added an extra include. Ordinarily this would have made the compiler barf, but in this instance, the code was actually syntactically correct, but it blew something up deep in the innards of Svelte, causing a 500 error with no log messages being generated. This was an absolute pig to debug. I didn't have any checked in versions close enough to bisect on, so I was down to the good old fashioned comment everything out until it stops blowing up then add things back until it goes bang approach. This worked, and turned up the weird line of code that Copilot gifted me. This is the blighter responsible – the commented out import. I didn't write that line of code, and definitely didn't consent to Copilot 'helping'. I swear, Copilot is going to turn us all into paperclips and/or grey goo.

Web Development AI Programming

▲ 0

0 views

Taranis 8 months ago

Why I Stopped Trusting the Cloud (and Built My Own)

We are witnessing the decay of social media platforms in real-time. I've been around the net quite a while – I remember a time before HTTP, in the dark ages of FTP, gopher and Usenet . I think I was probably the first person at my university to download the source code for the very first ever web browser from CERN, installing it on a Sun workstation and looking at the handful of web sites that existed at the time. The net exploded from there. Even before HTTP, it was always social. I was first hooked on Usenet (I particularly miss alt.alien.vampire – OK, I suppose you had to be there), then moved on to Cix and then LiveJournal (aka LJ) around the turn of the millennium. I loved LJ – it gave me community and a place where I could express myself in a literary way. Then came Facebook, and frankly the start of the rot. You had to be on there because everyone was on there. I still used LJ, and preferred it, but held my nose and used Facebook because it was either that or abadon my ability to stay in touch with my real-life social network. Twitter started out great, back when it was a handy way to send group texts over SMS. Then, years later, LJ was bought out by a Russian company with links to the Russian state. Since this was incompatible with my job (and could have been iffy for my immigration status in the US at the time), I had to drop it. I was gutted. Facebook was OK, but never really got me writing the longer form pieces I used to love posting to LJ. Then it started to get weird about actually showing posts to people who followed you, unless you paid substantial money. That really killed it for me. I felt no real draw toward being bothered to write for a platform that didn't care and just saw me as a mark. As Twitter grew, it got less and less friendly, to the point of becoming quite dangerous. Then my main account got stolen, and even though I was able to recover it, my friends list had been deleted and replaced with random bots. So I kind of gave up with Twitter too. This was the situation for a few years. In the last few months, this slow decline became a cliff edge, with the platform owners visibly and publicly capitulating to a fascist regime. Even my email and other online services started to get questionable, because of the danger that my user data will be shared without my consent. Frankly even the risk that this comms could just be shut down would be potentially devastating. I already had some sensible mitigations in place, like having automatic continuous backups of my online file storage and documents, etc. But this only addresses the risk of losing access – the terrifying thing, however, is having my user data handed over to someone who might seek to perform trial by database query . A few months ago I decided to start taking back control of this situation. I have a bit more hardware around the place than most people. My day job involves serious infrastructure, running thousands of machines across multiple datacenters, so putting a solution together probably looked a bit different to me than it might to most people. I started by repurposing a number of machines I already had running in a rack here in Ireland – a small cluster of Intel NUCs, two Synology NASes, a big (though remarkably cheap, thanks to eBay bottom-feeding!) dual Xeon server with multiple GPUs and 1.5TB of RAM, and a repurposed gaming box. A bit of a motley crew, but after some experimentation I landed on a solution based on the Xen hypervisor, specifically the open-source xcp-ng variant. I set everything up as a pair of Xen clusters, one with the NUCs and the GPU server in it, and a second with the single gaming PC. It would have made perfect sense (and would have been far easier) to build a single cluster, but I specifically wanted to be able to test making multiple clusters talk to each other seamlessly. Networking was interesting. The NUCs are all a couple of years old now, 6 core/12 thread 10th gen Intel CPUs with 32GB of RAM and two 2TB SSDs each. They only have 1GB ethernet as standard, so I am using external Thunderbolt to 10GbE adapters. The for-real server has 10GbE on the motherboard, so I added a 10GbE card to the gaming PC (a 13th gen Intel with 128GB of RAM). A bit of faffing around later, I had three VLANs set up, such that all the machines in both clusters were visible on a management network, and each cluster had its own separate local network. All of these are on different subnets, so comms can be routed with no need to do bridging (this is critically important for scalability, though admittedly bridging would have worked fine at this scale). I created OPNsense firewall VMs in each cluster, each set up with dual virtual network ports so they could see both the management network (and therefore also the internet) and their internal networks. I set them up to do DHCP and NAT for their internal networks, and set up a Wireguard tunnel between the clusters, which made it possible for VMs running on machines on both clusters to both see the internet and to be able to directly talk to each other. This was my first proof-of-concept. It took a while to get right. I think I reinstalled xcp-ng on the NUCs four or five times, and took a few attempts to get the firewall configs right, but in the end it worked and was rock solid. At this point, I only had a relatively slow internet connection, and by policy wasn't interested in using a commercial reverse proxy service like Cloudflare (too much US exposure, and some shocking business practices ). On a tip from a friend, I decided to rent a bare metal machine from Hetzner in Germany. Basically, this is a whole physical server sitting in a rack in a datacenter that I have complete control over, and that has an unmetered internet connection. As is my preference, I installed xcp-ng on it, and a series of VMs including an OPNsense firewall, nginx reverse proxy, a primary DNS server and an email server. It took a while, a few weeks of tweaking in evenings and at weekends, to get everything going, but I ended up with the machine set up as a single-machine cluster with a software-only internal network and my usual NAT and Wireguard tricks so everything can see everything. Deep breath... It's a whole lot of overkill, but what it effectively means is that I can now spin up VMs anywhere on any of the clusters that can serve the web just as if they were sitting in the DC in Germany. Running machines in my rack here is much cheaper than in a DC, particularly for the more powerful machines with lots of disc and RAM, and maintaining them involves a 15 second walk from my office to my machine room. It also makes it quite a bit harder for someone to get physical access to machines containing user data, because it just simply isn't where it appears to be from the outside. Running this Ghost-based blog server uses an absolutely tiny amount of the capabilities of the infrastructure. I have, let's say, plans. One thing that I absolutely love about VM-based virtualized infrastructure is just how much easier it makes backing up and restoring systems. With xcp-ng, there isn't really much to back up on the base system (though I do it anyway). The VMs are where it's at, and the Xen Orchestrator management front end makes it really straightforward to do disk-level backups automatically across the entire network. These get written to a primary NAS, which backs up these backups to a second volume locally and to a second physically separate NAS which keeps deeper history. Restoring a VM is ridiculously easy, you just restore from a single large backup file and boot the VM and you're done. I keep both complete image and incremental backups, so I can also pull back individual files if I need to. This was a LOT of work, and I totally get it that this kind of solution is probably out of reach for most people reading this. There are simpler and cheaper solutions, but like I sad, I have plans that don't end with publishing this blog. That said, doing what you need to do to break your reliance upon US-based/controlled platforms makes a huge amount of sense at this point in time. Regardless of your political views, Trump's unpredictability is a significant liability, both at a personal and business level for everyone, both inside and outside the US. Taking back the control of your user data and housing it somewhere that has data protection laws with real teeth (GDPR) , makes a huge amount of sense. If you happen to be a member of a minority that the Trump regime is specifically targeting for harm (like me), this is doubly important. The EU is taking digital sovreignty seriously. I feel much safer living in Ireland, which is an EU member state, and housing my infrastructure here and in Germany. Over the next weeks and months I hope to post more about this, both from the political and the technical points of view. My intention is to not (just) rant about this on the internet, but to hopefully actually do things to help in a practical way. If you have questions or comments, feel free to contribute, even if you just want to remind me that I'm crazy to overkill this to the extent that I am. But I knew that. 😸

Cloud

▲ 0

0 views

Taranis 8 months ago

Petition: Legally enshrine the right of adults to physically transition using NHS services

It has been possible for UK citizens (and residents, I think, too) to access gender-affirming care via the National Health Service for many years. The provision is slow, hasn't always been great (particularly in the dark and distant past – I have stories), but has always been there. Given that the UK government seems intent on sucking up to the Orange Monster, it is far from impossible that this or a future government might try to restrict access. This petiton aims to redress this by enshrining this right in UK law. If you happen to be a UK resident or a UK citizen living overseas, please consider signing this petition.

Media Culture Business

▲ 0

0 views

Taranis 8 months ago

From Dream Job to Exile: Why I Left the US for Ireland as a Trans Person

It's a hard thing to admit to myself, let alone others, but I'm effectively a refugee. I'm a UK citizen, but I lived in the US for more than a decade before leaving during Trump's first term in office. I went to the US, like many, for a job. The job. The absolute dream, offer you can't refuse, job. I arrived during George W's time. I wasn't particularly keen on him or his policies, but nothing he did either made me afraid or ashamed. Then came Obama, all eight years, which was mostly fine (though there were moments – something for another post sometime whenever). After a hair under a decade, I left the dream job and went to the good job. The one that turned my finances around for the first time in my life, with some resemblance toward work/life balance. A couple of months later, Trump was elected. I knew immediately that it would be bad. Possibly very bad, and there was a good chance I'd have to leave the US because of it. By this time, I was (same-sex) married to a US citizen and was a green card holder eligible to apply for citizenship myself. I lasted about another year and a half. Checking the news two or three times a day to see whether the final red line had been crossed, and it was time right now to head to the border, got to me eventually. I was done. My employer was multinational, so I was able to apply for an internal transfer to Europe. I got it, and my spouse and I left about two or three months later. We spent five years in mainland Europe, then decided to move to Ireland. Easier language issues, some economic advantages, long term stability, the chance to buy somewhere we could afford, a plausible off-ramp to retirement in a decade or so, all played into the decision. For me, the move was straightforward. As a UK citizen, I have right of residence here, so all I needed to do was show up and file some paperwork. For my spouse, it was far more difficult. The paperwork was significantly onerous, running to many hundreds of pages of forms, copies of documents, etc. They even requested our email and chat logs going back to the beginning of our relationship. It took nearly two years and an intervention from our immigration lawyers, but we're both now safely here. My spouse ended up living in Florida up until just before the election, so we were very grateful to be able to get them out before that (they are trans masc nonbinary, and potentially at risk). I transitioned in the UK about 30 years ago. It was hard then, and honestly didn't improve hugely before I left in the mid noughties. I survived an attempted murder there, transphobically motivated, in the late 90s. Things seemed to improve somewhat while I was out of the country, but due to the terf inundation of all of the major parties, and particularly the Tories using anti-trans rhetoric as an election wedge issue, things have gone back to as bad as they were when I first transitioned. Or possibly worse, given the recent idiotic supreme court decision. It's now pretty clear that I'm not welcome in either my chosen country (US) or my birth country (UK). My world has become a lot smaller, and I feel in far more danger than I have in decades. Though both myself and my spouse are in Ireland legally, via conventional immigration processes and not via a (legal!) asylum claim, we are nevertheless effectively refugees. I am not looking for sympathy or pity. I'm in a far better position than many of the trans people that I know. If anything, this is a call-to-action. Don't obey in advance. If you are a cis person reading this, actively disobey. Don't collaborate, don't capitulate. Check on the trans people you know and love, and offer them help as you can. Make plans. Things can , and quite possibly will, get worse from here. I'm not just suggesting in doing online keyboard warrior duty (though that isn't nothing!), but rather figuring out how you can directly help, personally, one on one if you have to. There is a good chance that NGOs and charities will be targeted and shut down if they have any kind of public profile, so run silent, run deep. I have my own plans I'm working toward. This blog is a (small) part of that. I won't announce things before they are ready, for obvious reasons, and I suggest that you are similarly careful with what you say, particularly online. For example, this blog is hosted outside the US, in jurisdictions with significant privacy protections. Since I post openly, it will be easy enough to track what I say (so, please be careful to not disclose anything in comments that you might not want to be seen), but it will be extremely difficult to stop me from speaking.

Career Culture

▲ 0

0 views