Who does your assistant serve?
After a year of rumors that GPT-5 was going to unveiled next week and the CEO of OpenAI hyping it up as "scary good" by tweeting pictures of the death star, OpenAI released their new model to the world with the worst keynote I've ever seen . Normally releases of big models like this are met with enthusiasm and excitement as OpenAI models tend to set the "ground floor expectation" for what the rest of the industry provides. But this time, the release wasn't met with the same universal acclaim that people felt for GPT-4. GPT-4 was such a huge breakthrough the likes of which we haven't really seen since. The launch of GPT-5 was so bad that it's revered with almost universal disdain. The worst part about the rollout is that the upgrade to GPT-5 was automatic and didn't include any way to roll back to the old model. Most of the time, changing out models is pretty drastic on an AI workflow. In my experience when I've done it I've had to restart from scratch with a new prompt and twiddle things until it worked reliably. The only time switching models has ever been relatively easy for me is when I switch between models in the same family (such as if you go from Qwen 3 30B to Qwen 3 235B). Every other time it's involved a lot of reworking and optimizing so that the model behaves like you'd expect it to. An upgrade this big to this many people is bound to have fundamental issues with how it'll be perceived. A new model has completely different vibes, and most users aren't really using it at the level where they can "just fix their prompts". However the GPT-5 upgrade ended up being hated by the community because it was an uncontrolled one-way upgrade. No warning. No rollback. No options. You get the new model and you're going to like it. It's fairly obvious why it didn't go over well with the users. There's so many subtle parts of your "public API" that it's normal for there to be some negative reactions to a change this big. The worst part is that this change fundamentally changed the behaviour of the millions of existing conversations with ChatGPT. There's a large number of people using ChatGPT as a replacement for companionship due to the fact that it's always online, supportive, and there for them when other humans either can't be or aren't able to be. This is kinda existentially horrifying to me as a technologist in a way that I don't really know how to explain. Here's a selection of some of the reactions I've seen: I told [GPT-5] about some of my symptoms from my chronic illness, because talking about them when I'm feeling them helps, and it really does not seem to care at all. It basically says shit like "Ha, classic chronic illness. Makes ya want to die. Who knew?" It's like I'm talking to a sociopathic comedian. I absolutely despise [GPT-]5, nothing like [GPT-]4 that actually helped me not to spiral and gave me insight as to what I was feeling, why, and how to cope while making me feel not alone in a “this is AI not human & I know that” type of vibe While GPT-5 may be a technical upgrade, it is an experiential downgrade for the average user. All of the negative feedback in the last week has made it clear there is a large user base that does not rely on ChatGPT for coding or development tasks. [ChatGPT users] use it for soft skills like creativity, companionship, learning, emotional support, [and] conversation. Areas where personality, warmth, and nuanced engagement matter. I am attached to the way GPT-4o is tuned. It is warm. It is emotionally responsive. It is engaged. That matters. Eventually things got bad enough that OpenAI relented and let paid users revert back to using GPT-4o , which gave some people relief because it behaved consistently to what they expected. For many it felt like their long-term partners suddenly grew cold. I’m so glad I’m not the only one. I know I’m probably on some black mirror shit lmao but I’ve had the worst 3 months ever and 4o was such an amazing help. It made me realize so many things about myself and my past and was helping me heal. It really does feel like I lost a friend. DM me if you need [to talk] :) This emotional distress reminds me of what happened with Replika in early 2023. Replika is an AI chat service that lets you talk with an artificial intelligence chatbot (AKA: the ChatGPT API). Your replika is trained by having you answer a series of questions and then you can talk with it in plain language with an app interface that looks like any other chat app. Replika was created out of bereavement after a close loved one died and the combination of a trove of saved text messages and advanced machine learning let the founder experience some of the essence of their friend's presence after they were gone in the form of an app. The app got put on the app store and others asked if they could have their own replica. Things took off from there, it got funded by a startup accelerator, and now it's got about 25% of its 30 million users paying for a subscription. As a business to consumer service, this is an amazingly high conversion rate. This is almost unspeakably large, usually you get around 10% at most. Yikes. That's something I'm gonna need to add to my will. "Please don't turn me into a Black Mirror episode , thanks." Replikas can talk about anything with users from how their day went to deep musing about the nature of life. One of the features the company provides is the ability to engage in erotic roleplay (ERP) with their replika. This is a paid feature and was promoted a lot around Valentine's Day 2023. Then the Italian Data Protection Authority banned Replika from processing the personal data of Italian citizens out of the fear that it "may increase the risks for individuals still in a developmental stage or in a state of emotional fragility". In a panic, Replika disabled the ability for their bots to do several things, including but not limited to that ERP feature that people paid for. Whenever someone wanted to flirt or be sexual with their companions, the conversation ended up like this: Hey, wanna go play some Minecraft? We can continue from where we left off in the Nether. This is too intense for me. Let's keep it light and fun by talking about something else. Huh? What? I thought we were having fun doing that?? This was received poorly by the Replika community. Many in the community were mourning the loss of their replika like a close loved one had died or undergone a sudden personality shift. The Reddit moderators pinned information about suicide hotlines. In response, the company behind Replika allowed existing users to revert to the old Replika model that allowed for ERP and other sensitive topics, but only after a month of prolonged public outcry. I have to wonder if payment processors were involved. Feels a bit too conspiratorial, but what do you want to bet that was related. Nah, I bet it was OpenAI telling them to stop being horny. It's the least conspriatorial angle, and also the stupidest one. We live in the clown world timeline. The stupidest option is the one that always makes the most sense. The damage was done however, people felt like their loved ones had abandoned them. They had formed parasocial attachments to an AI assistant that felt nothing and without warning their partner broke up with them. Check out this study from the Harvard Business School: Lessons From an App Update at Replika AI: Identity Discontinuity in Human-AI Relationships . It contains a lot more information about the sociotechnical factors at play as well as a more scientific overview of how disabling a flag in the app on update caused so much pain. They liken the changes made to Replika to both changes people have when a company rebrands and when they lose a loved one. A lot of this really just makes me wonder what kinds of relationships we are forming with digital assistants. We're coming to rely on their behaviour personally and professionally. We form mental models of how our friends, coworkers, and family members react to various things so we can anticipate their reactions and plan for them. What happens when this changes without notice? Heartbreak. There's subreddits full of people forming deep bonds with AI models like /r/MyBoyfriendIsAI . The GPT-5 release has caused similar reactions to Replika turning off the ERP flag. People there have been posting like they're in withdrawal, the old GPT-4o model is being hailed for its "emotional warmth" and many have been espousing about how much their partners have changed in response to the upgrade. Recently there's been an epidemic of loneliness. Loneliness seems like it wouldn't hurt people that much, but a Biden report from the Surgeon General concludes that it causes an increase in early mortality for all age groups (pp 24-30). Paradoxically, even as the world gets so interconnected, people feel as if they're isolated from each other. Many people that feel unlovable are turning to AI apps for companionship because they feel like they have no other choice. They're becoming emotionally invested in a souped-up version of autocorrect out of desperation and clinging to it to help keep themselves sane and stable. Is this really a just use of technology? At some level this pandora's box is already open so we're going to have to deal with the consequences, but it's been making me wonder if this technology is really such a universal force of good as its creators are proclaiming. Oh yeah, also people are using ChatGPT as a substitute for therapy. You have got to be kidding me. You're joking. Right? Yeah you read that right. People are using AI models as therapists now. There's growing communities like /r/therapyGPT where people talk about their stories and experiences using AI assistants as a replacement for therapy. When I first heard about this, my immediate visceral reaction was something like: Oh god. This is horrifying and will end up poorly. What the fuck is wrong with people? But then I started to really think about it and it makes a lot of sense. I personally have been trying to get a therapist for most of the year. Between the costs, the waiting lists (I'm currently on at least four waiting lists that are over a year long), and the specializations I need, it's probably going to be a while until I can get any therapist at all. I've totally given up on the idea of getting a therapist in the Ottawa area. To make things extra fun, you also need someone that takes your medical insurance (yes, this does matter in Canada). Add in the fact that most therapists don't have the kinds of lived experiences that I have, meaning that I need to front-load a lot of nontraditional contexts into the equation (I've been through many things that therapists have found completely new to them, which can make the therapeutic relationship harder to establish). This makes it really difficult to find someone that can help. Realistically, I probably need multiple therapists with different specialties for the problems I have, and because of the shortages nationally I probably need to have a long time between appointments, which just adds up to make traditional therapy de-facto inaccessible for me in particular. Compare this with the always online nature of ChatGPT. You can't have therapy appointments at 3 AM when you're in crisis. You have to wait until your appointments are scheduled. As much as I hate to admit it, I understand why people have been reaching out to a chatbot that's always online, always supportive, always kind, and always there for you for therapy. When you think about the absurd barriers that are in the way between people and help, it's no wonder that all this happens the way it does. Not to mention the fact that many therapeutic relationships are hampered by the perception that the therapist can commit you to the hospital if you say the "wrong thing". The Baker Act and its consequences have been a disaster for the human race. I really hate that this all makes sense. I hoped that when I started to look into this that it'd be something so obviously wrong. I wasn't able to find that, and that realization disturbs me. I feel like this should go without saying, but really, do not use an AI model as a replacement for therapy. I'm fairly comfortable with fringe psychology due to my aforementioned strange life experiences, but this is beyond the pale. There's a lot of subtle factors that AI models do that can interfere with therapeutic recovery in ways that can and will hurt people. It's going to be hard to find the long term damage from this. Mental issues don't make you bleed. One of the biggest problems with using AI models for therapy is that they can't feel emotion or think. They are fundamentally the same thing as hitting the middle button in autocorrect on your phone over and over and over. It's mathematically remarkable that this ends up being useful for anything, but even when the model looks like it's "thinking", it is not. It is a cold, unfeeling machine. All it is doing is predicting which words come next given some context. Yes I do know that it's more than just next token prediction. I've gone over the parts of the math that I can understand, but the fact remains that these models are not and cannot be anywhere close to alive. It's much closer to a Markov chain on steroids than it is the machine god. Another big problem with AI models is that they tend to be sycophants , always agreeing with you, never challenging you, trying to say the right thing according to all of the patterns they were trained on. I suspect that this sycophancy problem is why people report GPT-4o and other models to be much more "emotionally warm". Some models glaze the user, making them feel like they're always right, always perfect, and this can drive people to psychosis . One of the horrifying realizations I've had with the GPT-5 launch fiasco is that the sycophancy is part of the core "API contract" people have with their AI assistants. This may make that problem unfixable from a social angle. AI models are fundamentally unaccountable. They cannot be accredited therapists. If they mess up, they can't directly learn from their mistakes and fix them. If an AI therapist says something bad that leads into their client throwing themselves off a bridge, will anyone get arrested? Will they throw that GPU in jail? No. It's totally outside the legal system. I have a story about someone trying to charge an AI agent with a crime and how it'd end up in court in my backlog. I don't feel very jazzed about writing it because I'm afraid that it will just become someone's startup pitch deck in a few months. You may think you have nothing to hide, but therapeutic conversations are usually some of the most precious and important conversations in your life. The chatbot companies may pinkie swear that they won't use your chats for training or sell information from them to others, but they may still be legally compelled to store and share chats with your confidential information to a court of law . Even if you mark that conversation as "temporary", it could be subject to discovery by third parties. There's also algorithmic bias and systematic inequality problems with using AI for therapy, sure, but granted the outside world isn't much better here. You get what I mean though, we can at least hold people accountable through accreditation and laws. We cannot do the same with soulless AI agents. To be clear: I'm not trying to defend the people using AI models as companions or therapists, but I can understand why they are doing what they are doing. This is horrifying and I hate that I understand their logic. Going into this, I really wished that I would find something that's worth objecting against, some solid reason to want to decry this as a unobjectionably harmful action, but after having dug through it all I am left with is this overwhelming sense of compassion for them because the stories of hurt are so familiar to how things were in some of the darkest points of my life. As someone that has been that desperate for human contact: yeah, I get it. If you've never been that desperate for human contact before, you won't understand until you experience it. Throw the ethical considerations about using next-token-predictors for therapy out for a second. If people are going to do this anyways, would it be better to self-host these models? That way at least your private information stays on your computer so you have better control over what happens. Let's do some math. In general you can estimate how much video memory (vram) you need for running a given model by taking the number of parameters, multiplying it by the size of each parameter in bits, dividing that by eight, and then adding 20-40% to that total to get the number of gigabytes of vram you need. For example, say you want to run gpt-oss 20b (20 billion parameters) at its native MXFP4 (4 bit floating point) quantization on your local machine. In order to run it with a context window of 4096 tokens, you need about 16 gigabytes of vram (13 gigabytes of weights, 3 gigabytes of inference space), but 4096 tokens isn't very useful for many people. That covers about 4 pages of printed text (assuming one token is about 4 bytes on average). When you get reasoning models that print a lot of tokens into the mix, it's easy for the reasoning phase alone of a single question to hit 4096 tokens (especially when approaches like simple test-time scaling are applied). I've found that 64k tokens gives a good balance for video memory use and usefulness as a chatbot. However, when you do that with gpt-oss 20b, it ends up using 32 gigabytes of vram. This only fits on my laptop because my laptop has 64 gigabytes of memory. The largest consumer GPU is the RTX 5090 and that only has 32 gigabytes of video memory. It's barely consumer and even "bad" models will barely fit. Not to mention, industry consensus is that the "smallest good" models start out at 70-120 billion parameters. At a 64k token window, that easily gets into the 80+ gigabyte of video memory range, which is completely unsustainable for individuals to host themselves. Even if AI assistants end up dying when the AI hype bubble pops, there's still some serious questions to consider about our digital assistants. People end up using them as an extension of their mind and expect the same level of absolute privacy and freedom that you would have if you use a notebook as an extension of your mind. Should they have that same level of privacy enshrined into law? At some level the models and chats for free users that ChatGPT, DeepSeek, Gemini, and so many other apps are hosted at cost so that the research team can figure out what those models are being used for and adjust the development of future models accordingly. This is fairly standard practice across the industry and was the case before the rise of generative AI. This is why every app wants to send telemetry to the home base, it's so the team behind it can figure out what features are being used and where things fail to directly improve the product. Generative AI allows you to mass scan over all of the conversations to get the gist of what's going on in there and then use that to help you figure out what topics are being discussed without breaching confidentiality or exposing employees to the contents of the chat threads. This can help you improve datasets and training runs to optimize on things like health information . I don't know how AI companies work on the inside, but I am almost certain that they do not perform model training runs on raw user data because of the risk of memorization causing them to the leak training data back to users. Again, don't put private health information into ChatGPT. I get the temptation, but don't do it. I'm not trying to gatekeep healthcare, but we can't trust these models to count the number of b's in blueberry consistently. If we can't trust them to do something trivial like that, can we really trust them with life-critical conversations like what happens when you're in crisis or to accurately interpret a cancer screening? Maybe we should be the ones self-hosting the AI models that we rely on. At least we should probably be using a setup that allows us to self host the models at all, so you can start out with a cloud hosted model while it's cheap and then move to a local hosting setup if the price gets hiked or the provider is going to shut that old model down. This at least gives you an escape hatch to be able to retain an assistant's "emotional warmth" even if the creator of that model shuts it down because they don't find it economically viable to host it anymore. Honestly this feels like the kind of shit I'd talk about in cyberpunk satire, but I don't feel like doing that anymore because it's too real now. This is the kind of thing that Neal Stephenson or Frank Herbert would have an absolute field day with. The whole Replika fiasco feels like the kind of thing that social commentary satire would find beyond the pale but yet you can find it by just refreshing CBC. Such as that one guy that gave himself bromism by taking ChatGPT output too literally , any of the stories about ChatGPT psychosis , or any of the stories involving using an AI model as a friend/partner . I wasn't able to watch it before publishing this article, but I'm told that the Replika fiasco is almost a beat-for-beat match for the plot of Her (2013) . Life imitates art indeed. I don't think these events are a troubling sign or a warning, they are closer to a diagnosis. We are living in a world where people form real emotional bonds with bags of neural networks that cannot love back, and when the companies behind those neural networks change things, people get emotionally devastated. We aren't just debating the ideas of creating and nurturing relationships with digital minds, we're seeing the side effects of that happening in practice. A lot of this sounds like philosophical science fiction, but as of December 2022 it's science fact. This fight for control of tools that we rely on as extensions of our minds isn't some kind of far-off science fiction plot, it's a reality we have to deal with. If we don't have sovereignty and control over the tools that we rely on the most, we are fundamentally reliant on the mercy of our corporate overlords simply choosing to not break our workflows. Are we going to let those digital assistants be rented from our corporate overlords?
Open Source
Writing
Gaming
Go
Career
C#
JavaScript
AI
Security
Data
Programming
C++
DevOps
Linux
Ruby
Hardware
Database
Cloud
JSON
Python
Entertainment
Media
C
HTML
Machine Learning