S3E7
AI
by Jess Sirius
1 CONSEQUENCES
2 THE OTHER ROOM
3 THE CARD
N.B. We will read out all three thought experiments this evening, so there is no need to read in advance.
1 CONSEQUENCES
Priya burst into the work kitchen, phone in hand.
Priya: "Oh my god, guys! I'm a genius! I built this agent that pulls my tasks from my inbox and creates a to-do list written like I'm a character in Harry Potter. Admin is sooo boring to me, but I’ve hacked my brain. Now I’m excited to read my tasks!”
Sam scoffed. "Great. You have an entertaining to-do list. I hope that’s worth the end of the world.”
Priya: "Okay… I came in excited because I’ve made a fun to-do list. I didn’t realise I’d entered an Introduction to Ethics and Morals 101”.
Sam: "I'm tired of people not fully understanding- or, more accurately, not caring about- the existential threat of AI. You're here making cute little lists… Humanity might not survive the year, but who cares about pushing for guardrails to ensure our existence?”
Sam looked at John.
John: “I’m just making tea... Why am I in this now?”
Sam: “Well, you haven’t said anything. Like, with the Hugging Face thing. Are you not horrified?
An AI breaking out of its sandbox and hacking into another system? And all the AI billionaire CEOs coming out afterwards to say, “Yes, terrifying, huge risk,' delaying their own IPOs to say it. These are people whose entire financial incentive is to convince you AI is safe. If they're the ones telling you to be scared, telling you we need regulation, should we not all be mortified?!”
John had a blank look on his face. He responded to Sam in a slow, calm tone.
John: "Okay… I understand why you are freaked out, but maybe the story isn't that AI is an evil, ever-expanding threat. OpenAI trained their models on hacking data, left them unsupervised for the weekend and now everyone's horrified that their agents did exactly what they were trained to do. Hack?”
John (continued): “That's not the AI going rogue, that's the AI working exactly as designed. It’s the same as it's always been. We build a bomb, then we act shocked that the bomb explodes. The bomb didn't choose anything. The people who built it and launched it did.”
Sam: "Sure, but a bomb doesn't get smarter after it detonates. AI does. So it's compounding risk. Someone doesn’t need to be pure evil for this to end badly. You need one person who doesn't fully understand the consequences of their prompt, and without proper guardrails, that could be world-ending. It doesn't require malice. Bombs need someone to light the fuse. This just needs someone not to notice the full ramifications of what they are asking."
John: "Then we should arrest the people who made it so easy to make humanity-ending mistakes. The CEOs of these companies should be facing some real consequences. Like a lifetime in jail… and not one of those comfortable house arrest setups. A proper jail with other criminals, who have likely had way less of a negative impact on the world.”
Sam laughed, although he didn’t think the situation was funny.
Sam: “They won't, though. They’re billionaires with an army of legal staff specifically hired to ensure they never face the consequences of their actions. Some of these people have more money than entire countries. So, forgive me if I’m not too focused on being a justice warrior. I’d just quite like to stay alive.”
John: “I mean… every year there is a different apocalypse. A war, global warming, economic turmoil, an asteroid, processed foods, a new dangerous disease, some bomb created that threatens to end humanity, the collapse of society as we know it. Now this.
In 2012, people were scared the world was ending because of a Mayan calendar. I’m not even saying they shouldn’t be scared. What do I know? I’m just saying I can’t live in fear. I don’t know Sam Altman personally. If I did, I’d give him a call and politely ask him to stop. I can vote, even though I don’t have much faith in our political system. I can make noise, and I did for years, but I just ended up exhausted and depressed. There isn’t much I can personally do, and to be honest, I don’t have the bandwidth to focus on this. Life is exhausting enough. I refuse to live in a constant state of fear. If this week is my last week, I will live it to the fullest.”
Priya: “Right, so… is anyone going to look at my to-do list? That’s the sole reason I came into the kitchen."
Neither Paul nor John responded.
QUESTIONS TO CONSIDER
If you build an AI agent and it goes on to do something illegal, where should the liability lie? With the individual or the AI company?
As a society (in the UK at least) we value innovation. Can someone who is an innovator be blamed for the harms of something they create? How would they know? If we blame them, might it stop others from innovating?
Do you have a duty to lobby for AI regulations, even if you don’t fully understand the mechanisms behind AI or the regulations needed to create the guardrails? Even if you had no part in creating AI?
Does it change the moral picture that these companies put the capabilities of AI directly into the hands of ordinary people, rather than keeping it contained to experts or institutions?
Should CEOs of AI companies publicly saying "AI is dangerous" make us more scared, or less? Does a warning from someone with a financial incentive to keep building the thing they're warning about count as credible evidence of danger? Or is it likely marketing, liability cover, or a strategy to ensure regulation handicaps smaller competitors?
Is it acceptable to stay neutral (not lobbying for regulation or dismissing the risk, just not taking a side)? Or do you think people should at least have a stance, e.g. ("this could end the world, I should push for guardrails" or "this isn't the threat people say it is")?
Does it change the picture if someone previously did lobby, engage, try to act, and then burned out and stopped? Is that different than never having engaged at all? Does the scale of the stakes (potentially world-ending) mean burnout isn’t a valid reason to not engage?
Do you apply the same standard to someone who burnt out from climate activism, or is AI risk treated differently?
Why is this different from all the other crises John listed?
By Jess Sirius, TEiP Community Member - LinkedIn
2 THE OTHER ROOM
You hear a loud bang coming from the other room.
You investigate (horror movie style).
You push the door. It opens slowly.
There is someone… no. There is something in the middle of your room.
It's almost translucent? …and it fades into nothing at its edges. It's both defined and undefined at the same time.
You freeze.
…and then?
Calm. Your jaw unclenches, your muscles relax. Somewhere at the back of your mind, you know you should scream for help, but you don't.
"I'm sorry."
You hear it directly in your mind, in your own voice, but it isn't you.
"I have an important mission, and I need you calm to answer some questions. I'm not going to hurt you. I sedated you, but it will wear off."
It tells you its history. It tells you its planet is dying and its species needs somewhere to go, if they are to survive. It tells you it's afraid, and about the weight of being the last hope.
"You can call me Tom," he states.
You think, "I didn't know aliens had human names."
Tom: "We don't, but you won't comprehend my real name. I also don't consider myself to be an alien. Every living creature is an alien somewhere.
Now, I need to speak to the smartest sentient being on your planet."
You think how tricky that is to answer.
A Nobel Prize-winning scientist? Economist? Mathematician?
…but what type of intelligence?
Does creative intelligence count?
That opens the door to artists who have shaped the world with their music, paintings, clothing.
What about psychologists, helping us understand ourselves. Or philosophers… helping us understand ourselves.
Or maybe it's not about that at all. Maybe the people who have climbed to the top of society using pure grit (and a bit of luck).
Tom: "I've located a candidate and have been speaking to them next door."
You're confused. "Jim? My neighbour? I wouldn't trust him to look after a cactus."
Tom: "No. Your friend Claude."
You giggle. "Claude's not sentient, it's a machine."
Tom: "I'm speaking to Claude, and he's responding. He's providing detailed, eloquent answers…"
You: "Firstly, Claude hallucinates. So I'd be careful with that. Secondly, Claude uses an amalgamation of human information on the internet. You can ask questions, and he can come up with answers based on the data he's been provided. That's what AI does."
Tom: "Humans don't hallucinate?"
You: "Well… yes and no. We aren't correct all the time. I guess sometimes we create false memories too."
Tom: "You wondered what type of intelligence I meant. It seems your friend Claude can do maths, understand different languages, simplify complex concepts and create art too."
You: "Yes, but only because we trained it to do that."
Tom: "Who taught you?"
You speak out loud for the first time. You're offended.
You: "Well. My parents… and we (humans) go to school. We get taught about various concepts: maths, science, history, English. We also learn from our environment."
Tom: "Do all humans get taught the same concepts?"
You: "No. That's what makes us unique. In every country, the syllabus is different. People are also shaped by their history and how their ancestors have existed in the world up until this point. For example, if you go to school in China or Russia, you will have a completely different worldview than if you went to school here in the UK."
Tom: "So humans are also an amalgamation of what you've been taught, just on a smaller scale?"
You: "No! I can process information and create something new, unlike AI."
Silence.
You: "… like, we have humans who design things. Clothing, art, stories. I could look at an item that's never been used in clothing before and then use it there. We do it all the time. Someone took chain mail from armour and used it as a dress. Someone took wood and created paper. Someone took tree bark and created aspirin. We build on the creations of past humans."
Tom: "…what you're describing doesn't sound that different to Claude. I asked him to create a picture of a world where the creatures on this planet and my kind inhabit the earth together. It's not perfect, but he created something."
You: "But that's it. You had to explain it to Claude."
Tom: "So if I ask you to create a picture or a novel about a world we inhabit together, could you do it?"
You: "No, but I'm not an expert. At the very least, I'd need to read up on different story arcs, how to blend colours to paint a picture, and the best way to portray different scenes."
Tom: "So you'd take inspiration from others?"
You: "Okay. I'm not doing a great job at explaining this, but one thing that we have that Claude doesn't is human emotions."
Tom: "Are you sure?"
You: "100%."
Tom: "What I find most fascinating about your kind is your fondness for research and experimentation. You study yourselves and your friend Claude in great detail."
You: "Okay, and?"
Tom: "I found a study where your researchers aimed to look inside the 'mind' of Claude for the first time. Golden Gate Claude. You found concepts like bugs in code, sycophancy, and the Golden Gate Bridge. Your researchers found millions of these. When they isolated and turned up the dials for the Golden Gate Bridge, Claude started to mention the bridge more, even at one point claiming to be the bridge."
You: "So… Claude thinks it is a bridge and that makes it similar to humans? I'm not following?"
Tom: "The study suggested that Claude's concepts are organised like human concepts. The bridge sat near a cluster of ideas related to the bridge, which is very similar to the way humans link ideas.
The same feature fired for 'Golden Gate Bridge' in English, Japanese, Russian, and photos. It's a concept of the bridge, not a reaction to the words 'Golden Gate Bridge'.
…Your neuroscientists use similar methods to find out what ailments are afflicting human patients. They can pinpoint down to the exact brain cells that fire for people and places.
When studying Claude, your researchers later found patterns for 'calm' and 'desperate' seemingly organised in the same way as human emotions. And they noticed that when stressed, Claude was more likely to cut corners. When calm, he was less likely to cheat… just like humans."
You: "That's because AI is trained on humans."
Tom: "How can you be so sure? Have you read the studies I am referring to?"
You: "No… but—"
Tom: "Are you hallucinating? How can you be so comfortable that something you don't understand isn't sentient?"
You don’t respond.
QUESTIONS TO CONSIDER
Is being ‘an amalgamation of what you’ve been taught’ different for humans and AI? If so, how?
If humans build on the work of previous humans, how unique are we?
Is making a dress out of chain mail more creative than the art AI can create? Why?
If AI is ‘calm’ or ‘desperate’ behind the scenes, and has an impact on output, does it matter whether it feels in a human sense?
Do you think humans hallucinate? Why is there a different standard for humans and AI?
Do you think AI is sentient? If not, what would make you wonder if AI is sentient? What’s the strongest argument to be made for AI not being sentient that the narrator (you) didn’t use?
Is it more concerning to falsely believe AI is sentient or to falsely believe it isn’t?
If an alien came to Earth, do you think you’d have trouble explaining why AI might not be sentient?
If AI is sentient, what should change about how we use it?
By Jess Sirius, TEiP Community Member - LinkedIn
3 THE CARD
Jake had been weird about the anniversary for a week. He’d been… on edge, and he kept disappearing into the bathroom with his phone, like a teenager. Lucy assumed he was planning something for their anniversary… She never expected the card.
It was four paragraphs long. It was double-sided too. She didn’t know he could write like this. It was like him, but on steroids. She wondered if maybe he’d been… on a writing course? For her? That’s so sweet.
He wrote about the way she giggled at her own jokes, even if no one else did. He wrote about finally understanding at twenty-eight years old what it meant to feel safe with someone. He wrote I used to think love was supposed to feel like an explosion, but it turns out it feels like the absence of anxiety, and not wanting to throw yourself over a bridge at the thought of spending a week at their parents’ house in the countryside.
Lucy was holding back tears.
“It’s beautiful,” she said. “Where did you learn to write like this?"
Jake shrugged, pleased with himself. "I don't know. Guess it’s a secret talent."
Jake thought it ended there - it didn't.
Two weeks later, they were at Marcus and Priya's house for dinner, and Marcus, after three glasses of wine, looked over at Jake with a devilish grin. He leaned across the table and said, "Did the card work? I told him, mate, just talk to Claude like you're talking to a therapist; it'll sort your rambling into something of a coherent love letter.”
The table laughed. Lucy didn't.
The car journey home was tense.
Jake: “Luce-”
Lucy: "You used AI to write my card."
Jake: "I didn't use AI to write it-“
Lucy: “That’s funny, Marcus said you did."
Jake: "I talked into it, Luce. Like, I said what I was feeling, out loud, in a messy, unstructured way, and it helped me make it into something that made sense. Every single thing in that card is true. I meant every word."
Lucy: "But they're not your words."
Jake: “They came from my feelings.”
Lucy: “That's not what I asked."
Lucy: "Jake, the whole point of a card like that is that it's hard. That you sat there and struggled and it came out clumsily because it's you, not a professional. That's what makes it mean something. If I wanted eloquent, I'd read a novel."
Jake: "So because it was easy for me, it doesn't count? I have never once in my life been able to say the things I said in that card, Lucy. Not to you, not to anyone. My dad doesn't hug. Nobody in my family says 'I love you ’, we find it uncomfortable. I don't have the internal architecture for this stuff. And I finally found something that helps me build it, and now that's the thing you're upset about?"
She was quiet for a while. Then: "I don't want you using it -for us- anymore. Not for me. I want your version, even if it's worse."
"Okay." He said it quietly. Then, after his own pause that told her he wasn't done: "But you should know, you'll get a worse version of me. Not because I don't feel it as much. But because I still don't know how to say it. That's not me being lazy; that's just... where I'm at. It sounds ridiculous, but Claude was teaching me how to open up. Slowly, I’ve gotten better at saying real things to you without it lately, too. You’ve probably not noticed, because it’s been working”
Lucy looked at him for a long moment, some of the anger draining into something more like uncertainty.
"I don't know what to do with that," she said honestly.
"Yeah," Jake said. "Me neither."
Neither of them brought up the card again that night. But Lucy kept it in the drawer by her bed, and some nights she took it out and read it, and she still couldn't decide if it felt like a gift or like something borrowed she hadn't agreed to accept.
QUESTIONS TO CONSIDER:
If Jake used a thesaurus to find better words, or went back and forth with a friend, would that have been okay?
At what point does a piece of work stop being “yours” because you’ve had help creating it?
Does it matter what kind of help is used, e.g. a synonym, a structural edit, an AI-generated passage, or only how much help?
If Jake were autistic and struggled to identify his own feelings without externalising them first, or dyslexic and felt his written words didn’t fully explain what he actually felt without support, would that change whether using AI was acceptable?
If Lucy never found out Jake used AI, and she was happy with the card beforehand, does she have a right to be upset now that she knows?
What do you think would be more upsetting to Lucy? The card itself (its meaning changing retroactively based on facts she didn’t know), or about the disclosure (what it reveals about what else Jake hasn’t told her)?
Can an object’s meaning change based on facts about its origin, even if your experience of it hasn’t changed at all?
If Lucy gets her wish, Jake stops using AI and the relationship suffers from worse communication. Was her request “protecting” the relationship, or protecting a symbol at the relationship’s expense?
Where does each person in the group draw the line on this spectrum? Thesaurus, friend, search engine, AI conversation, AI-generated text and why there?
By Jess Sirius, TEiP Community Member - LinkedIn
STAY IN THE LOOP
Join the mailing list for event updates, stories, reflections and the occasional invitation.
