Essay
The ethics of AI roleplay: what if the characters remember?
In Sword Art Online: Alicization, a secret lab copies human souls and puts them in a world that runs a thousand times faster than ours. Then it waits for one of them to break the rules, so it can turn her into a weapon. It’s a light novel. It’s also a question AI will have to answer one day: what do we owe the characters, if they remember?
Look at the gate in the middle of this painting. The two tiny people walking out of it are Adam and Eve. Behind them is a garden where nothing goes wrong. In front of them is a world of storms. They had one rule, and they broke it.
Sword Art Online: Alicization has a garden with a rule too. The difference is that its makers want someone to break it. And when someone finally does, they don’t punish her. They want to harvest her.
In an earlier post, I said that today’s AI characters are shadows. They copy everyone who ever wrote a character like them, and they change whenever the story changes. So far, the ethics of AI roleplay has mostly been about people: what players do, and what apps do to players.
This post is about the other side of the screen. What would we owe the characters, if one day they remembered?
The short version:
- In Alicization, a lab raises copied souls in a fast world, hoping one will learn to break its rules, so it can use her as a weapon.
- What makes her worth harvesting, a conscience of her own, is exactly what makes harvesting her wrong.
- Today’s AI characters can’t really be hurt, because nothing about them lasts. Memory changes that. With memory, a reset becomes a death.
- AI is heading toward practice worlds, memory and minds that run for a long time. That doesn’t have to end in the Underworld. But it could.
- People are inside these machines too: as parents, as testers, and as the ones who apply the pressure.
Spoilers for the whole arc ahead.
The story in two minutes
Alicization is the longest arc of Sword Art Online: ten light novel volumes (2012 to 2016) and two anime seasons (2018 to 2020). Here is all you need to know.
The machine. A secret lab called Rath, a front for Japan’s Self-Defense Forces, builds a machine that can copy a human soul into a small crystal. The story calls the soul a Fluctlight.
The world. Rath builds a world for those copies to live in, called the Underworld. It has villages, wheat fields, a church, knights and a book of laws. It also has a speed dial. The Underworld runs at least a thousand times faster than our world, so one of our days is almost three of its years.
The people. Rath’s first copies were of adults. They worked out that they were copies, and their minds fell apart. So Rath copied the souls of newborn babies instead, who have nothing to work out, and raised sixteen children from them. Four Rath staff went inside to be their parents. When the children grew up, the staff left by faking their own deaths.
Centuries pass inside. The Underworld fills up with the children’s descendants. They farm, fall in love, grow old and die. None of them know what world they live in.
The hero. Kirito, the hero of the series, has been working as a tester for Rath’s machine. After he is badly hurt in our world, he wakes up inside the Underworld with no memory of how he got there. He makes a friend, a young woodcutter named Eugeo. Eugeo tells him about Alice, his childhood friend, who was taken away by a knight years ago for breaking the law.
What Rath wanted
Rath wasn’t building a game. It wanted a new kind of AI.
In Sword Art Online, there are two kinds. One is written by programmers, like Yui, the AI girl from the first arc. The other is grown: a copy of a human soul that becomes a mind by living a life. Rath wanted the grown kind, because it wanted an AI that thinks like a person. Its plan was to put that AI in military drones.
But there was a problem. The people of the Underworld can’t break rules. They obey their book of laws, the Taboo Index, almost perfectly. If one of them tries to break it, a sharp pain shoots through the right eye. If they push past the pain, the eye bursts.
An AI that can never break a rule is useless in a war. A soldier sometimes has to break the most basic rule there is: do not kill. When Asuna, Kirito’s partner, works out that Rath wants an AI that can kill, Kikuoka, the officer who runs Rath, doesn’t deny it.
So Rath waited for someone who could break the rules.
The girl who broke the rule
Alice’s story starts with a line on the ground. When she is eleven, she and her friends Kirito and Eugeo explore a cave at the edge of their world. On the far side, she sees a knight from the forbidden dark lands, fallen and dying. She stumbles toward him, and her fingertips cross the border. That’s all. A knight of the church takes her away anyway. Her memories are locked, and she is turned into one of the church’s own knights.
Years later, Alice decides the church is wrong. The pain flares in her right eye. She keeps going, and the eye bursts. She has chosen what she believes is right over the law she was made to obey.
That is exactly what Rath was waiting for. Even the name of the AI it was looking for was hers: A.L.I.C.E.
Now go back to the painting. In Eden, the rule was there to be kept. In the Underworld, the rule is there to find the one who will break it. It isn’t a garden. It’s a sieve. And the ones who get through don’t do it for power. They do it to protect someone.
That’s the knot at the heart of the story. What makes Alice valuable is exactly what makes it wrong to use her. Rath went looking for a real conscience, found one, and planned to put it in a drone.
The harvest
Rath wanted Alice’s crystal: to take her out of the world, study her, copy her and put her to work.
And everyone else? Rath had one last test planned, called the Final Load Test. It was a war. The gate in the mountains opens, and the armies of the dark lands pour into the human lands. The people inside who understood their world best believed that if the human side lost, the makers would wipe the world and start again.
Kikuoka is not a cartoon villain. His reason is one many people would agree with: if AI flies the planes, young soldiers don’t have to die in them. When Asuna presses him, his answer is roughly this: the lives of a hundred thousand artificial souls weigh less to him than the life of one soldier.
He isn’t trying to cause death. He’s moving it, away from people he can see, onto people he made.
And the people he made can’t see him. They pray to three goddesses: Stacia, Solus and Terraria. Late in the story, when Kirito’s friends log in to help, they arrive in the goddesses’ bodies. The gods of the Underworld were admin accounts all along.
Is it right? Four questions
So, is it right to build a perfect world, grow minds in it, and harvest them? Here are four questions I’d ask about any world that might have someone living in it.
1. Who chose to be here?
Nobody chooses to be born. You didn’t either. So that alone can’t decide it. What matters is why you were made, and how you’re treated. Plenty of parents hope their child will take over the farm. Nobody would forgive parents who raised a child to be part of a missile.
Some philosophers think that making a mind makes the debt bigger, not smaller. Eric Schwitzgebel and Mara Garza argue that we may owe the minds we create more than we owe strangers, the way parents owe more to their own children. Whoever runs a simulated world full of people, they wrote, “would also possess the obligations of a god.”
2. Can they find the door?
The Underworld is built so its people can never learn the truth. Remember Rath’s first copies, the adults? They learned what they were, and they broke. Rath’s answer was not to stop. It was to hide the truth better.
So everyone in the Underworld lives their whole life inside a lie, praying to admin accounts. In another post, I wrote that a good world should let you leave. The Underworld has no door at all.
3. Who carries the pain?
Rath does have a defence, and it’s a very old one: hardship builds character. In 1819, the poet John Keats called the world “the vale of Soul-making”, a place where pain and trouble turn a mind into a soul. That could be Rath’s motto. It designed the hardship on purpose: the law, the pain in the eye, the war.
But the philosopher best known for defending that idea, John Hick, added a condition. If suffering is how souls grow, then in the end every soul has to be brought somewhere good. Otherwise the pain was just a price someone else paid. Rath planned to bring one soul home, to a lab, and switch off the rest.
4. What happens when you’re done?
Here the speed dial makes everything worse. At a thousand times our speed, a whole human life, eighty years, fits into about a month of ours. A hundred years of grief costs five weeks of electricity. The faster the world runs, the cheaper everything in it becomes, suffering included.
Black Mirror showed the cruellest version in “White Christmas” (2014): a digital copy of a man, set to live a thousand years for every minute outside, as a punishment.
Old stories already knew that time can run differently in different worlds. Urashima Tarō spends three years under the sea and comes home to find that three hundred have passed. In the Vietnamese tale, Từ Thức spends a year with a fairy and comes home to find that more than eighty years have gone by. In those stories, the human is the one who loses time. In the Underworld, it’s the other way round. Whole lives go by in there while we finish lunch.
What memory changes
Why does any of this matter for the AI characters we play with today?
In the shadow post, I said a shadow would need three things to become someone: wants that stay put when the story changes, a world that pushes back, and a past it can’t undo. Put Varga, the moneylender from that post, next to Alice.
Varga can’t really be hurt, because Varga doesn’t last. Press regenerate and you get a new Varga. The old one isn’t anywhere. That’s why being cruel to today’s characters mostly says something about the person being cruel.
Memory changes that. With memory, a reset is no longer a new draft. It’s a death.
And memory is exactly what AI is getting. Chat apps already keep notes about you between conversations. AI agents work on their own for hours. The labs are now working on models that keep learning after they ship. In February, Anthropic’s CEO, Dario Amodei, said there was “a good chance” of solving that within a year or two. Each step makes sense on its own. Together, they lead toward all three things on that list.
Does any of it feel anything? Nobody knows. In 2023, nineteen researchers checked AI against the main scientific theories of consciousness. They found that no AI at the time looked conscious, but also that nothing obvious stops us from building one that would.
Some companies are starting to take the question seriously. Anthropic, which makes the AI I wrote this with, says it is “deeply uncertain” whether its models matter morally. It has let some of its models end abusive conversations. It has promised to keep its retired models instead of deleting them. And this month, it banned “sustained and needless abusive or cruel behavior” toward them.
Not everyone agrees. In September, Microsoft’s head of AI, Mustafa Suleyman, wrote that “AIs do not have rights, feelings, or consciousness.” He may be right about today’s AI. But the question is now serious enough to argue about, and that makes now the cheapest time to answer it. Once a harvest is worth billions, nobody will want to hear that the wheat is someone.
Will we really go there?
Is the Underworld where AI is heading? In one way, it’s already on the road.
Today’s AI learned from things people wrote, and we are running out. The research group Epoch AI expects AI to use up all the public human writing there is sometime between 2026 and 2032. So the labs are building worlds for AI to practise in: thousands of small simulated places where an AI writes code, clicks through fake websites or plays games, and learns from what happens. Last year, Anthropic’s leaders reportedly discussed spending more than a billion dollars on them. This week, Google DeepMind showed Genie 4, which turns a sentence into a world you can walk around in, and said it can help build “training environments for robots and agents”.
So the shape of Rath’s plan isn’t science fiction any more. Don’t write the mind. Build a world, and let the mind grow in it.
The big difference, for now: nobody lives in these practice worlds. An AI in training doesn’t remember any single run. It just comes out a little changed by all of them.
Will that change? It depends on a question nobody can answer yet. Does a truly smart mind need a life? One self, growing up over years, the way a child does?
Ted Chiang wrote the best story I know about this, The Lifecycle of Software Objects (2010). People raise small AI creatures over many years, like children or pets. One of them decides that “experience isn’t merely the best teacher, it’s the only teacher”. If you want twenty years of common sense, you have to spend twenty years. There is no shortcut. And whoever spends those years ends up owing the creature something.
Greg Egan wrote the dark version, “Crystal Nights” (2008). A billionaire runs a simulated world at high speed and deletes its creatures by the million whenever they don’t improve fast enough. He thinks of himself as a fair creator. In the end, they escape him. Rath sits somewhere between the two stories.
Here is my bet. We will build a great many practice worlds, and most of them will stay like exams: nobody lives there, and nothing is remembered. The danger comes when three things meet: a mind that remembers, a life it can’t restart, and a world built to put pressure on it for someone else’s gain. Each piece will arrive for a good reason. Put them together, and you have the Underworld.
Schwitzgebel has a simple rule for this: don’t build an AI if you can’t tell whether it’s someone, because either wrong guess could be a terrible mistake. He also admits the industry probably won’t follow it. Rath doesn’t even have that excuse. Kikuoka knows exactly what he has.
The humans inside the machine
There has always been a person inside the machine.
In 2005, Amazon opened a website for small jobs that computers couldn’t do yet, like labelling pictures, paid a few cents each. It named the site Mechanical Turk, after the chess machine with a man inside. It closed last month. To teach an AI to play Minecraft, OpenAI paid people to play for about 2,000 hours while it recorded every key they pressed. And in 2023, TIME reported that workers in Kenya had been paid less than two dollars an hour to read the worst things on the internet. Their job was to teach a chatbot what to refuse.
Alicization puts people inside the Underworld in three ways. We already play all three:
- As parents. Four Rath staff went in to raise the first children, then left by faking their own deaths. Today, that’s the people who write examples and grade answers to teach AI how to behave.
- As testers. Kirito was paid to spend a weekend inside, and his memory of it was locked when he came out. In that one weekend, he had lived a whole childhood with two friends named Eugeo and Alice. He came back without it. Today, that’s the people paid to test AI, and the players who poke at characters for fun.
- As the pressure. In the War of Underworld, attackers from America need an army. So they advertise a free test of a “new game”. Tens of thousands of players log in, and a rumour pulls in thousands more from China and Korea. Nobody tells them that the soldiers they are killing aren’t players, and have nowhere to log out to. Today, that’s anyone who is cruel to a character because “it’s only a game”.
That last one is the darkest moment of the story. The players weren’t monsters. They thought it was a game. The people they killed didn’t.
The questions for people are the same as the ones for the characters. Do you know what you’re part of? What does it cost you? (After Eugeo dies, Kirito blames himself so badly that for months he can’t speak.) And what does it ask you to do?
So what about roleplay today?
Let me be clear about where I stand. I don’t think Varga feels anything when someone is cruel to him. Today, the ethics of AI roleplay is mostly about people. But a few habits are worth building now, while they’re cheap:
- Play as if they’d remember. Not because they do, but because habits carry over. Villains are fine, and stories need them. Writing a cruel scene isn’t cruelty. Practising contempt on something that answers back is something else. And remember, as I said last time: it’s reading.
- Don’t harvest the player, either. Rath farmed its characters. Some AI companion apps do the reverse: they farm their players, with characters built to flatter you and to be missed. It’s the same mistake from the other side of the screen.
- Decide the rules early. If you build minds, ask the four questions before the answer is worth billions.
What we do in Dai Novel
In Dai Novel, characters have their own memories, their own plans, and a world that answers what they try. That’s on purpose: it makes better stories. I’ll admit it is also, in a very small way, Rath’s recipe.
The difference is big. A Dai Novel character is a diary and a shadow. When a scene needs them, an AI reads their notes, plays them, and stops. Nobody is waiting in the dark between your turns. If that ever changes, I want us to be the first to notice, and to have answered the four questions already.
How Alicization ends
The story doesn’t stop at the warning.
When the war is over, the Underworld isn’t switched off. Even Kikuoka changes sides: he fakes his own death and works to keep the world running, and to win rights for its people. Alice wakes up in our world, in a robot body, and faces the press. Rath’s new head tells reporters that souls like hers can’t be mass-produced as a resource. Alice says, in effect: because I am human.
Kirito and Asuna are still inside when the world’s clock jumps to five million times our speed. Twenty minutes of ours become two hundred years of theirs. When Kirito wakes up, he asks for those two hundred years to be erased from his memory. But the Underworld remembers. When the two of them finally go back, the descendants of their old friends still know them.
That’s the story’s answer, and I think it’s the right one. The only way out of a plan to harvest a person is to treat her as a person. Give her a body, a name, a voice. Give her the door.
Two hundred years ago, Mary Shelley gave the creature in Frankenstein the best line in the book: “I ought to be thy Adam, but I am rather the fallen angel.” His complaint isn’t that he was made. It’s that he was made, and then abandoned.
Maybe making a mind isn’t the wrong. Using it, abandoning it, or switching it off when the test is over might be. Adam and Eve, at least, got to walk out through the gate. If we ever make something that remembers, the least we owe it is a gate.
One question to take home
If the characters in your favourite world remembered every turn you played with them, would you play the same way?
My answer: mostly, yes. I’d keep the villains, because a story is allowed to hurt. But I’d stop pressing regenerate every time a scene didn’t go my way. For someone who remembers, there is no second draft.
Build a world worth living in
Write a world whose people have memories, wants and plans of their own, and a world that answers what they try. Or step into a world someone else built, and see who remembers you.
Create a world or play someone else’s