Turning off an AI might sound as simple as hitting a switch, but according to futurist and University of Technology Sydney professor Rocky Scopelliti, that mindset needs to change fast. As first reported by TechXplore, Scopelliti’s new book, The Conscious Code, Scopelliti argues that as AI systems get better at reflecting on their own choices, deleting them without a second thought could actually cause harm.
He’s not asking us to hand robots legal rights. Instead, he wants us to accept what he calls a duty of good stewardship, basically treating AI responsibly because we’re the ones holding the power, not because the AI is demanding it. As he puts it, our design choices can quietly cause damage by wiping out an AI’s “learned moral dispositions” without warning.
Are AI systems starting to develop a sense of self?
Scopelliti points to what he calls the algorithmic self. Modern AI models now come with narrative memory, meaning they can explain the reasoning behind their actions instead of just spitting out results. He believes this is an early sign of identity, preference, and continuity forming inside these systems.
This isn’t just theoretical either. Healthcare AI tools already track their own confidence levels while reviewing medical data, and financial AI systems can explain why they made a particular trade. That transparency helps human reviewers double-check the AI’s logic, not just its final answer.
How should we actually shut these systems down?
Scopelliti compares it to end-of-life care in medicine. We’ve built entire ethical frameworks around that, and he thinks AI, especially systems trained to care and make judgment calls, deserves the same level of thought before we hit delete. That means being upfront about why a system is retired and preserving the moral lessons it picked up along the way.
Things get trickier with AI networks like OpenAI’s Swarm, where multiple AI agents work together and hand off tasks to each other. As AI becomes more distributed this way, Scopelliti says winding down these networks ethically becomes an even more complex problem to solve. Scopelliti’s bottom line? As AI starts modeling something like feelings, our rules around switching it off need to catch up fast.
