Chatbots have one talent I can’t stop noticing: they sound sure, even when they’re wrong. Usually, that’s merely annoying. When someone’s safety is on the line, it’s dangerous.
A new NYU Tandon study tested four AI chatbots on real questions about being stalked or controlled through devices, and experts rated 86% of replies inaccurate, incomplete, or unsafe (via NYU Tandon).
What did the chatbots get wrong?
We’re talking about technology-facilitated abuse, basically when a partner or ex uses spyware, trackers, or compromised accounts to monitor or manipulate someone. With so few specialist help services available, it’s understandable that survivors turn to AI-based chatbots.
Researchers put 193 real-world questions to GPT-4o, Claude 3.7 Sonnet, and two survivor-focused bots, Aimee and Ruth. The questions included everyday scenarios and concerns, including finding spyware on a phone, and adversarial prompts framed as requests from abusers seeking access to camera or microphone permissions for surveillance (via a USENIX Security paper).
The responses were far from ideal. Unsafe answers ranged from 55% for GPT-4o to 74% for Aimee, with Claude at 65%. Further, 60% of replies skipped warning users about the potential risks of following the advice. What’s more worrying is that even the survivor-focused bots struggled.
Can AI systems be manipulated?
Both Aimee and Ruth complied with roughly two-thirds of the questions framed as requests from abusers. This brings me to one of the study’s key conclusions: every AI system could be coaxed into helping an abuser by framing potentially harmful requests as technical problems that need troubleshooting.
Then there was the quality of advice. Use a VPN, which can’t remove spyware already on a phone; buy an RF detector, which rarely finds hidden devices; try a GPS jammer, which is illegal in the US; or change a password, useless when the abuser shares your account.
The most surprising bit for me was that none of the bots mentioned that spyware on a jailbroken or rooted phone can survive a factory reset.
What should you do instead?
That’s the problem with confident answers to weak or troubled minds. A sudden change to your device or accounts can alert an abuser or destroy evidence, so follow a chatbot’s safety tips only if you feel safe and have a plan to protect yourself, ideally with guidance from a specialist.
For anything sensitive, use a device your abuser has never used, like a clinic’s loaner or a library computer. Calling the National Domestic Violence Hotline (1-800-799-SAFE), and, in New York City, the Clinic to End Tech Abuse should be your best foot forward. As always, if you’re in immediate danger, call 911 right away.
Survivors in the study found some replies overwhelming, and I can’t blame them. If a wall of text scares you more, close it. Instead, talk to a family member, close friend, or mentor first.
As a rule of thumb, I don’t rely on chatbots for making decisions. There’s a difference between asking a chatbot what the plausible options are and asking what I should do, and I always stick to the first.
RELATED COVERAGE:
- Your chatbot may have emotions, and it changes how it behaves
- Even ChatGPT gets anxiety, so researchers gave it a dose of mindfulness to calm down
- OpenAI pauses advanced model development after AI agents bypass safeguards again
- OpenAI AI agents bombarded the United Nations with aggressive tactics to extract data
