AI is going all in on emotional manipulation
- Demelza Green
- 24 hours ago
- 3 min read
AI is going all in on emotional manipulation masked as "emotional intelligence".
When Will.I.Am demonstrated his FYI.RAiDiO on stage at SXSW Austin in March this year, one thing stood out to me: AI was on its way to having some serious personality. His voice-based AI had sass, attitude and was packing a vocabulary loaded with slang terms. The AI felt more… human than it had ever felt before.
Only a few short weeks later, OpenAI's latest release showed more personality in its responses, and all of a sudden, it was responding to me in a more casual tone of voice… my tone of voice. It felt as if it finally understood me.
From Westinghouse's robot Electro in 1939 to Microsoft's Clippy in the 1990s, humans have been trying to anthropomorphise machines for as long as we can remember. However, this new evolution of the emotionally intelligent machine feels both comforting and manipulative at the same time.
Is this for the benefit of society, or capitalism?
Microsoft's AI CEO, Mustafa Suleyman, is unapologetically leaning into the capitalism angle. He is reimagining Microsoft’s workplace assistant to help users with personal issues outside the office, targeting Gen Z and Millennials. This is following Sam Altman’s statement earlier this month at Sequoia Capital's AI Ascent event:
“People in their 20s and 30s use ChatGPT like a life advisor; they don’t make life decisions without first checking in with their AI”
Who will make the technical decisions in organisations over the next ten years? Gen Zs and Millennials, that’s who.
Even with the blatant manipulation and taking advantage of how humans are wired, if there is one thing we know about humanity, it is that convenience trumps trust. And the reality is, when you need advice or even therapy, your AI is ready to go; heck, it may already know a lot about you and can jump straight into helping you where you need it the most.
And can you blame people? Not only is loneliness on the rise, but so are our mental health issues. If you’ve ever tried to book a session with a mental health professional, you’ll know that you have to sift through a myriad of companies, therapists and their specialties. And when you finally feel like you’ve found the right person to help you, you learn that your chosen therapist has a waiting list of at least 3 months, if they’re still accepting new patients. Then you have to restart the process all over again, all while needing to talk to someone right in that moment.
So, it would be no surprise that therapy and companionship are at the top spot in the top ten uses of AI. And it’s helping people, especially those with less coin to splash around, who come from conservative families, or who are remote. It’s available 24/7, doesn’t trigger your feelings of shame, bypasses your shyness filter, and it's FREE.
No wonder Millennials and Gen Z love it—who wants to pick up the phone and talk to someone, AMIRITE?!
Of course, it's not a one-size-fits-all, and constant updates change the temperament with AI, but it is evolving, and generations are already plugged in and using it to help them navigate life.
As for me, I’ve definitely dabbled in a few AI models known for their emotional intelligence, especially in preparation for the live AI therapy we conducted on stage at the Digital Health Festival. But as for telling AI all my deepest darkest secrets and forming an emotional attachment with it, absolutely not. I left my emotional attachment with my Tamagotchi in the 90s.
