Amazon, just say no: The looming horror of AI voice replication

Do we really want to put the power of perfectly simulating a voice in the hands of stalkers and abusers?

Getty Images

Last week, we ran a news article entitled, "Amazon's Alexa reads a story in the voice of a child's deceased grandma." In it, ZDNet's Stephanie Condon discussed an Amazon presentation at its re:MARS conference (Amazon's annual confab on topics like machine learning, automation, robotics, and space).

Ethics of AI: Benefits and risks of artificial intelligence

The increasing scale of AI is raising the stakes for major ethical questions.

Read now

Hard stop. Did the hairs on the back of your neck just raise up? 'Cause that's not creepy at all. Not at all.

Prasad, though, characterized it as beneficial, saying "Human attributes of empathy and affect are key for building trust. They have become even more important in these times of the ongoing pandemic, when so many of us have lost someone we love. While AI can't eliminate that pain of loss, it can definitely make their memories last."

The best smart speakers

Here are your best options from Google, Apple, and more.

Read now

Now, let's take it up a notch. Just how easy would it be to send someone off the edge if they kept hearing the voice of their dead father or mother? If an abuser can convince someone they're being haunted or are losing control of their ability to discern reality, that abuser could then substitute in a malevolent subjective reality.

The whole idea sounds like bad fiction, but gaslighting is so prevalent in domestic abuse that the National Domestic Violence Hotline has an entire page dedicated to the gaslighting techniques an abusive partner might use. If you find yourself in this situation, you can reach the hotline at 1-800-799-7233.

Let's take it up another notch: add stalkers to the mix. Let's say you're at home and you get a call from your mom. It's your mom's number on caller ID. You answer and it sounds like your mom. She's been in an accident, or is in some kind of trouble. She begs you to come out and get her. And you do. Because your mom called, and of course you know what she sounds like.

Which Amazon Echo to buy? How to pick the best Alexa device for your needs

Amazon now has an entire army of Echo devices. Some listen to you. Some also watch you. Which should you choose? We help you decide.

Read now

My wife and I had a great time traveling across the country with the wise guidance of Yoda. We had added his voice to our old GPS. During our travels, we had Yoda's voice guiding us, turn by turn. It was comforting, especially during those long open and empty stretches, to have Yoda's calming voice and statements like "left, you must turn" to keep us on track.

Sadly, the Yoda Positioning System is no longer available, which may (or may not) say something about the market viability for celebrity character voices in personal electronics.

Stopping to think

There is a line in Jurassic Parkthat comes to mind at times like this: "Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should." In the movie's case, it was about recreating dinosaurs from dino DNA. But it applies just as well to using AI to recreate people's voices.

Entertainment AI software Respeecher needs one to two hours of voice samples to recreate a voice. Amazon's new technology requires less than a minute of recordings. That opens the door to a great many more recordings, including messages captured from voice mail and even commands given to Alexa and Siri.

Given the prevalence of sales of PII information, and even medical information, on the so-called Dark Web, it's logical to expect that hackers will also traffic in short voice recordings of potential victims, especially when those recordings only have to be a minute or so.

That means that if Amazon does release its dead grandma skill to the Alexa platform, it will be accessible to a broad audience. It's even possible to use Alexa and Alexa skills on home-grown non-Alexa devices, as this article shows. That means that even if this technology is limited to Alexa, it has the potential to be very problematic.

I have to ask: does Amazon really need to release this technology into the wild as a skill?

To be fair, Amazon isn't going to be the only company exploring voice replication. Fortune Business Insights predicts the global speech and voice recognition market will reach $28.3 billion by 2026 at an annual growth rate of almost 20%. With those kinds of numbers, you can be sure there will be other participants in this arena.

Protecting users from digital gaslighting, stalking, and scams will get progressively more difficult, and voice replication only makes it worse.

Writing in the Lawfare Blog, Dr. Irving Lachow, deputy director, cyber strategy and execution at the MITRE Corporation and a visiting fellow at the Hoover Institution, describes this situation as "PsyOps in the home."

He states that although there are anti-stalking laws on the books, "Many of these measures cannot be applied directly to cyber gaslighting because, unlike the stalkerware situation, abusers are not adding software to home-based smart devices in order to harass their victims. Instead, they are using the devices as they were intended to be used."

He also says that legal challenges are more difficult where executives at companies producing these technologies are somewhat untouchable. He states, "The technologies in question have legitimate and positive uses, and one cannot realistically target the executives of smart device companies just because their equipment has been used to cause someone harm."

Clearly, this is an issue that needs more consideration. Companies like Amazon need to evaluate carefully whether the features they're adding do more harm than good. Cybersecurity experts need to continue to harden IoT devices against outside hacking. Therapists and psychologists need to increase their awareness of digital gaslighting and other 21st-century threats.

But individuals can also protect themselves. Malware intelligence analyst Christopher Boyd, writing on behalf of the Malwarebytes blog, recommends keeping detailed records of incidents and log any data produced by devices. We add that it's important to manage your own passwords, use strong passwords, and if you're expecting trouble, be sure to learn how to lock down your devices.

Lachow reports, "Smart devices are ripe for exploitation in domestic abuse scenarios because often one person, usually a man, controls the information technology (IT) for the house. If the IT manager moves out but retains access to home-based smart devices via mobile apps or online interfaces, he or she could control the household environment."

Keep that in mind and learn all you can. As for AI-based systems learning to replicate the voices of deceased relatives, I have to say "Just say no." No. No. No-no-no. Baaaadthings could happen.

I'm sure you can think of plenty of horrifying scenarios. Share those with us in the comments below. The more we're thinking about this and the more we're aware of it, the better we can prepare for it.

Artificial Intelligence

AI in 2023: A year of breakthroughs that left no human thing unchanged These are the jobs most likely to be taken over by AI AI at the edge: 5G and the Internet of Things see fast times ahead Almost half of tech executives say their organizations aren't ready for AI or other advanced initiatives AI in 2023: A year of breakthroughs that left no human thing unchanged

These are the jobs most likely to be taken over by AI

AI at the edge: 5G and the Internet of Things see fast times ahead

Almost half of tech executives say their organizations aren't ready for AI or other advanced initiatives

---

You can follow my day-to-day project updates on social media. Be sure to follow me on Twitter at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.

Editorial standards Show Comments