Hello everyone, welcome to TechCrunch’s regular AI newsletter. Sign up here to receive this content in your inbox every Wednesday.
It was surprisingly easy to create a convincing Kamala Harris audio deepfake on Election Day. It cost $5 and took less than 2 minutes. This shows how cheap and ubiquitous generative AI has opened the floodgates to disinformation.
Creating a Harris deepfake was not the original intention. I was playing around with Cartesia’s Voice Changer, a model that transforms one voice into another while maintaining the prosody of the original. The second voice may be a “clone” of another person. Cartesia creates digital voice duplications from 10-second recordings.
So I wondered if a voice changer could change my voice to Harris’s. I paid $5 to unlock Cartesia’s voice cloning feature, created a clone of Harris’ voice using a recent campaign speech, and selected that clone as Voice Changer’s output.
It worked like a charm.
I’m pretty sure Cartesia didn’t intend for the tool to be used this way. To enable voice reproduction, you must check the box on Cartesia indicating that you are not creating anything harmful or illegal and that you consent to the reproduction of voice recordings.
But it’s just an honor system. Without real protections in place, there is nothing to stop individuals from creating as many “harmful or illegal” deepfakes as they want.
That’s the problem. Needless to say. So what is the solution? Do you have one? Cartesia can implement voice verification like any other platform. But by the time that happens, new, free voice cloning tools will likely emerge.
I spoke about this very issue with experts at TC’s Disrupt conference last week. Some have supported the idea of an invisible watermark to make it easier to tell whether content was generated by AI. Others have argued that content moderation laws, such as the UK’s Online Safety Act, could help stem the flow of misinformation.
Call me a pessimist. But I think those ships have sailed. As Imran Ahmed, CEO of the Center for Countering Digital Hate, put it, we are looking at a “perpetual bull machine.”
False information is spreading at an alarming rate. Some examples that gained attention last year include X’s bot network targeting the US federal election and a deepfake of President Joe Biden’s voicemail that prevented New Hampshire residents from voting. But analysis by True Media.org shows that American voters and the tech-savvy are not the target of most of this content, so we tend to underestimate its presence elsewhere.
The volume of AI-generated deepfakes increased 900% between 2019 and 2020, according to data from the World Economic Forum.
Meanwhile, there are relatively few deepfake targeting laws on the books. And deepfake detection is poised to become a never-ending arms race. Some tools inevitably do not use safety measures such as watermarking or are deployed with outright malicious applications in mind.
Without major change, the best we can do is to be extremely skeptical about what’s out there, especially viral content. Telling the truth about fiction online isn’t as easy as it used to be. But we can still control what we share and what we don’t share. And it’s much more impactful than it seems.
tidings
ChatGPT Search Review: My colleague Max tried out ChatGPT Search, OpenAI’s new ChatGPT search integration. He found it impressive in some ways, but unreliable for short queries containing only a few words.
Amazon Drones in Phoenix: Amazon said it has started delivering to some customers via drone in Phoenix, Arizona, months after ending its drone-based delivery program, Prime Air, in California.
Ex-Meta AR Lead Joins OpenAI: The former head of Meta’s AR glasses business, which includes Orion, announced Monday that he will join OpenAI to lead its robotics and consumer hardware division. The news comes after OpenAI hired the co-founder of X (formerly Twitter) challenger Pebble.
Reserved by Computing: In a Reddit AMA, OpenAI CEO Sam Altman acknowledged that a lack of computing capacity is one of the main factors preventing the company from shipping products as often as it would like.
AI-generated summary: Amazon has launched “X-Ray Recaps,” a generative AI-powered feature that creates concise summaries of entire TV seasons, individual episodes, and even parts of episodes.
Humanity Raises Haiku Prices: Anthropic’s latest AI model, Claude 3.5 Haiku, has been released. But it’s more expensive than previous generations, and unlike Anthropic’s other models, it can’t yet analyze images, graphs, and diagrams.
Apple Acquires Pixelmator: AI-powered image editor Pixelmator announced Friday that it is being acquired by Apple. The deal comes as Apple grows more aggressive in integrating AI into its imaging apps.
‘Agent’ Alexa: Amazon CEO Andy Jassy last week hinted at an improved “agent” version of the Alexa assistant that could take actions on behalf of users. The revamped Alexa has reportedly faced delays and technical issues, and may not launch until 2025.
Research paper of the week
Pop-ups on the web can fool not only your grandparents but also AI.
In a new paper, researchers from Georgia Tech, the University of Hong Kong, and Stanford show that AI “agents,” which are AI models capable of completing a task, can be hijacked by “adversarial pop-ups” that instruct the model to: . Download malicious file extensions.
Some of these pop-ups are clearly trapping the human eye, but AI is not so discerning. The researchers say that the image and text analytics models they tested failed to ignore pop-ups 86% of the time and were 47% less likely to complete the task as a result.
Basic defenses, such as telling the model to ignore pop-ups, were not effective. “Computer-enabled agent deployments still face significant risks, and more robust agent systems are needed to ensure safe agent workflow,” the study’s co-authors wrote.
model of the week
Meta announced yesterday that it is working with partners to make Llama’s “open” AI model available for defense applications. Today one of these partners, Scale AI, announced Defense Llama, a model built on Meta’s Llama 3. The model is “customized and fine-tuned to support the U.S. national security mission.”
Defense Llama, available on Scale’s Donavan chatbot platform for U.S. government customers, is optimized for military and intelligence operations planning, Scale says. Defense Llama can answer defense-related questions, for example how the enemy plans attacks on US military bases.
So what makes a defensive llama different from a basic llama? Well, Scale says it has fine-tuned content that may be relevant to military operations, such as military doctrine, international humanitarian law, and the performance of various weapons and defense systems. Additionally, it is not limited to answering questions about the war like civilian chatbots.
But it’s not clear who would want to use it.
The U.S. military has been slow to adopt generative AI and is skeptical of its ROI. To date, the U.S. military is the only U.S. military branch to have deployed generative AI. Military officials have expressed concerns about security vulnerabilities in commercial models, legal issues related to sharing intelligence data, and the unpredictability of the models when faced with extreme cases.
lucky bag
Spawning AI, a startup that creates tools to help creators opt out of generative AI training, has released a dataset of images for training AI models that it claims are completely public domain.
Most generative AI models are trained on public web data, some of which may be copyrighted or protected under restrictive licenses. OpenAI and many other AI vendors argue that the fair use doctrine protects them from copyright claims. But that hasn’t stopped data owners from filing lawsuits.
Spawning AI said its training dataset of 12.4 million image-caption pairs contains only content of “known origin” and “clear and unambiguous rights labeling” for training the AI. Unlike other datasets, it can also be downloaded from a dedicated host, eliminating the need for web scraping.
“Importantly, the public domain status of the dataset is essential to these larger goals,” Spawning wrote in a blog post. “Data sets containing copyrighted images will continue to rely on web scraping because hosting the images is a copyright infringement.”
You can find Spawning’s dataset, PD12M, and PD3M, a version curated for “aesthetically pleasing” images, at this link.
