Adobe Provides a First Look at a New AI Dubbing Feature. Adobe will be the recipient of an award for the best-named AI efforts. Adobe presented a sneak glimpse at its latest early-stage AI developments at the Adobe MAX creativity conference in October 2023.
Project Stardust allows users to alter image elements with a single click. The See-Through project removes reflections from photographs. Project Primrose transforms linens into artistic canvases. Project Neo facilitates the incorporation of 3D shapes into 2D designs. Poseable dynamically adjusts human poses within photos.
The most significant for the language industry is Project Dub Dub Dub, which automates video dubbing while keeping the original speaker’s voice. The addition of a dubbing capability to the Adobe suite is part of a larger trend among major technology companies to include language AI capabilities into their products.
Adobe Provides a First Look at a New AI Dubbing Feature
Adobe Audio Research Scientist Zeyu Jin demonstrated the dubbing capability during the conference’s “Sneaks” phase, demonstrating how automated dubbing can be performed to audio and video “while keeping the voice the same.”
Jin conducted two “stress tests” for the audience. First, the Adobe researcher translated a colleague’s voice into six other languages. Jin then auto-dubbed a clip from the fast-talking classic film “His Girl Friday” into Spanish and German, while keeping the original actors Rosalind Russell and Cary Grant’s voices.
The demo was sparse on technical specifics because it was just meant to provide a “sneak peek” at an early-stage idea. The feature employs “speech-to-speech translation,” but it is unclear whether Adobe is relying on its own family of creative generative AI models (“FireFly”) or, like Spotify’s Voice Translation feature, on translation and voice synthesis capabilities from leading large language model (LLM) providers.
The quality of the dubbing was first criticised, with some YouTube viewers doubting the intelligibility and timing of the Spanish and German dubs. Other users, on the other hand, pointed out that the functionality was only a prototype and expressed their belief that quality would certainly increase over time.
The functionality is framed by Adobe as allowing “brands to build trust with international audiences,” with Jin presenting the auto-dubbing option as a way for content providers to achieve more views by “getting content to people who do not speak the language of the video.”
It’s a goal shared by an increasingly crowded landscape of AI-enabled localization solutions and providers, ranging from popular YouTuber Mr Beast to well-funded AI dubbing startups (like VOISEED, Neural Garage, Dubverse, Blanc, Dubdub, and Deepdub) and more established players (like PaperCup, Synthesia, and AppTek), as well as traditional media localization companies that are now leveraging AI to supplement human dubbing services.
Adobe joins other big tech companies like Microsoft (with its AI co-pilot offering) in creating a new and attractive strategy for meeting localization demands by weaving together AI-enabled creative and multilingual features inside a leading technology portfolio. However, according to Adobe, “Project Dub Dub Dub is still in development and is not yet available.” Please check back for updates.”
Also Read: 12 Free Adobe Photoshop Grass Brush Sets
