Adobe Firefly Adds AI-Powered Soundtrack and Speech Generation

First, there was image generation in Adobe Firefly. Then came video, sound effects, and audio translation. Now, Adobe is adding two new capabilities to its generative AI platform: soundtrack and speech creation.

“The introduction of the Generate Soundtrack application and model from Adobe…is the first model that allows you to generate instrumental soundtrack that you can use in film, TV—you can use anywhere, because it’s commercially safe, and we’ve trained it like we trained the other Adobe Firefly models with artists in mind and giving them the full power of using those capabilities wherever they need,” Alexandru Costin, Adobe’s vice president of generative AI and Sensei, shared during a press briefing last week.

“We’re also introducing the Generate Speech module that enables you to put in a text, emphasize particular words, and then generate from a large library of voices, generate a speech based on the text you’ve inputted,” he went on to say. To help power this feature, Adobe has enlisted ElevenLabs, which is also one of the newest partner models for Firefly.

Soundtrack Generation (Beta)

For video creators looking for custom songs for their short films but who don’t have the budget to hire professional musicians—or aren’t even sure what they want—Firefly’s soundtrack generator can be a good starting point. Sifting aimlessly through marketplaces like Epidemic Sound, Artlist, PremiumBeat, or Bandcamp is one thing, but there’s something about a song that fits perfectly with a film—and those marketplaces often lack the right filters to help narrow the selection.

Using a Mad-Lib-like interface in Firefly, creators can type in the type of song (“sentimental”), the style (“acoustic”), and the kind of scene or film (“thriller”). Alternatively, using natural language, they can describe the vibe (“fun” or “upbeat”), genre (“hip hop”, “electronic”, or “K-Pop”), the energy level, tempo (“slow”, “medium”, or “high”), and duration (up to 60 seconds). The video can also be directly uploaded to help the AI identify the best tracks to generate. Ultimately, creators will receive multiple tracks to choose from.

One initial drawback appears to be that Firefly doesn’t automatically match a selected track to the right scene, leaving creators to align the music with the action manually.

Generating soundtracks will cost you Firefly credits, though the shorter the track, the fewer credits are charged.

Speech Generation (Beta)

For those unable to hire professional voiceover talent, Adobe offers a solution with its speech synthesis. Within Firefly, users input the text they want read back, specify the words they want emphasized, and generate speech choosing from an extensive library of voices.

Adobe isn’t the only company offering speech generation. Competitors include Murf.ai, Microsoft’s VibeVoice, Amazon Polly, Lovo.ai, and ElevenLabs. What separates Adobe from the rest is its focus on commercially safe generations. And even though ElevenLabs is a partner, creators and brands may feel more assured knowing that through Firefly, the AI voice startup won’t use their data to train its models.

The maximum length of a speech that Firefly can generate remains unclear. I’ve asked Adobe, but haven’t yet heard back. Moreover, the company has not stated the cost of using the speech generation feature. However, it’s likely to cost about the same as soundtrack generation, requiring fewer credits depending on the length of the speech the creators want.

Firefly Video Editor (Beta)

Another thing Adobe is announcing is a web-based video editor.

“We know that generating clips, downloading them, and using them in Premiere is a workflow customers do today. We want to make this workflow even easier. So the Firefly video editor, which is opening up its beta waitlist at Max enables…nonlinear editing, a timeline-based editing generation, importing videos, applying effects, and bringing all of that together into a really functional web video editor,” Costin shares.

In other words, the Firefly video editor is a simplified version of professional video editing software. Creators can drag and drop video clips, use text prompts to generate new video segments, add effects easily, and create videos directly in the browser without downloading professional software.

These new tools are one of many major announcements coming out of Adobe Max this week. Along with them, Adobe is unveiling Firefly 5, integrating new third-party LLMs into the platform, rolling out Firefly Custom Models for all creators, launching an Adobe Premiere app designed specifically for YouTube Shorts, embedding AI assistants into Express and Photoshop, and teasing a new initiative dubbed Project Moonlight.

Featured Image: Credit: Adobe