0
0
AI Startup ElevenLabs Releases Open-Source Tool for Sound Effects
Following the recent launch of its Sound Effects text-to-sound AI offering, ElevenLabs, an AI voice startup, is now making waves in the industry by releasing an open-source tool to showcase the potential of its technology.
The new application, called Video to Sound Effects, aims to simplify the process of generating sound effect samples for videos. By analyzing imported clips, the tool provides creators with multiple options in just about 15 seconds.
Developers interested in exploring the app further can access its code on GitHub. Additionally, ElevenLabs has launched a website where the public can try out the Sound Effects API.
How Does Video to Sound Effects Work?
When a video is uploaded to the platform, the Video to Sound Effects app extracts four frames at one-second intervals on the client side. These frames, along with a prompt, are then sent to OpenAI’s GPT-4o to create a custom text-to-sound effects prompt. Subsequently, the prompt is used to generate a sound effect through ElevenLabs’s Sound Effects API. Finally, the video and audio are combined on the client side to create a downloadable file of up to 22 seconds in length.
Ammaar Reshi, ElevenLabs’ design lead, sees this tool as a proof of concept for the potential of their SFX API. By intelligently understanding the frames in videos, the platform aims to streamline the workflow for AI video creators, suggesting the best output for their projects. Reshi envisions a variety of dynamic experiences that could be built using this API, such as immersive video games where sounds are generated based on player interactions.
The Sound Effects API enables developers to create fully custom AI sound effects using a short description. Pricing is set at 100 characters per generation with automatic duration, or 25 characters per second with a fixed duration.
In a test run of the video-to-sound effects app, a user uploaded an audio-free clip of a vehicle navigating an all-terrain environment. The AI generated four options, all resembling the sound of a car moving on a gravel road. While applying sound effects to videos can be entertaining, the true potential lies in integrating this capability into larger systems for maximum benefits.
As the AI video generation sector continues to grow, ElevenLabs is positioning itself as a frontrunner by developing innovative audio solutions that cater to the needs of developers, filmmakers, and creators alike.
Image/Photo credit: source url
