A ComfyUI node designed to replace spoken audio with synchronized "woof" sounds, ensuring that the mouth movements of a dog character align with the rhythm of the original speech. This tool utilizes advanced transcription techniques to create a seamless integration of audio and character animation.
- Utilizes Whisper transcription for precise timing of woof sounds based on Japanese mora boundaries.
- Offers two modes of operation: a syllable-based mode for accuracy and a volume-based mode for simplicity.
- Allows users to customize the woof sound, enhancing the personalization of audio output.
Context
This tool, known as ComfyUI-Voice2Woof, serves a specific function within the ComfyUI framework by translating Japanese audio into a series of timed "woof" sounds. Its primary purpose is to maintain the synchronization of a dog character's mouth movements with the rhythm of spoken language, enhancing the realism of animations.
Key Features & Benefits
The tool's standout feature is its use of Whisper transcription to identify and utilize mora boundaries, which are essential in Japanese phonetics. By grouping these morae and placing woofs at specific intervals, it ensures that the audio output closely mirrors the original speech cadence. Additionally, the ability to select between syllable and volume modes allows users to tailor the tool's performance according to their needs, whether they seek precision or simplicity.
Advanced Functionalities
In syllable mode, the node employs Whisper's advanced capabilities for forced alignment, providing character-level timestamps that facilitate accurate audio placement. This functionality is particularly beneficial for projects that require high fidelity in audio-visual synchronization. The volume-based mode, which uses onset detection, offers a more straightforward alternative, making it accessible for various applications beyond Japanese audio.
Practical Benefits
By integrating this tool into their workflow, users can significantly enhance the quality and efficiency of their audio-visual projects within ComfyUI. The ability to dynamically adjust woof spacing based on the energy of the speech allows for a more engaging and lifelike representation of character animations. Moreover, the customizable woof feature provides an additional layer of creative control, enabling users to personalize their projects further.
Credits/Acknowledgments
This tool builds upon several dependencies, including stable-ts for Whisper functionality, pykakasi for accurate mora counting, and librosa for audio analysis. It is a collaborative effort, reflecting contributions from various developers, and is subject to the open-source licenses associated with its components.
