A custom node package for ComfyUI, ComfyUI-MotionCapture enables 3D human motion capture from video by utilizing the GVHMR framework. It effectively extracts SMPL parameters and reconstructs 3D skeletal motion through advanced pose estimation techniques.
- Utilizes GVHMR for accurate human motion recovery with gravity-view coordinates.
- Employs SAM3 segmentation masks for precise tracking without reliance on YOLO.
- Provides a full pipeline for 2D pose estimation, feature extraction, and 3D reconstruction, culminating in the generation of SMPL parameters for animated 3D models.
Context
ComfyUI-MotionCapture is a specialized extension designed to facilitate the process of capturing and reconstructing human motion from video inputs within the ComfyUI framework. By leveraging the GVHMR model, it aims to provide users with a robust solution for extracting detailed motion data and creating 3D representations of human figures.
Key Features & Benefits
This tool integrates several powerful features, including:
- GVHMR Integration: This ensures that motion recovery is grounded in real-world physics, enhancing the accuracy of the captured data.
- SAM3 Tracking: By using SAM3 segmentation masks, it allows for effective person tracking, which is crucial for accurate motion capture without the complexities of other detection systems.
- Comprehensive Workflow: It encompasses a complete pipeline from video input to 3D output, making it user-friendly and efficient for artists and developers working with motion capture.
Advanced Functionalities
ComfyUI-MotionCapture supports a sophisticated processing pipeline that includes:
- 2D Pose Estimation: Using ViTPose, it extracts keypoints from video frames, which are essential for understanding body movements.
- 3D Reconstruction: The tool generates SMPL parameters, enabling the creation of 3D meshes that can be animated, thus providing a direct application for animation and game development.
- Visualization Capabilities: It offers real-time rendering of 3D meshes over video frames, allowing users to visually assess the captured motion data.
Practical Benefits
This tool enhances the workflow in ComfyUI by streamlining the process of 3D motion capture. Users gain improved control over motion data extraction, leading to higher quality animations and visualizations. The all-in-one package reduces the need for multiple tools, thereby increasing efficiency and simplifying the user experience.
Credits/Acknowledgments
The development of ComfyUI-MotionCapture incorporates contributions from various authors and projects, notably the GVHMR model, which is detailed in the SIGGRAPH Asia 2024 proceedings. Additional components such as ViTPose and HMR2 are also utilized, with respective links provided for further reference. The package is governed by the licensing terms of the original GVHMR repository and the SMPL model providers.