ComfyUI InfoComfyUI Info

ComfyUI-MotionDiff

ComfyUI MotionDiff integrates 4DHuman, MotionGPT, MDM, MotionDiffuse and ReMoDiffuse into ComfyUI. It offers multiple installation methods. The product can perform 3D pose estimation from 4DHuman and generate and render motions. It outputs two types of stickmen: pseudo - openpose (faster but with limitations) and real openpose (better but needs gradient - based optimization). Key parameters like frames, smplifiy_iters, etc., can be adjusted. It also supports saving and loading SMPL data, with nodes for saving as.pt files, exporting meshes, and loading from the input folder.

Fannovel16

v1.0.1
200 stars
GitHub

Description

ComfyUI MotionDiff

Implementation of 4DHuman, MotionGPT, MDM, MotionDiffuse and ReMoDiffuse into ComfyUI

Installation

Installing prerequisites on Ubuntu (and maybe other Linux distros)

Run the following command in your terminal:

sudo apt-get install libglfw3-dev libgles2-mesa-dev freeglut3-dev

Install ComfyUI Manager and follow the steps provided there to install this repository.

Alternative

If you are using Linux or a non-admin account on Windows, make sure that the /ComfyUI/custom_nodes and comfyui_controlnet_aux directories have write permissions.

There is now an install.bat file. If the portable version is detected, you can run this file to install the software. Otherwise, it will default to the system installation, assuming you have followed ComfyUI's manual installation steps.

If you cannot run install.bat (e.g., if you are a Linux user), open the CMD/Shell and perform the following steps:

  • Navigate to your /ComfyUI/custom_nodes/ directory.
  • Run git clone https://github.com/Fannovel16/comfyui_controlnet_aux/.
  • Navigate to your comfyui_controlnet_aux directory:
    • For the portable/venv version:
      • Run path/to/ComfUI/python_embeded/python.exe -s -m pip install -r requirements.txt.
    • For the system Python version:
      • Run pip install -r requirements.txt.
  • Start ComfyUI.

Examples

3D Pose Estimation from 4DHuman

Generate and render

There are two stickmen outputs: pseudo-openpose and real openpose. The pseudo-openpose output takes less time but lacks a head, depth, and does not exactly match the Openpose format. The real openpose output is better overall but requires gradient-based optimization. Some important parameters:

  • frames: The number of frames to be generated. All models are currently trained on HumanML3D, which is a 20 fps dataset.
  • smplifiy_iters: The number of optimization iterations in the process of converting motion data to SMPL, similar to a training step.
  • smplify_step_size: Similar to a learning rate. A good loss value is around 1 to 15 for camera translation optimization and 1e+6 to 2e+6 for the second step. You can try decreasing the step size and increasing the number of iterations, similar to the training process.
  • smpl_model: The SMPL model for visualization. Only 1.0.0 models (10 shape PCs) have been tested. You can place your own models (.pkl files, with chumpy removed) in the ComfyUI-MotionDiff/smpl_models directory.
  • yfov: The vertical field of view in radians, which is similar to the inverse of zoom. Explanation by ChatGPT:

The floating-point vertical field of view in radians refers to the angle of the vertical viewing area in a graphical application or game. It determines how much of the vertical space is visible to the viewer. The value is usually specified in radians, which is a unit of measurement for angles. A larger value will result in a wider vertical viewing area, while a smaller value will result in a narrower vertical viewing area.

P/s: When using the depth map from SMPL rendering with ControlNet, it may generate half-naked women. The theory is that the model has learned the half-naked pattern in the depth map.

Saving and loading SMPL

The Save SMPL node saves SMPL data as a .pt file.

The Export SMPL to 3DCGI Software node saves meshes to a folder containing mesh files that can be imported into software such as Blender, Unreal Engine, Unity, etc.

Both nodes save the results in Comfy's output directory.

The Load SMPL node retrieves files from Comfy's input directory. You need to copy the .pt files from the input directory to the output directory.