ComfyUI InfoComfyUI Info

ComfyUI-CLIPSeg

This repository provides two custom nodes for ComfyUI. The CLIPSeg node uses the CLIPSeg model to generate a binary mask and heatmap overlay for an input image based on a text prompt, with options to control blur, threshold, and dilation. The CombineSegMasks node combines two or three masks into one. Installation involves cloning the repo and placing a file in the custom_nodes directory. Usage examples are provided in a JSON file. Requirements include PyTorch, CLIPSeg, OpenCV, numpy, and matplotlib.

time-river

25 stars
GitHub

Description

Custom Nodes for ComfyUI: CLIPSeg and CombineSegMasks

This repository includes two custom nodes for ComfyUI. These nodes leverage the CLIPSeg model to generate masks for image inpainting tasks based on text prompts.

1. CLIPSeg

The CLIPSeg node creates a binary mask for a given input image and text prompt.

Inputs

  • image: A torch.Tensor that represents the input image.
  • text: A string that represents the text prompt.
  • blur: A float value used to control the amount of Gaussian blur applied to the mask.
  • threshold: A float value used to control the threshold for creating the binary mask.
  • dilation_factor: A float value used to control the dilation of the binary mask.

Outputs

  • tensor_bw: A torch.Tensor that represents the binary mask.
  • image_out_hm: A torch.Tensor that represents the heatmap overlay on the input image.
  • image_out_bw: A torch.Tensor that represents the binary mask overlay on the input image.

2. CombineSegMasks

The CombineSegMasks node combines two or optionally three masks into a single mask to enhance the masking of different areas.

Inputs

  • image: A torch.Tensor that represents the input image.
  • mask1: A torch.Tensor that represents the first mask.
  • mask2: A torch.Tensor that represents the second mask.
  • mask3 (optional): A torch.Tensor that represents the third mask. The default value is None.

Outputs

  • combined_mask: A torch.Tensor that represents the combined mask.
  • image_out_hm: A torch.Tensor that represents the heatmap overlay of the combined mask on the input image.
  • image_out_bw: A torch.Tensor that represents the binary mask overlay of the combined mask on the input image.

Installation

To use these custom nodes in your ComfyUI project, follow these steps:

  1. Clone this repository or download the source code.
  2. Place the clipseg.py file into your custom_nodes directory.
  3. ???
  4. Enjoy the benefits.

Usage

Below is an example of the intended workflow. The json file for the example can be found inside the 'workflow' directory.

Requirements

  • PyTorch
  • CLIPSeg
  • OpenCV
  • numpy
  • matplotlib

Ensure that you have the required libraries installed in the virtual environment (venv) of ComfyUI.