ComfyUI InfoComfyUI Info

ComfyUI_Cutoff

ComfyUI_Cutoff is an implementation of the cutoff script in ComfyUI. It allows users to limit the influence of specific words in a prompt to certain regions. By masking tokens and calculating vector differences, it can isolate color and other relations in the embedding. It introduces four nodes: Cutoff BasePrompt, Cutoff Set Region, Cutoff Regions To Conditioning, and Cutoff Regions To Conditioning (ADV). The nodes can be found under `conditioning>cutoff`, and while they work with SDXL, the effect might not be ideal for version 0.9.

BlenderNeko

386 stars
GitHub

Description

Cutoff for ComfyUI

screenshot of workflow

What is Cutoff?

Cutoff is a script or extension for the Automatic1111 webui. It allows users to limit the influence of certain attributes on specified parts of the prompt. For example, when the prompt is a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt, Cutoff enables you to specify that the word "blue" applies to the hair and not the shoes, and "green" to the tie and not the skirt. This is an implementation of Cutoff in the form of 3 nodes that can be used in ComfyUI. ComfyUI_Cutoff brings this useful functionality to the ComfyUI environment.

How Does This Work?

When you give Stable Diffusion some text, the text is tokenized, and CLIP creates a vector (embedding) for each token in the text. For instance, if the prompt contains "blue hair, yellow eyes", some of the vectors from CLIP will correspond to "blue hair", and some to "yellow eyes". When CLIP does this, it tries to consider the context of the whole sentence. However, CLIP isn't always good at determining that the "blue" in "blue hair" should only modify the noun "hair" and not the noun "eyes" later in the sentence.

So, how do we handle this? We can mask out the tokens corresponding to "blue" and ask CLIP to create another embedding. In this new embedding, the vectors for "yellow eyes" are not affected by "blue" because "blue" was not part of the tokens. If we then find the difference between the original vectors and these new vectors, we get a direction. We can use this direction to make the eyes more affected by "yellow" and less by "blue". If we do this for all color - related words in the text, we can reach an embedding where these relationships are more distinct. Of course, this effect is not limited to just colors. ComfyUI_Cutoff utilizes this mechanism to enhance the precision of prompts in ComfyUI.

ComfyUI Nodes

To achieve all this, the following 4 nodes are introduced in ComfyUI_Cutoff:

Cutoff BasePrompt

This node takes the full original prompt.

Cutoff Set Region

This node sets an "area of influence" for specific target words and has the following inputs:

  • region_text: Defines the set of tokens that the target words should affect. This should be part of the original prompt. You can define multiple regions in a single CLIPSetRegion node by writing each region on a new line.
  • target_text: Defines the set of tokens that will be masked off (i.e., the tokens we want to limit to the region). This is a list of words separated by spaces. If you want to match a sequence of words, use underscores instead of spaces, e.g., "a_series_of_connected_tokens". If you want to match a word that actually contains underscores, escape the underscore, e.g., "the_target_tokens". You can target textual inversion embeddings using the default syntax, but note that any underscores in the embedding name must be escaped in this input field.
  • weight: Determines how far to move in the direction of the isolated vector.

Cutoff Regions To Conditioning

This node converts the base prompt and regions into an actual conditioning for use in the rest of ComfyUI and has the following inputs:

  • mask_token: The token used for masking. If left blank, it defaults to the <endoftext> token. If the string converts to multiple tokens, it will give a warning in the console and only use the first token in the list.
  • strict_mask: When set to 0.0, the specified target tokens will not affect the other specified areas but will affect anything outside those areas. When set to 1.0, the specified target tokens will only affect their own region.
  • start_from_masked: When set to 0.0, the starting point for the adjustment is the original prompt. When set to 1.0, the starting point is the completely masked - off prompt. Note that when all region weights are 1.0, there is no difference between the two settings.

Cutoff Regions To Conditioning (ADV)

This node provides the same functionality as the above node but also offers options on how to interpret prompt weighting. More information about these settings can be found here. You can find these nodes under conditioning>cutoff in ComfyUI_Cutoff.

SDXL

The nodes in ComfyUI_Cutoff will not cause errors when used with SDXL, but at least for version 0.9, it doesn't seem to work very well.

Finally, here are some example images that you can load into ComfyUI using ComfyUI_Cutoff:

first example generation of a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt using cutoff first example generation of a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt using cutoff first example generation of a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt using cutoff first example generation of a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt using cutoff