How to install ComfyUI and How to use ComfyUI for Stable Diffusion

What is ComfyUI ?

ComfyUI: A Node-Based Interface for Stable Diffusion

ComfyUI, developed in January 2023 by Comfyanonymous, is a unique user interface tailored for Stable Diffusion. The main drive behind its creation was Comfyanonymous’s curiosity and desire to delve deeper into the workings of Stable Diffusion.

StabilityAI, the genius team behind Stable Diffusion, didn’t just admire ComfyUI from afar. They saw its potential and started using it for their internal testing. Impressed by Comfyanonymous’s expertise, they even brought him on board to enhance their internal tools.

If you’ve ever dabbled in 3D Design or VFX, you might be familiar with node-based interfaces. Tools like Maya and Blender3D are prime examples. In such interfaces, each node is a piece of code waiting to be executed. They have inputs, where you feed in values, and outputs, where you get the results.

With ComfyUI, it’s all about dragging, dropping, and connecting. Using a mouse, you can create new nodes, tweak their parameters, and link them up. Each node in ComfyUI represents a step in the Stable Diffusion process. By connecting them in the right sequence, you’re all set to run Stable Diffusion.

ComfyUI vs. AUTOMATIC1111 WebUI

When diving into the world of ComfyUI, many newcomers, previously accustomed to simpler interfaces like AUTOMATIC1111, InvokeAI, or SD.Next, often find themselves pondering a common question:

“Why opt for such a complex tool when the end goal seems identical?”

A meme by u/unlimitedduck humorously captures this sentiment. But the reality is, while the basic functionalities might overlap, ComfyUI offers a realm of possibilities that its counterparts can’t match.

1. Extreme Configurability

With ComfyUI, you’re not just limited to basic operations. Want a sneak peek of your images at any stage of the generation? ComfyUI has got you covered. Feel the need to run dual generations simultaneously for a side-by-side comparison of sampling methods? ComfyUI makes it a breeze. This versatility is precisely why many prefer ComfyUI.

2. Multitasking Made Easy

Imagine a routine process for your images: generating an image, sending the result to img2img, and finally upscaling it. In AUTOMATIC1111, this would mean manual execution of each step. But with ComfyUI, all these actions are condensed into a single click. This feature particularly shines with SDXL v1.0, which employs a 2-step process involving a base model for generating noisy latents and a specialized refiner model for denoising, enhancing image sharpness and detail.

3. Unparalleled Performance and Speed

ComfyUI’s high degree of configurability means that its generations can be fine-tuned in ways that AUTOMATIC1111 can’t fathom. The result? A significant boost in speed. Users have reported ComfyUI churning out results 3-5 times faster than AUTOMATIC1111.

4. A Learning Experience

ComfyUI isn’t just a tool; it’s a learning platform. Its node setup mirrors the intricate workings of the diffusion process. So, as you master ComfyUI, you’re simultaneously unraveling the mysteries of how Stable Diffusion operates.

How to install ComfyUI？ComfyUI Installation Guide

Quick-Install (Windows, NVIDIA)

For those using NVIDIA GPUs or CPUs on Windows, there’s a portable standalone version of ComfyUI. However, be warned, CPU generation can be quite slow, so it’s almost like an NVIDIA-exclusive installer.

Download: Click on the provided Download Link to start your download.
Extraction: Once downloaded, extract the .zip file using 7-Zip.
Access: Post extraction, you’ll find a folder named ComfyUI_windows_portable which contains the ComfyUI folder.
Execution:
- For NVIDIA GPU: Double-click on run_nvidia_gpu.bat.
- For CPU: Double-click on run_cpu.bat.

Clone from Github (Windows, Linux)

NVIDIA GPU

Open Command Prompt on Windows (Search for “cmd”) or Terminal on Linux (Ctrl+Alt+T).
Execute the following commands sequentially:

git clone https://github.com/comfyanonymous/ComfyUI cd ComfyUI pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu118 xformers -r requirements.txt

Launch ComfyUI:

python main.py

Troubleshooting: If you encounter the “Torch not compiled with CUDA enabled” error, uninstall torch with pip uninstall torch (press “y” when prompted) and reinstall using the previous command.

AMD GPU

Open Command Prompt on Windows (Search for “cmd”) or Terminal on Linux (Ctrl+Alt+T).
Execute the following commands:

git clone https://github.com/comfyanonymous/ComfyUI cd ComfyUI python -m pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/rocm5.4.2 -r requirements.txt

If your GPU isn’t officially supported by ROCm (like 6700 XT), try:

HSA_OVERRIDE_GFX_VERSION=10.3.0 python main.py

Launch ComfyUI:

python main.py

Clone from Github (Mac)

For Apple Mac silicon users (M1 or M2) with a recent macOS version:

Install pytorch as per the Accelerated PyTorch training on Mac guide.
Open the terminal and clone the repository:

git clone https://github.com/comfyanonymous/ComfyUI

Install ComfyUI dependencies:

cd ComfyUI pip install -r requirements.txt

Note: If you have another Stable Diffusion UI, you might be able to reuse the dependencies.

Launch ComfyUI:

python main.py --force-fp16

Remember, --force-fp16 will only function if you’ve installed the latest pytorch nightly.

How to Updating ComfyUI

You can only update ComfyUI if you cloned it from Github (you did not get the Quick-Install standalone version).

To update ComfyUI navigate to the ComfyUI folder in your Command Prompt/Terminal and type:

git pull

ComfyUI User Interface Guide

1. Nodes

Adding a Node: Right-click on a blank space -> Select “Add Node”.

Viewing All Nodes: Double-click on a blank space to see a list of all nodes along with a search bar.

Selecting Multiple Nodes: Use CTRL + drag to select multiple nodes. With SHIFT + drag, you can move them simultaneously.

Changing Node Color: To help you stay organized, you can change the color of the nodes. Right-click -> Select “Color” -> Choose the desired color.

2. Connecting Nodes

Establishing a Connection: If you drag an input and release it onto a blank space, you’ll get a list of compatible nodes.

Compatibility: Inputs and outputs are only compatible if they share the same color. Note that I can connect the purple input to the purple output, but not to any other nodes.

3. Execution

Queue Prompt: When you click on “Queue Prompt”, the workflow will pass through the nodes in the order they’re connected, starting from “Loaders” which have no inputs, only outputs.

Missing Inputs: If any nodes are lacking inputs, you won’t be able to execute the prompt.

ComfyUI Nodes Demystified

Understanding the Workflow

To truly grasp the intricacies of ComfyUI nodes, it’s essential to first get a handle on how Stable Diffusion operates. Let’s dive into the default workflow.

If you’ve deviated from the default workflow or have been experimenting with the interface, simply click on “Load Default” located in the right sidebar.

1. Load Checkpoint Node

The checkpoint models (either .safetensors or .ckpt) utilized for image generation consist of three primary components:

CLIP model: Transforms text into a format that the Unet can comprehend.
Unet: Executes the “diffusion” process, which is the sequential image processing we term as generation.
VAE: Decodes the image from latent space to pixel space. It’s also employed to encode a standard image from pixel space to latent space for img2img tasks.

In the ComfyUI workflow, this is symbolized by the Load Checkpoint node and its three outputs, with MODEL referring to the Unet.

2. CLIP Text Encode Node

The CLIP output from the Load Checkpoint node is linked to the CLIP Text Encode nodes.

The CLIP model’s primary function is to transform text into a format (numeric representation) that the Unet can interpret, known as embeddings. The CLIP Text Encode nodes manage this encoding process, converting your prompts (both positive and negative) into embeddings, which are then passed to the next node, the KSampler.

3. KSampler

Image generation in Stable Diffusion is achieved through a method termed “sampling”, which in ComfyUI, is managed by the KSampler node. This node is responsible for the actual “generation”, and you’ll observe that the KSampler consumes the most time during prompt queuing.

The KSampler requires several inputs:

model: The MODEL output (Unet) from the Load Checkpoint node.
positive: The positive prompt encoded by the CLIP model.
negative: The negative prompt, similarly encoded.
latent_image: An image in latent space, provided by the Empty Latent Image node for txt2img tasks.

4. VAE

The VAEDecode node is tasked with two primary inputs:

Latent space image: The denoised image from the KSampler.
VAE model: From the checkpoint or a your custom one.

The VAE’s role is to convert an image from latent space to pixel space. This final pixel image is then passed to the Save Image node, which displays the image and offers a download option.

In essence, the default workflow is the most straightforward one you’ll encounter in ComfyUI, designed to provide a seamless experience for users.

Downloading Models Guide

1. Checkpoints

To run SDXL, you’ll initially need to download the checkpoint models. Depending on your requirements, you might need additional models later on.

Location: Store the checkpoints in the ComfyUI/models/checkpoints directory.

2. VAE (Variational Autoencoder)

Location: Store the VAEs in the ComfyUI/models/vae directory.

Fixed SDXL 1.0 VAE

3. LoRAs (Local Randomizations)

Location: Store the LoRAs in the ComfyUI/models/loras directory.

SDXL Offset Noise LoRA

4. Upscaler

Depending on your workflow, you might need upscalers. Some workflows might not include them, while others might necessitate them.

Location: Store the upscalers in the ComfyUI/models/upscaler directory.

5. Sharing Models Between Different UIs

If you have another UI, like AUTOMATIC1111, and you don’t want to duplicate storage by copying all your checkpoint models into both the AUTOMATIC1111 and ComfyUI directories, there’s a solution.

Configuration File: Check the Config file to set the search paths for models. In the standalone windows build, you can locate this file in the ComfyUI directory. Rename this file to extra_model_paths.yaml and modify it using your preferred text editor.

Recommended ComfyUI Workflows Guide

Swapping Workflows with Ease

One of ComfyUI’s standout features is the simplicity with which users can download and interchange workflows. The official ComfyUI repository offers a variety of example workflows for users to explore.

Loading a Workflow: To integrate a workflow, click on the ‘Load’ button located on the right sidebar. Then, select the desired workflow .json file. Some workflows might necessitate cloning the repository into the ComfyUI/custom_nodes directory and subsequently restarting ComfyUI.

Popular Workflows in the Stable Diffusion Community

Sytan’s SDXL Workflow

Documentation
A highly favored workflow tailored for SDXL. It boasts an upscaling feature.

Searge SDXL Workflow

Documentation
This workflow offers three distinct operating modes: text-to-image, image-to-image, and inpainting. Users can effortlessly switch between these modes using an option.

Installation Recommendation for Searge SDXL Workflow:

Navigate to the ComfyUI/custom_nodes/ directory on your system.
Open a command line window within the custom_nodes directory.
Execute the command: git clone https://github.com/SeargeDP/SeargeSDXL.git
Restart ComfyUI to complete the integration.