• Sun. May 26th, 2024

How to use AI tool

How to use AI tools and How to use AI to make money

How to use Deforum in Stable Diffusion with AUTOMATIC1111

Byadmin

Jul 31, 2023 #Deforum, #txt2video

What is Deforum

Deforum is an extension for AUTOMATIC1111 that allows you to create beautiful AI generated videos. You might’ve seen these type of videos going viral on TikTok and Youtube

How to achieve video generation with Deforum

  • You enter a series of prompts, and the exact times you want the images generated by these prompts to appear.
  • Stable Diffusion generates these images and fills in the rest with transition images, creating a sequence that looks like a continuous video.
  • Meanwhile, you can adjust the position of the camera creating effects like zooming and rotating.

How to Install Deforum: A Comprehensive Guide

Deforum is a powerful extension that allows users to create captivating videos through Stable Diffusion technology. Here’s a detailed step-by-step guide on how to install Deforum on various platforms.

Installation Options

You have two choices for installing Deforum: installing the Stable Diffusion WebUI on your computer or running everything in the cloud with Google Colab.

Installing Deforum on Windows (NVIDIA GPU)

  1. Install WebUI: Follow the NVIDIA tutorial to install the WebUI, and you’ll have a folder named stable-diffusion-webui.
  2. Open Command Prompt: Search for “command prompt” and open it.
  3. Navigate to the Folder: Use cd FOLDER-NAME to navigate to stable-diffusion-webui.
  4. Download the Deforum Extension: Enter the following command:bashCopy codegit clone https://github.com/deforum-art/deforum-for-automatic1111-webui extensions/deforum
  5. Apply and Restart UI: Once the download is complete, go to the extensions tab and click “Apply and restart UI.”

Installing Deforum on Windows (AMD GPU)

Please refer to the AMD tutorial for installation.

Installing Deforum on Mac (Mac M1/M2)

  1. Install WebUI: Follow the Mac tutorial to install the WebUI.
  2. Open Terminal: Spotlight search for “terminal” and open it.
  3. Navigate to the Folder: Use cd FOLDER-NAME to navigate to stable-diffusion-webui.
  4. Download the Deforum Extension: Enter the following command:bashCopy codegit clone https://github.com/deforum-art/deforum-for-automatic1111-webui extensions/deforum
  5. Apply and Restart UI: After the installation is complete, go to the extensions tab and click “Apply and restart UI.”

Running Deforum on Google Colab

Choosing a Model

As with all things related to Stable Diffusion, the checkpoint model you choose will have the most significant impact on your results. Here’s a list of the most popular Stable Diffusion checkpoint models.

For the rest of this guide, we’ll either use the generic Stable Diffusion v1.5 model or the popular general-purpose model, Deliberate.

Don’t feel constrained by our choices: the examples will work with any model, but the results may vary greatly.

Checkpoint models come in either .ckpt or .safetensors format. All you have to do with these files is place them in the folder stable-diffusion-webui/models/Stable-diffusion.

Basic Usage

Setting Running Parameters: Similar to text generation, you can set parameters like sampler width and height. You can control the first frame by filling in the seed value of your generated image.

Keyframes: Set the animation mode, total frame number, etc. You can choose between 2D and 3D motion, border mode, maximum frame number, intensity table, CFG parameters, and motion parameters.

Prompt Words: Set the corresponding frame’s prompt words in JSON format.

{   
   “0”: “1gir. –neg nsfw, nude”,
   “30”: “1boy. –neg nsfw, nude”,
   “60”: “1dog. –neg nsfw, nude”,
   “90”: “1cat. –neg nsfw, nude”
}

Initialization: Control the beginning with a fixed image or video.

ControlNet: More precise control over the generated content.

Video Synthesis: Not yet tested.

Output: Usually, you only need to set FPS for smoother video effects.

Motion Settings

Motion settings are the most commonly used options in Deforum. You can create decent videos by simply changing them and the prompts. There are two most common animation modes:

  • 2D: Treats the image as 2D and performs various transformations like scaling and rotating to create a motion illusion.
  • 3D: Treats the image as a view of a 3D scene, allowing you to move the camera’s viewport in any 3D operation.

2D Motion Settings

2D Zoom: Enlarge or shrink the image.

zoom0.99-97
zoom 0:(0.99)
zoom1.01
zoom 0:(0.1.01)

2D Angle: Rotate the image clockwise or counterclockwise.

angle2.gif_256
2D angle: 2
angle-2.gif_256
2D angle: -2

2D Translation X/Y: Move the image horizontally or vertically.

translateX5_256
2D translation X: 5
translateX-5_256
2D translation X: -5
translateY5.gif_256
2D translation Y: 5
translateY-5.gif_256
2D translation Y: -5

2D Transform Center: Change the focus of scaling and/or rotating.

The default value is 0.5 for both X and Y, which is the center of the image. (X, Y) = (0, 0) is the top left corner, and (1, 1) is the bottom right corner. See the following diagram for other common locations.

image-116

You can specify values less than 0 or larger than 1. They will be outside of the image.

Below are two examples of zooming in at the top left corner (0, 0) and the bottom right (1, 1).

transformX0Y0_zoom1.01.gif_256
Transform Center (0,0) with zoom
transformX1Y1_zoom1.01.gif_256
Transform Center (1,1) with zoom

2D Perspective Flip: Perform 3D-like transformations.

You will need to select Enable perspective flip to enable these options.

image-117
theta12.gif_256
theta: 12
phi12.gif_256
phi: 12
gamma12.gif_256
gamma: 12

3D Motion Settings

3D Translation X

Translation X moves the camera sideways. A positive value moves the camera to the right. A negative value moves the camera to the left.

3D_translation_X2.mp4
3D translation X: 2

3D Translation Y

Translation Y moves the camera up and down. Using a positive value moves the camera up. A negative value moves the camera down.

3D_translation_Y2.mp4
3D translation Y: 2

3D Translation Z

Translation Z in 3D is similar to zoom in 2D motions.

3D_translation_Z2.mp4
3D translation Z: 2

3D rotation X

Rotation X rotates the camera about the X-axis.

3D_rotation_X2.mp4
3D rotation X: 2

3D rotation Y

Rotation Y rotates the camera about the Y-axis.

3D_rotation_Y2.mp4
3D rotation Y: 2

3D rotation Z

Rotation Z rotates the camera about the Z-axis.

3D_rotation_Z2.mp4
3D rotation Z: 2

Motion Timetable

The motion settings are placed in a table with frame numbers and corresponding change numbers. You always need an entry for the 0th frame, and you can have any number of entries.

Conclusion

The first step is to choose the large model you want to use and generate a few images to see the effect. Then set the running parameters, keyframe settings, prompt words, and output items. Creating satisfying animations requires careful study and experimentation. The motion settings in Deforum are essential, and you should master how they work and their effects.

Leave a Reply

Your email address will not be published. Required fields are marked *