What is Deforum
Deforum is an extension for AUTOMATIC1111 that allows you to create beautiful AI generated videos. You might’ve seen these type of videos going viral on TikTok and Youtube
How to achieve video generation with Deforum
- You enter a series of prompts, and the exact times you want the images generated by these prompts to appear.
- Stable Diffusion generates these images and fills in the rest with transition images, creating a sequence that looks like a continuous video.
- Meanwhile, you can adjust the position of the camera creating effects like zooming and rotating.
How to Install Deforum: A Comprehensive Guide
Deforum is a powerful extension that allows users to create captivating videos through Stable Diffusion technology. Here’s a detailed step-by-step guide on how to install Deforum on various platforms.
Installation Options
You have two choices for installing Deforum: installing the Stable Diffusion WebUI on your computer or running everything in the cloud with Google Colab.
Installing Deforum on Windows (NVIDIA GPU)
- Install WebUI: Follow the NVIDIA tutorial to install the WebUI, and you’ll have a folder named
stable-diffusion-webui
. - Open Command Prompt: Search for “command prompt” and open it.
- Navigate to the Folder: Use
cd FOLDER-NAME
to navigate tostable-diffusion-webui
. - Download the Deforum Extension: Enter the following command:bashCopy code
git clone https://github.com/deforum-art/deforum-for-automatic1111-webui extensions/deforum
- Apply and Restart UI: Once the download is complete, go to the extensions tab and click “Apply and restart UI.”
Installing Deforum on Windows (AMD GPU)
Please refer to the AMD tutorial for installation.
Installing Deforum on Mac (Mac M1/M2)
- Install WebUI: Follow the Mac tutorial to install the WebUI.
- Open Terminal: Spotlight search for “terminal” and open it.
- Navigate to the Folder: Use
cd FOLDER-NAME
to navigate tostable-diffusion-webui
. - Download the Deforum Extension: Enter the following command:bashCopy code
git clone https://github.com/deforum-art/deforum-for-automatic1111-webui extensions/deforum
- Apply and Restart UI: After the installation is complete, go to the extensions tab and click “Apply and restart UI.”
Running Deforum on Google Colab
Choosing a Model
As with all things related to Stable Diffusion, the checkpoint model you choose will have the most significant impact on your results. Here’s a list of the most popular Stable Diffusion checkpoint models.
For the rest of this guide, we’ll either use the generic Stable Diffusion v1.5 model or the popular general-purpose model, Deliberate.
Don’t feel constrained by our choices: the examples will work with any model, but the results may vary greatly.
Checkpoint models come in either .ckpt
or .safetensors
format. All you have to do with these files is place them in the folder stable-diffusion-webui/models/Stable-diffusion
.
Basic Usage
Setting Running Parameters: Similar to text generation, you can set parameters like sampler width and height. You can control the first frame by filling in the seed value of your generated image.
![](https://howtouseaitool.com/wp-content/uploads/2023/07/3f2928949b2df1ce922ed282bee89108_640_wx_fmtpngwxfrom5wx_lazy1wx_co1-1024x369.png)
Keyframes: Set the animation mode, total frame number, etc. You can choose between 2D and 3D motion, border mode, maximum frame number, intensity table, CFG parameters, and motion parameters.
![](https://howtouseaitool.com/wp-content/uploads/2023/07/05bcea3da87b59f00e7f401ca33bfd48_640_wx_fmtpngwxfrom5wx_lazy1wx_co1-1024x926.png)
Prompt Words: Set the corresponding frame’s prompt words in JSON format.
{
“0”: “1gir. –neg nsfw, nude”,
“30”: “1boy. –neg nsfw, nude”,
“60”: “1dog. –neg nsfw, nude”,
“90”: “1cat. –neg nsfw, nude”
}
![](https://howtouseaitool.com/wp-content/uploads/2023/07/c20105172b7626c5c60143afef588294_640_wx_fmtpngwxfrom5wx_lazy1wx_co1-1024x675.png)
Initialization: Control the beginning with a fixed image or video.
![](https://howtouseaitool.com/wp-content/uploads/2023/07/d155be1a46c6a626879e2f28b73bcacb_640_wx_fmtpngwxfrom5wx_lazy1wx_co1-1024x470.png)
ControlNet: More precise control over the generated content.
![](https://howtouseaitool.com/wp-content/uploads/2023/07/154cc2551fbaed1b5ee1f03baa996aba_640_wx_fmtpngwxfrom5wx_lazy1wx_co1-1024x666.png)
Video Synthesis: Not yet tested.
![](https://howtouseaitool.com/wp-content/uploads/2023/07/bbfabf626b153c7619c5116b5d87e2c6_640_wx_fmtpngwxfrom5wx_lazy1wx_co1.png)
Output: Usually, you only need to set FPS for smoother video effects.
![](https://howtouseaitool.com/wp-content/uploads/2023/07/08dda6e6cdb19939968be2f4d9c935c7_640_wx_fmtpngwxfrom5wx_lazy1wx_co1-1024x482.png)
Motion Settings
Motion settings are the most commonly used options in Deforum. You can create decent videos by simply changing them and the prompts. There are two most common animation modes:
- 2D: Treats the image as 2D and performs various transformations like scaling and rotating to create a motion illusion.
- 3D: Treats the image as a view of a 3D scene, allowing you to move the camera’s viewport in any 3D operation.
2D Motion Settings
2D Zoom: Enlarge or shrink the image.
![zoom0.99-97 zoom0.99-97](https://howtouseaitool.com/wp-content/uploads/2023/10/zoom0.99-97.gif)
![zoom1.01 zoom1.01](https://howtouseaitool.com/wp-content/uploads/2023/10/zoom1.01.gif)
2D Angle: Rotate the image clockwise or counterclockwise.
![angle2.gif_256 angle2.gif_256](https://howtouseaitool.com/wp-content/uploads/2023/10/angle2.gif_256.gif)
![angle-2.gif_256 angle-2.gif_256](https://howtouseaitool.com/wp-content/uploads/2023/10/angle-2.gif_256.gif)
2D Translation X/Y: Move the image horizontally or vertically.
![translateX5_256 translateX5_256](https://howtouseaitool.com/wp-content/uploads/2023/10/translateX5_256.gif)
![translateX-5_256 translateX-5_256](https://howtouseaitool.com/wp-content/uploads/2023/10/translateX-5_256.gif)
![translateY5.gif_256 translateY5.gif_256](https://howtouseaitool.com/wp-content/uploads/2023/10/translateY5.gif_256.gif)
![translateY-5.gif_256 translateY-5.gif_256](https://howtouseaitool.com/wp-content/uploads/2023/10/translateY-5.gif_256.gif)
2D Transform Center: Change the focus of scaling and/or rotating.
The default value is 0.5 for both X and Y, which is the center of the image. (X, Y) = (0, 0) is the top left corner, and (1, 1) is the bottom right corner. See the following diagram for other common locations.
![image-116 image-116](https://howtouseaitool.com/wp-content/uploads/2023/10/image-116.png)
You can specify values less than 0 or larger than 1. They will be outside of the image.
Below are two examples of zooming in at the top left corner (0, 0) and the bottom right (1, 1).
![transformX0Y0_zoom1.01.gif_256 transformX0Y0_zoom1.01.gif_256](https://howtouseaitool.com/wp-content/uploads/2023/10/transformX0Y0_zoom1.01.gif_256.gif)
![transformX1Y1_zoom1.01.gif_256 transformX1Y1_zoom1.01.gif_256](https://howtouseaitool.com/wp-content/uploads/2023/10/transformX1Y1_zoom1.01.gif_256.gif)
2D Perspective Flip: Perform 3D-like transformations.
You will need to select Enable perspective flip to enable these options.
![image-117 image-117](https://howtouseaitool.com/wp-content/uploads/2023/10/image-117.png)
![theta12.gif_256 theta12.gif_256](https://howtouseaitool.com/wp-content/uploads/2023/10/theta12.gif_256.gif)
![phi12.gif_256 phi12.gif_256](https://howtouseaitool.com/wp-content/uploads/2023/10/phi12.gif_256.gif)
![gamma12.gif_256 gamma12.gif_256](https://howtouseaitool.com/wp-content/uploads/2023/10/gamma12.gif_256.gif)
3D Motion Settings
3D Translation X
Translation X moves the camera sideways. A positive value moves the camera to the right. A negative value moves the camera to the left.
![3D_translation_X2.mp4 3D_translation_X2.mp4](https://howtouseaitool.com/wp-content/uploads/2023/10/3D_translation_X2.mp4.gif)
3D Translation Y
Translation Y moves the camera up and down. Using a positive value moves the camera up. A negative value moves the camera down.
![3D_translation_Y2.mp4 3D_translation_Y2.mp4](https://howtouseaitool.com/wp-content/uploads/2023/10/3D_translation_Y2.mp4.gif)
3D Translation Z
Translation Z in 3D is similar to zoom in 2D motions.
![3D_translation_Z2.mp4 3D_translation_Z2.mp4](https://howtouseaitool.com/wp-content/uploads/2023/10/3D_translation_Z2.mp4.gif)
3D rotation X
Rotation X rotates the camera about the X-axis.
![3D_rotation_X2.mp4 3D_rotation_X2.mp4](https://howtouseaitool.com/wp-content/uploads/2023/10/3D_rotation_X2.mp4.gif)
3D rotation Y
Rotation Y rotates the camera about the Y-axis.
![3D_rotation_Y2.mp4 3D_rotation_Y2.mp4](https://howtouseaitool.com/wp-content/uploads/2023/10/3D_rotation_Y2.mp4.gif)
3D rotation Z
Rotation Z rotates the camera about the Z-axis.
![3D_rotation_Z2.mp4 3D_rotation_Z2.mp4](https://howtouseaitool.com/wp-content/uploads/2023/10/3D_rotation_Z2.mp4.gif)
Motion Timetable
The motion settings are placed in a table with frame numbers and corresponding change numbers. You always need an entry for the 0th frame, and you can have any number of entries.
Conclusion
The first step is to choose the large model you want to use and generate a few images to see the effect. Then set the running parameters, keyframe settings, prompt words, and output items. Creating satisfying animations requires careful study and experimentation. The motion settings in Deforum are essential, and you should master how they work and their effects.