Hi everyone! Arki from the
Stable Diffusion Discordhere!I've got two sections of guides available right now.If you check out the
Stable Diffusion Guides section below, I'll walk you through how to get started with fine-tuning your own Stable Diffusion models!I also have a HUGE Dreamer's Guide that has everything you could need to help you get started with Stable Diffusion, and beef up your prompting skills in no time!In the
DreamStudio Guides section below, I'll help you get started dreaming with Stability AI's
DreamStudio ASAP and along the way help you learn how to prompt efficiently and save credits too!If you'd like to support me and my work, please consider dropping by my
Ko-Fi.Let’s get started!
Stable Diffusion Guides
In this section I'll be primarily focusing on training guides for Stable Diffusion and how to take advantage of the various fine-tuning methods that currently exist.Note: Stable Diffusion depends primarily on Nvidia GPUs.
Dreamer's Guide to Stable Diffusion
If you're looking for resources, you've hit the jackpot! Below you'll find a plethora of information including links that'll help you start dreaming with Stable Diffusion in no time!If you haven't yet, be sure to come join the community over at the Stable Diffusion Discord.
Note: The community resources provided here are not endorsed, vetted, nor provided by Stability AI, and are purely community operated. Discretion is advised.
DreamStudio - DreamStudio homepage. New users get 200 free credits to spend on testing out the site. Perfect for people who don't want to or can't install it locally, and still want to start dreaming up awesome art!DreamStudio: Tips and Tricks - DreamStudio Guide covering tips and tricks on how to be smart with your credit spend, and advice on how to get DreamStudio to deliver the best results.
How do I use Stable Diffusion?
CMDR2 Stable Diffusion UI - Easy to set up Stable Diffusion UI for Windows and Linux. Works on CPU (albeit slowly) if you don't have a compatible GPU.Simple instructions for getting the CompVis repo of Stable Diffusion running on Windows. This is the initial release of the code that all of the recent open source forks have been developing off of. If you'd like to see where everything started from, this is it.A special thanks goes out to /u/nightkall for his awesome research into additional resources I added into this guide! His original thread can be found here.
Will it run on my machine?
Stable Diffusion requires a 4GB+ VRAM GPU to run locally. However, much beefier graphics cards (10, 20, 30 Series Nvidia Cards) will be necessary to generate high resolution or high step images.However, anyone can run it online through DreamStudio or hosting one of the local installs on their own GPU compute cloud server.Only Nvidia cards are officially supported.AMD support is available here unofficially.Apple M1 Chip support is available here unofficially.Intel based Macs currently do not work with Stable Diffusion.
Is the NSFW filter mandatory?
DreamStudio does not allow the disabling of their NSFW filter at the moment. Only local or remote installs may remove the filter currently.Google Colab Notebook with the filter disabled.Removal using the original CompVis RepoRemoval using HuggingFace Diffusers
Refer to the section below if you're looking to step up your prompting game.If you're using the CompVis repo, add the argument
--seed -1 to automatically generate a new seed each time you pass your prompt through.The maximum prompt length is roughly 320 characters (75 tokens).DreamStudio is currently in beta, if you encounter any credit related issues please submit this form.
Stable Diffusion is highly verbose. Prompt engineering is key when it comes to getting solid results.Spend time researching into the content of your prompt, find well known artists, research certain painting or design techniques, sculpt your prompt to represent the concepts you are looking for including descriptive details.Prompt engineering is how you learn to speak Stable Diffusion's language.As you play with it you'll learn more about how it interprets your words, and soon you'll be able to tweak your prompts to deliver what you're looking for.Don't be afraid to experiment!Check out the style guides listed as a part of this helpful community resource.Promptomania - Well designed and highly detailed prompt builder. Includes support for Stable Diffusion.SD Guide for Artists and Non-Artists - Highly detailed guide covering nearly every aspect of Stable Diffusion, goes into depth on prompt building, SD's various samplers and more.OpenArt - Search powered by OpenAI's CLIP model, provides prompt text with images. Includes the ability to add favorites.PromptHero - Search engine that offers all elements needed to recreate the results shown on the site, and the ability to favorite prompts. Also offers a Random wall of prompts to browse.Libraire - Search engine that offers all elements needed to recreate the results shown on the site.Lexica.art - Search engine built off of the public domain images generated during Stable Diffusion's initial Discord beta phase. Can be super helpful when developing prompts.Prompt Modifier Studies by parrot zoneKrea - Prompt keyword focused UI, offers the ability to favorite prompts.Phraser - Offers the ability to search for prompts via text search as well as image search.Img2Prompt - Image to Prompt generator optimized for SD (CLIP ViT-L/14) by Methexis Inc.Prompt Parrot Notebook - A notebook that can be fed a list of prompts to train on, and then generate new prompts based on the training data. Will additionally generate images based on the prompts. Requires Hugging Face token with write access to run.BLIP - Image analyzer that provides descriptions based on the image.AI Text Prompt Generator - Prompt generation tool that supports Stable Diffusion and offers different styles to generate prompts from.Dynamic Prompt generator - Prompt generator that offers drop down menus with various options to choose from to customize your prompt.The Ai Art - Gallery of prompt modifiers including both artists and stylistic keywords.
Introduction to Training - People (Cloud) Arki's introductory guide to getting started with training people into Stable Diffusion compatible models with a cloud GPU.
Emad's Sept. 9th 2022 Reddit AMAEasy Diffusion Notebook - One of the best notebooks available right now for generating with Stable Diffusion.A truly incredible sampler comparison written by /u/Any-Winter-4079Akashic Record - Huge collection of community resources that cover all aspects of Stable Diffusion usage.Awesome Stable Diffusion - Huge collection of awesome Stable Diffusion related software.A frequently updated thread of Stable Diffusion systems.How does Stable Diffusion work? (Technical Explanation)Pinegraph - Free generation website (with a daily limit of 50 uses) that offers both Stable Diffusion as well as Waifu Diffusion models.Phase.art - Free generation website that helps you build prompts by clicking on tokens, also offers a share option that includes all elements needed to recreate the results shown on the site.Artist Style Studies by parrot zoneCLIP Retrieval Search Engine query the laion-5b dataset which was used to help train Stable Diffusion.Query the LAION-Aesthetic Dataset - A subset of the LAION-5B dataset that has been estimated by a model trained on top of CLIP embeddings to be aesthetic.CLIP Interrogator - A helpful tool that will analyze an image that you feed into it, and give you a prompt based on how it interprets the image.CLIP Artist Evaluator Notebook - Allows you to provide a sample of art, then have CLIP evaluate the images and tell you who it thinks the artist is, as well as artists that might be good stylistic matches for use in your prompts.CLIP Ranked Artists Artists ranked by CLIP score, also provides artists with stylistic proximity as well as what artists CLIP detects from the input artist's work.urania.ai - Lists the top 500 artists represented in the Stable Diffusion 1.4 Model, ordered by the frequency of their representation in the model.Same Energy - Deep learning based visual search engine. Allows you to search for images that are the "same energy" an an uploaded image, or images already on the site.Avyn - Search engine with 9.6 million images generated by Stable Diffusion, also allows you to select an image and generate a new image based on its prompt. Now offers CLIP image searching, masked inpainting, as well as text-to-mask inpainting.Study on understanding Stable Diffusion w/ the Utah Teapot.Pharmapsychotic's Tools and Resources for AI Art
Introduction to Training - People (Cloud)
Hi everyone! I’ll be covering how to get started with training in this guide. The goal is to provide a cursory introduction into training people into Stable Diffusion compatible models.
The implementation that is going to allow us to do this is an open source adaptation of a technique originally developed by Google called Dreambooth, and adapted to function with Stable Diffusion originally in this repository on Github.
For this guide we’re going to be using my fork. I’ll be installing onto an NVIDIA 3090 cloud GPU from Runpod, and I’ve created a number of scripts included in this repo to help you get started.In the future I may be covering a local Windows installation as well if hardware allows (haha).
I’ve also created a
Runpod Template to help you get started as quickly as possible.
Note: In this guide I’ll be showing you the process that has worked for my personal model, others will have different ideas about how to get the best results out of training.Training's finer details are a rabbit hole that you can get lost in, and as such this guide is simply meant to get you running your first trained model ASAP.If you’d like to learn more about training, check out the
#finetune-corner in the
Stable Diffusion Discord or join a community-run discord for training enthusiasts
Before we get started you might find it educational to check out this short glossary of terms I’ve put together to help you wrap your head around some of the training terminology.
As this is an introduction and not a deep dive, I’ll mostly be covering how to get everything installed and how to quickly train and use your first model. However, my glossary should help those who’d like to learn a little bit more about what’s going on.
If you’d like to take a deeper dive, feel free to join the discord I linked above.
Let’s get started!
First, open up a web terminal after launching a pod with my
The web terminal will automatically start you out in the
Enter the command:
chmod +x DB-Install.sh && bash DB-Install.sh
This will run through all of the installation processes required to get your pod ready for training.
Feed it your direct model link when prompted, I recommend dropbox for this.Link Format:
fileidhere with the string of letters and numbers given to you when you get a shared dropbox file link.
The rest of the installation will take some time, so feel free to go do something else for a while.
Once the installation has been finished, you’ll be served a
.lt link, which you’ll want to open up in another tab.
During the previous step we installed a web-based file browser for Linux called Cloud Commander. Following this link will take us to it and it's what will allow us to add in all of the images required for training.
/home/user/Dreambooth-SD-optimized directory in Cloud Commander.
For this next part of the guide you are going to want 10 512x512 cropped images of yourself or another person prepared and then dragged and dropped into the
training_images directory.Ideally you want to have varied backgrounds in each of your training images, various angles, some neutral front pictures, some from each side, a few with various different expressions (happy, sad, angry, neutral) etc. for the best results.
Next you are going to want to place your regularization images in the
reg_images folder. These are meant to help prevent your trained concept from bleeding too much into related areas of the rest of the model. It’s not perfect yet, but it tries.
If you’re training a male individual, I would recommend using this set of reg images, originally from Joe Penna’s regularization images repo.If you’re training a female individual, I would recommend using this set of reg images, originally from Joe Penna’s regularization images repo.If you’re training an anime character, I would recommend using this set of reg images, originally from the Danbooru 2021 Cropped Portrait Dataset.
I recommend zipping up your reg images, and then dragging and dropping that zip into the destination folder. After it has finished uploading it, you can right click on the zip, unpack it, and then delete the zip file.
Next you are going to want to edit the
DB-Train.sh file and change
chris to a different name, ideally one associated with your training. Class word should stay as person as long as you are training a person.
batch_size will need to be changed to the number of reg images you are using.
Next you are going to want to go into the
ldm/data/ directory and edit
Change the name that you see under
chris, to the name of the concept you are trying to train in. This is referred to as your
identifier which is a special
token that references your trained data when used in a prompt.
Another configuration file can be found in
I’ve already adjusted this based on my best person training results, so there’s no need for you to edit it.
However, if in the future you would like to train for a longer amount of time with more or a different amount of steps, you’ll want to open this file and go all the way down to the bottom of it where
max_steps is and increase it to your desired number.
Next you are going to want to go back to your web terminal.
Enter the command:
source ~/anaconda3/bin/activate && conda activate ldm && chmod +x DB-Train.sh && bash DB-Train.sh
What we’re doing here is chaining multiple commands together.
The first command is activating conda in our web terminal, the second command is activating the ldm environment which we need for training, and the third and fourth commands are activating the training script.
This will start the training process, which will take about 30-40 minutes on a 3090 with the settings we’re using in this guide.
If you occasionally check the
/home/user/Dreambooth-SD-optimized/logs/projectfolder/images/train/ folder (and hit the little refresh button next to the path) as your model is training, you'll be able to see progress image generations as it tests your model every 200 steps.Double clicking on one will open a preview in Cloud Commander.For a person based model the
samples_scaled_gs images are the ones you're going to want to look at to see how your model is progressing.
Once the training has completed, you’ll see a folder called
checkpoints inside of your
Last.ckpt is going to be your fully trained model.
Optionally, you can use the
prune_ckpt.py script to cut down
last.ckpt to a smaller more manageable size. There has been no evidence that this perceptibly hurts the performance of your model, so it is safe to do.
To use it, go back to your web terminal and enter the command:
python3 prune_ckpt.py --ckpt /home/user/Dreambooth-SD-optimized/logs/projectfolder/checkpoints/last.ckpt replacing
projectfolder with the name of your actual project folder. This path can be easily copied using the clipboard button shown in the above screenshot, and then adding
last.ckpt to it at the end.
The custom model you've trained should now be able to be used in any local installation. Some installations may require you to rename your
model.ckpt and place it in a specific directory.Remember that when prompting with your custom model, the only way it knows how to use your trained concept is if you include
token class somewhere in the prompt. IE:
chris person in the style of hayao miyazakiPrompting with your trained model is just as important as prompting with the pretrained SD v1.4 (or later versions) model. In order to get the best results, experiment with your favorite prompts / negative prompts.Keep
CFG in the
7 - 11 range for the best results when prompting.Now you’re free to have as much fun with your custom model as you’d like!
DreamStudio: Tips and Tricks! (How to save credits!)
DreamStudio is run by Stability A.I., the company responsible for managing and developing Stable Diffusion alongside their industry and educational partners.
The first important thing to mention, is that DreamStudio is an entirely optional service, and only one of many ways in which you can access Stable Diffusion.
Stable Diffusion has recently gone open source, which means that if you have a powerful enough GPU you can run it on your own hardware for free!Check out the Dreamer's Guide for more information on that.
That aside, DreamStudio is currently in its lite form, which means that it is early days yet for the app and you may run into some bugs. Additionally, certain aspects of the app may not be quite as clear as you would like them to be yet.
If you encounter any bugs, please be sure to submit your issue through
It can be a little bit confusing the first time you open DreamStudio, so I’m aiming to clear up as much of that as I can and help enable you to have as much fun as possible!
All users get
100 Free Credits when they first sign up to DreamStudio. By purchasing additional credits, you can help support Stability A.I. as they continue to push the boundaries of creative expression with cutting edge technology.
To understand how DreamStudio consumes your credits, check out the diagram below.
As you can see here, credits are consumed at a higher rate as you increase the amount of steps and the total resolution of your generation. Fortunately however, there are ways to be careful about your credit usage, and you’ll almost never have to go into the extreme range of
150 Steps or
1024 x 1024.Let’s get into it!
Firstly, it’s important to understand the content of your prompt when starting to use DreamStudio. What is it that you are actually looking for? Diligent research will go a long way when you are prompting.
Look into things like:
Artists whose styles you like.
Names of specific artistic genres or disciplines.
Lighting and rendering terms etc.
Building a descriptive prompt goes a long way, and knowing how to tell SD what you’re looking for will always pay off in the end.
Credit Saving & Generation Setting Tips
An easy way to be strategic about your credit spend while working inside of DreamStudio is to first test out your prompts at low steps and low resolution settings. When generating, you’ll see a counter at the top of the page that shows how many credits your current generation will cost.
It's easy to save a boatload of credits this way, while still being able to understand how SD is interpreting your prompt as you develop it further.
I recommend doing all of your testing at
512 x 512 and
50 Steps or lower, as this was what Stable Diffusion was trained on, and will give you the greatest chance of understanding how it will interpret your prompts.
If you get something that looks like it has potential in this testing stage, copy down the prompt and the seed that you used for it. Later on regen the same prompt and seed but at higher dimensions and higher steps if you desire.
Changing the resolution of your image, will lead to a different image, but by starting with lower settings first you can get an understanding of how SD is interpreting your prompt which can help in refining it.
If you keep the same resolution settings however, and lock the seed, you can see how your current generation would be affected at higher steps or higher
CFG Scale, which is tremendously useful.
Higher step count is
NOT always equivalent to higher quality. Past a certain point (that will continue to change as the model evolves) all you are doing is burning through credits for diminishing returns.IE: I’ve almost never had to approach
100 Steps or above. Additionally, playing with
CFG Scale in concert with your step count has a huge impact on your final result.
CFG Scale does not affect credit consumption, so experimenting with that can also be an interesting way to learn more about how SD sees your prompt.
According to the site’s description
CFG Scale affects
“how much the image will be like your prompt. Higher values keep your image closer to your prompt.” I tend to enjoy working in a range between the default of
7.5 up to about
14 depending on the content and the sampler.
Seed Editing can be a lot of fun and lead to a lot of interesting results! DreamStudio offers a lot of interesting settings that can be modified by sliders, but did you know that if you manually edit the numbers in your seed you can discover wildly cool different results?
Depending on how many numbers you change in your seed and how far apart your changes are from your original generation, you can either gently affect your generation or affect it in a much greater manner.
SAVE 👏 YOUR 👏 IMAGES 👏!DreamStudio currently stores your historical generations in your browser’s cache, which means that they could disappear if you aren’t careful. I recommend downloading all of your generations as soon as they come up, as well as recording your prompts, seeds, and settings in a text document. You can always delete them later if you’d like to.
Redreaming images from the history panel costs credits, since you’re asking DreamStudio to essentially make it from scratch again, which costs GPU compute.
If you end up getting a blurred result to your prompt, don’t worry! That’s just the NSFW filter doing its job. It’s a little overactive right now as it's being trained to be more accurate, but you won’t lose any credits for blurred images.
Don’t be afraid to upsample your generations! If you generate an image, for example, that is
768 x 512 and you wish you could make it a desktop wallpaper or large enough to be printed, upsampling is your friend!
Installation Troubleshooting Tips
Occasionally you may run into some trouble when trying to install Stable Diffusion, especially if you are coming to this guide after trying to install it yourself first without following the steps I’ve laid out here.
More often than not, the number one way to get Stable Diffusion running properly on your computer is by clearing out your existing
conda environments, and following this guide from the top again (or for the first time).
In order to clear out your existing environments, open up your
Anaconda3 / Miniconda3 Prompt and enter:
conda deactivate followed by:
conda info --envs to verify the names of the environments you have installed.
Next, you’ll want to enter
conda remove --name myenv --all where you replace
myenv with the name of the environment you’re looking to remove.
yes when prompted to confirm the removal of your environment.
This won’t remove the folder that you have already downloaded Stable Diffusion into, but it will allow you to start this guide fresh if your environment was giving you errors when trying to get it running.
My advice however would be to save your
model.ckpt file somewhere safe, clear out your environment, delete the folders you downloaded and then run through the entire installation guide from scratch again to ensure no problems arise.
Anaconda3 / Miniconda3 should install everything that you need in order for each Stable Diffusion repo to work when running the
conda env create -f environment.yaml command during that part of the installation process.