19:36:11-887145 INFO Version: v22.6.2
19:36:11-887145 INFO nVidia toolkit detected
19:36:38-226469 INFO Torch 2.1.2+cu118
19:36:38-242091 INFO Torch backend: nVidia CUDA 11.8 cuDNN 8700
19:36:38-242091 INFO Torch detected GPU: NVIDIA GeForce RTX 4080 SUPER VRAM 16376 Arch (8, 9) Cores 80
19:36:38-242091 INFO Verifying modules installation status from requirements_windows_torch2.txt...
19:36:38-242091 INFO Installing package: torch==2.1.2+cu118 torchvision==0.16.2+cu118 torchaudio==2.1.2+cu118
--index-url https://download.pytorch.org/whl/cu118
19:36:42-925923 INFO Installing package: xformers==0.0.23.post1+cu118 --index-url
https://download.pytorch.org/whl/cu118
19:36:45-476670 INFO Verifying modules installation status from requirements.txt...
19:36:50-010949 INFO headless: False
19:36:50-010949 INFO Load CSS...
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
19:37:30-698481 INFO Loading config...
19:39:22-153649 INFO Save...
19:40:02-327279 INFO Start training LoRA Standard ...
19:40:02-328279 INFO Checking for duplicate image filenames in training data directory...
19:40:02-334280 INFO Valid image folder names found in: D:/sdlanuncher/LoRA_test/image
19:40:02-336279 INFO Folder 100_face: 38 images found
19:40:02-337279 INFO Folder 100_face: 3800 steps
19:40:02-338279 INFO Total steps: 3800
19:40:02-339281 INFO Train batch size: 2
19:40:02-339281 INFO Gradient accumulation steps: 1
19:40:02-341279 INFO Epoch: 4
19:40:02-342279 INFO Regulatization factor: 1
19:40:02-343279 INFO max_train_steps (3800 / 2 / 1 * 4 * 1) = 7600
19:40:02-344279 INFO stop_text_encoder_training = 0
19:40:02-345279 INFO lr_warmup_steps = 760
19:40:02-346279 WARNING Here is the trainer command as a reference. It will not be executed:
accelerate launch --num_cpu_threads_per_process=2 "./train_network.py" --bucket_no_upscale --bucket_reso_steps=64 --cache_latents --caption_extension=".txt" --clip_skip=2 --enable_bucket --min_bucket_reso=256 --max_bucket_reso=2048 --keep_tokens="1" --learning_rate="0.0001" --logging_dir="D:/sdlanuncher/LoRA_test/log" --lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles="4" --lr_warmup_steps="760" --max_data_loader_n_workers="1" --max_grad_norm="1" --resolution="512,512" --max_train_steps="7600" --mixed_precision="fp16" --network_alpha="128" --network_dim=128 --network_module=networks.lora --optimizer_type="AdamW8bit" --output_dir="D:/sdlanuncher/LoRA_test/model" --output_name="mengwenwen" --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" --save_every_n_epochs="1" --save_model_as=safetensors --save_precision="fp16" --seed="1234" --text_encoder_lr=5e-05 --train_batch_size="2" --train_data_dir="D:/sdlanuncher/LoRA_test/image" --unet_lr=0.0001 --xformers
19:40:10-471477 INFO Start training LoRA Standard ...
19:40:13-659082 INFO Checking for duplicate image filenames in training data directory...
19:40:13-659082 INFO Valid image folder names found in: D:/sdlanuncher/LoRA_test/image
19:40:13-659082 INFO Folder 100_face: 38 images found
19:40:13-659082 INFO Folder 100_face: 3800 steps
19:40:13-659082 INFO Total steps: 3800
19:40:13-705962 INFO Train batch size: 2
19:40:13-705962 INFO Gradient accumulation steps: 1
19:40:13-705962 INFO Epoch: 4
19:40:13-705962 INFO Regulatization factor: 1
19:40:13-705962 INFO max_train_steps (3800 / 2 / 1 * 4 * 1) = 7600
19:40:13-705962 INFO stop_text_encoder_training = 0
19:40:13-705962 INFO lr_warmup_steps = 760
19:40:13-705962 INFO Saving training config to D:/sdlanuncher/LoRA_test/model\mengwenwen_20240225-194013.json...
19:40:13-705962 INFO accelerate launch --num_cpu_threads_per_process=2 "./train_network.py" --bucket_no_upscale --bucket_reso_steps=64 --cache_latents --caption_extension=".txt" --clip_skip=2 --enable_bucket --min_bucket_reso=256
--max_bucket_reso=2048 --keep_tokens="1" --learning_rate="0.0001" --logging_dir="D:/sdlanuncher/LoRA_test/log" --lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles="4" --lr_warmup_steps="760"
--max_data_loader_n_workers="1" --max_grad_norm="1" --resolution="512,512" --max_train_steps="7600" --mixed_precision="fp16" --network_alpha="128" --network_dim=128 --network_module=networks.lora
--optimizer_type="AdamW8bit" --output_dir="D:/sdlanuncher/LoRA_test/model" --output_name="mengwenwen" --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" --save_every_n_epochs="1"
--save_model_as=safetensors --save_precision="fp16" --seed="1234" --text_encoder_lr=5e-05 --train_batch_size="2" --train_data_dir="D:/sdlanuncher/LoRA_test/image" --unet_lr=0.0001 --xformers
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
prepare tokenizer
Using DreamBooth method.
prepare images.
found directory D:\sdlanuncher\LoRA_test\image\100_face contains 38 image files
3800 train images with repeating.
0 reg images.
no regularization images / 正則化画像が見つかりませんでした
[Dataset 0]
batch_size: 2
resolution: (512, 512)
enable_bucket: True
network_multiplier: 1.0
min_bucket_reso: 256
max_bucket_reso: 2048
bucket_reso_steps: 64
bucket_no_upscale: True
[Subset 0 of Dataset 0]
image_dir: "D:\sdlanuncher\LoRA_test\image\100_face"
image_count: 38
num_repeats: 100
shuffle_caption: False
keep_tokens: 1
keep_tokens_separator:
caption_dropout_rate: 0.0
caption_dropout_every_n_epoches: 0
caption_tag_dropout_rate: 0.0
caption_prefix: None
caption_suffix: None
color_aug: False
flip_aug: False
face_crop_aug_range: None
random_crop: False
token_warmup_min: 1,
token_warmup_step: 0,
is_reg: False
class_tokens: face
caption_extension: .txt
[Dataset 0]
loading image sizes.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 38/38 [00:00<00:00, 1216.26it/s]
make buckets
min_bucket_reso and max_bucket_reso are ignored if bucket_no_upscale is set, because bucket reso is defined by image size automatically / bucket_no_upscaleが指定された場合は、bucketの解像度は画像サイズから自動計算されるため、min_bucket_resoとmax_bucket_resoは無視されます
number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む）
bucket 0: resolution (512, 512), count: 3800
mean ar error (without repeats): 0.0
preparing accelerator
loading model for process 0/1
load Diffusers pretrained models: runwayml/stable-diffusion-v1-5
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 15.99it/s]
You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
UNet2DConditionModel: 64, 8, 768, False, False
U-Net converted to original U-Net
Enable xformers for U-Net
import network module: networks.lora
[Dataset 0]
caching latents.
checking cache validity...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 38/38 [00:00<?, ?it/s]
caching latents...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 38/38 [00:01<00:00, 22.10it/s]
create LoRA network. base dim (rank): 128, alpha: 128.0
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
create LoRA for Text Encoder:
create LoRA for Text Encoder: 72 modules.
create LoRA for U-Net: 192 modules.
enable LoRA for text encoder
enable LoRA for U-Net
prepare optimizer, data loader etc.
False
===================================BUG REPORT===================================
D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\cuda_setup\main.py:167: UserWarning: Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
warn(msg)
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAhV2tNImUekiwt7/G1ZRvUwQAAAACAAAAAAAQZgAAAAEAACAAAADFGLlvVY9C1APSO8qclvAlOgni55HdQGNgcUyVgA9A1wAAAAAOgAAAAAIAACAAAACyclj1wPq0n4ldOzIgplRViyKxJFnGpyQ9xQ73BbuAdmAAAACkqRZMXBCaVoOfUZ+nAHQBImK4IkBLfTwcICGKjP9HRm/fKOwS+S8LkqqK1q+aeK01HPVZS9L7je6FUX36JTz39dYkhbnQaXr8MlqMs6Pm6nrmaSvojbZS/wAHUJburT1AAAAA2pCtZhovG96YwGg5Xovq+SxYbVN9WZgqNVLFs8Kp7mPixTISey7/ifPlhMRijwqPC0TJhzUceKPI/lyxEU9P9g==')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.9.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://raw.githubusercontent.com/TimDettmers/bitsandbytes/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
Traceback (most recent call last):
File "D:\kohya_ss\kohya_ss\train_network.py", line 1033, in <module>
trainer.train(args)
File "D:\kohya_ss\kohya_ss\train_network.py", line 345, in train
optimizer_name, optimizer_args, optimizer = train_util.get_optimizer(args, trainable_params)
File "D:\kohya_ss\kohya_ss\library\train_util.py", line 3510, in get_optimizer
import bitsandbytes as bnb
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\__init__.py", line 6, in <module>
from . import cuda_setup, utils, research
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\research\__init__.py", line 1, in <module>
from . import nn
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\research\nn\__init__.py", line 1, in <module>
from .modules import LinearFP8Mixed, LinearFP8Global
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in <module>
from bitsandbytes.optim import GlobalOptimManager
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\optim\__init__.py", line 6, in <module>
from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py", line 20, in <module>
raise RuntimeError('''
RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:
python -m bitsandbytes
Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
Traceback (most recent call last):
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\kohya_ss\kohya_ss\venv\Scripts\accelerate.exe\__main__.py", line 7, in <module>
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
simple_launcher(args)
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['D:\\kohya_ss\\kohya_ss\\venv\\Scripts\\python.exe', './train_network.py', '--bucket_no_upscale', '--bucket_reso_steps=64', '--cache_latents', '--caption_extension=.txt', '--clip_skip=2', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--keep_tokens=1', '--learning_rate=0.0001', '--logging_dir=D:/sdlanuncher/LoRA_test/log', '--lr_scheduler=cosine_with_restarts', '--lr_scheduler_num_cycles=4', '--lr_warmup_steps=760', '--max_data_loader_n_workers=1', '--max_grad_norm=1', '--resolution=512,512', '--max_train_steps=7600', '--mixed_precision=fp16', '--network_alpha=128', '--network_dim=128', '--network_module=networks.lora', '--optimizer_type=AdamW8bit', '--output_dir=D:/sdlanuncher/LoRA_test/model', '--output_name=mengwenwen', '--pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5', '--save_every_n_epochs=1', '--save_model_as=safetensors', '--save_precision=fp16', '--seed=1234', '--text_encoder_lr=5e-05', '--train_batch_size=2', '--train_data_dir=D:/sdlanuncher/LoRA_test/image', '--unet_lr=0.0001', '--xformers']' returned non-zero exit status 1.
19:36:11-887145 INFO nVidia toolkit detected
19:36:38-226469 INFO Torch 2.1.2+cu118
19:36:38-242091 INFO Torch backend: nVidia CUDA 11.8 cuDNN 8700
19:36:38-242091 INFO Torch detected GPU: NVIDIA GeForce RTX 4080 SUPER VRAM 16376 Arch (8, 9) Cores 80
19:36:38-242091 INFO Verifying modules installation status from requirements_windows_torch2.txt...
19:36:38-242091 INFO Installing package: torch==2.1.2+cu118 torchvision==0.16.2+cu118 torchaudio==2.1.2+cu118
--index-url https://download.pytorch.org/whl/cu118
19:36:42-925923 INFO Installing package: xformers==0.0.23.post1+cu118 --index-url
https://download.pytorch.org/whl/cu118
19:36:45-476670 INFO Verifying modules installation status from requirements.txt...
19:36:50-010949 INFO headless: False
19:36:50-010949 INFO Load CSS...
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
19:37:30-698481 INFO Loading config...
19:39:22-153649 INFO Save...
19:40:02-327279 INFO Start training LoRA Standard ...
19:40:02-328279 INFO Checking for duplicate image filenames in training data directory...
19:40:02-334280 INFO Valid image folder names found in: D:/sdlanuncher/LoRA_test/image
19:40:02-336279 INFO Folder 100_face: 38 images found
19:40:02-337279 INFO Folder 100_face: 3800 steps
19:40:02-338279 INFO Total steps: 3800
19:40:02-339281 INFO Train batch size: 2
19:40:02-339281 INFO Gradient accumulation steps: 1
19:40:02-341279 INFO Epoch: 4
19:40:02-342279 INFO Regulatization factor: 1
19:40:02-343279 INFO max_train_steps (3800 / 2 / 1 * 4 * 1) = 7600
19:40:02-344279 INFO stop_text_encoder_training = 0
19:40:02-345279 INFO lr_warmup_steps = 760
19:40:02-346279 WARNING Here is the trainer command as a reference. It will not be executed:
accelerate launch --num_cpu_threads_per_process=2 "./train_network.py" --bucket_no_upscale --bucket_reso_steps=64 --cache_latents --caption_extension=".txt" --clip_skip=2 --enable_bucket --min_bucket_reso=256 --max_bucket_reso=2048 --keep_tokens="1" --learning_rate="0.0001" --logging_dir="D:/sdlanuncher/LoRA_test/log" --lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles="4" --lr_warmup_steps="760" --max_data_loader_n_workers="1" --max_grad_norm="1" --resolution="512,512" --max_train_steps="7600" --mixed_precision="fp16" --network_alpha="128" --network_dim=128 --network_module=networks.lora --optimizer_type="AdamW8bit" --output_dir="D:/sdlanuncher/LoRA_test/model" --output_name="mengwenwen" --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" --save_every_n_epochs="1" --save_model_as=safetensors --save_precision="fp16" --seed="1234" --text_encoder_lr=5e-05 --train_batch_size="2" --train_data_dir="D:/sdlanuncher/LoRA_test/image" --unet_lr=0.0001 --xformers
19:40:10-471477 INFO Start training LoRA Standard ...
19:40:13-659082 INFO Checking for duplicate image filenames in training data directory...
19:40:13-659082 INFO Valid image folder names found in: D:/sdlanuncher/LoRA_test/image
19:40:13-659082 INFO Folder 100_face: 38 images found
19:40:13-659082 INFO Folder 100_face: 3800 steps
19:40:13-659082 INFO Total steps: 3800
19:40:13-705962 INFO Train batch size: 2
19:40:13-705962 INFO Gradient accumulation steps: 1
19:40:13-705962 INFO Epoch: 4
19:40:13-705962 INFO Regulatization factor: 1
19:40:13-705962 INFO max_train_steps (3800 / 2 / 1 * 4 * 1) = 7600
19:40:13-705962 INFO stop_text_encoder_training = 0
19:40:13-705962 INFO lr_warmup_steps = 760
19:40:13-705962 INFO Saving training config to D:/sdlanuncher/LoRA_test/model\mengwenwen_20240225-194013.json...
19:40:13-705962 INFO accelerate launch --num_cpu_threads_per_process=2 "./train_network.py" --bucket_no_upscale --bucket_reso_steps=64 --cache_latents --caption_extension=".txt" --clip_skip=2 --enable_bucket --min_bucket_reso=256
--max_bucket_reso=2048 --keep_tokens="1" --learning_rate="0.0001" --logging_dir="D:/sdlanuncher/LoRA_test/log" --lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles="4" --lr_warmup_steps="760"
--max_data_loader_n_workers="1" --max_grad_norm="1" --resolution="512,512" --max_train_steps="7600" --mixed_precision="fp16" --network_alpha="128" --network_dim=128 --network_module=networks.lora
--optimizer_type="AdamW8bit" --output_dir="D:/sdlanuncher/LoRA_test/model" --output_name="mengwenwen" --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" --save_every_n_epochs="1"
--save_model_as=safetensors --save_precision="fp16" --seed="1234" --text_encoder_lr=5e-05 --train_batch_size="2" --train_data_dir="D:/sdlanuncher/LoRA_test/image" --unet_lr=0.0001 --xformers
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
prepare tokenizer
Using DreamBooth method.
prepare images.
found directory D:\sdlanuncher\LoRA_test\image\100_face contains 38 image files
3800 train images with repeating.
0 reg images.
no regularization images / 正則化画像が見つかりませんでした
[Dataset 0]
batch_size: 2
resolution: (512, 512)
enable_bucket: True
network_multiplier: 1.0
min_bucket_reso: 256
max_bucket_reso: 2048
bucket_reso_steps: 64
bucket_no_upscale: True
[Subset 0 of Dataset 0]
image_dir: "D:\sdlanuncher\LoRA_test\image\100_face"
image_count: 38
num_repeats: 100
shuffle_caption: False
keep_tokens: 1
keep_tokens_separator:
caption_dropout_rate: 0.0
caption_dropout_every_n_epoches: 0
caption_tag_dropout_rate: 0.0
caption_prefix: None
caption_suffix: None
color_aug: False
flip_aug: False
face_crop_aug_range: None
random_crop: False
token_warmup_min: 1,
token_warmup_step: 0,
is_reg: False
class_tokens: face
caption_extension: .txt
[Dataset 0]
loading image sizes.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 38/38 [00:00<00:00, 1216.26it/s]
make buckets
min_bucket_reso and max_bucket_reso are ignored if bucket_no_upscale is set, because bucket reso is defined by image size automatically / bucket_no_upscaleが指定された場合は、bucketの解像度は画像サイズから自動計算されるため、min_bucket_resoとmax_bucket_resoは無視されます
number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む）
bucket 0: resolution (512, 512), count: 3800
mean ar error (without repeats): 0.0
preparing accelerator
loading model for process 0/1
load Diffusers pretrained models: runwayml/stable-diffusion-v1-5
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 15.99it/s]
You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
UNet2DConditionModel: 64, 8, 768, False, False
U-Net converted to original U-Net
Enable xformers for U-Net
import network module: networks.lora
[Dataset 0]
caching latents.
checking cache validity...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 38/38 [00:00<?, ?it/s]
caching latents...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 38/38 [00:01<00:00, 22.10it/s]
create LoRA network. base dim (rank): 128, alpha: 128.0
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
create LoRA for Text Encoder:
create LoRA for Text Encoder: 72 modules.
create LoRA for U-Net: 192 modules.
enable LoRA for text encoder
enable LoRA for U-Net
prepare optimizer, data loader etc.
False
===================================BUG REPORT===================================
D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\cuda_setup\main.py:167: UserWarning: Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
warn(msg)
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAhV2tNImUekiwt7/G1ZRvUwQAAAACAAAAAAAQZgAAAAEAACAAAADFGLlvVY9C1APSO8qclvAlOgni55HdQGNgcUyVgA9A1wAAAAAOgAAAAAIAACAAAACyclj1wPq0n4ldOzIgplRViyKxJFnGpyQ9xQ73BbuAdmAAAACkqRZMXBCaVoOfUZ+nAHQBImK4IkBLfTwcICGKjP9HRm/fKOwS+S8LkqqK1q+aeK01HPVZS9L7je6FUX36JTz39dYkhbnQaXr8MlqMs6Pm6nrmaSvojbZS/wAHUJburT1AAAAA2pCtZhovG96YwGg5Xovq+SxYbVN9WZgqNVLFs8Kp7mPixTISey7/ifPlhMRijwqPC0TJhzUceKPI/lyxEU9P9g==')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.9.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://raw.githubusercontent.com/TimDettmers/bitsandbytes/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
Traceback (most recent call last):
File "D:\kohya_ss\kohya_ss\train_network.py", line 1033, in <module>
trainer.train(args)
File "D:\kohya_ss\kohya_ss\train_network.py", line 345, in train
optimizer_name, optimizer_args, optimizer = train_util.get_optimizer(args, trainable_params)
File "D:\kohya_ss\kohya_ss\library\train_util.py", line 3510, in get_optimizer
import bitsandbytes as bnb
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\__init__.py", line 6, in <module>
from . import cuda_setup, utils, research
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\research\__init__.py", line 1, in <module>
from . import nn
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\research\nn\__init__.py", line 1, in <module>
from .modules import LinearFP8Mixed, LinearFP8Global
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in <module>
from bitsandbytes.optim import GlobalOptimManager
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\optim\__init__.py", line 6, in <module>
from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py", line 20, in <module>
raise RuntimeError('''
RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:
python -m bitsandbytes
Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
Traceback (most recent call last):
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\kohya_ss\kohya_ss\venv\Scripts\accelerate.exe\__main__.py", line 7, in <module>
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
simple_launcher(args)
File "D:\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['D:\\kohya_ss\\kohya_ss\\venv\\Scripts\\python.exe', './train_network.py', '--bucket_no_upscale', '--bucket_reso_steps=64', '--cache_latents', '--caption_extension=.txt', '--clip_skip=2', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--keep_tokens=1', '--learning_rate=0.0001', '--logging_dir=D:/sdlanuncher/LoRA_test/log', '--lr_scheduler=cosine_with_restarts', '--lr_scheduler_num_cycles=4', '--lr_warmup_steps=760', '--max_data_loader_n_workers=1', '--max_grad_norm=1', '--resolution=512,512', '--max_train_steps=7600', '--mixed_precision=fp16', '--network_alpha=128', '--network_dim=128', '--network_module=networks.lora', '--optimizer_type=AdamW8bit', '--output_dir=D:/sdlanuncher/LoRA_test/model', '--output_name=mengwenwen', '--pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5', '--save_every_n_epochs=1', '--save_model_as=safetensors', '--save_precision=fp16', '--seed=1234', '--text_encoder_lr=5e-05', '--train_batch_size=2', '--train_data_dir=D:/sdlanuncher/LoRA_test/image', '--unet_lr=0.0001', '--xformers']' returned non-zero exit status 1.
