ERNIE doesn't work

[morti86]

All I get is garbled images like this:

I use Docker vulkan image and q4_k_m/q5_k_m GGUF's.

[wbruna]

Could you please post a full --verbose generation log, and tell us where you got the model files?

[morti86]

Model files: Unsloth from HF,

This is the command, although I have tried several text encoders (don't seem to be the problem).

sudo docker run --rm \
	--device /dev/dri \
	--device /dev/kfd \
	-v /home/morti/models/flux:/models \
	-v /home/morti/Pictures:/output \
	ghcr.io/leejet/stable-diffusion.cpp:master-vulkan \
	--diffusion-model /models/ernie-image-Q5_K_M.gguf \
	--vae /models/flux2-vae.safetensors \
	--llm /models/ernie-image-abliterated-te-Q6_K.gguf \
	--mmap \
	--diffusion-fa \
	-H 720 \
	-W 1280 \
	--cfg-scale 5.0 \
	--steps 50 \
	-p "$(cat $1)" \
	-o /output/$2 \
	-s -1 \
	--offload-to-cpu

The log:

gml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XT (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: KHR_coopmat
[INFO ] stable-diffusion.cpp:306  - loading diffusion model from '/models/ernie-image-Q5_K_M.gguf'
[INFO ] model_loader.cpp:216  - load /models/ernie-image-Q5_K_M.gguf using gguf format
[INFO ] stable-diffusion.cpp:360  - loading llm from '/models/ernie-image-abliterated-te-Q6_K.gguf'
[INFO ] model_loader.cpp:216  - load /models/ernie-image-abliterated-te-Q6_K.gguf using gguf format
[INFO ] stable-diffusion.cpp:374  - loading vae from '/models/flux2-vae.safetensors'
[INFO ] model_loader.cpp:219  - load /models/flux2-vae.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:416  - Version: Ernie Image 
[INFO ] stable-diffusion.cpp:444  - Weight type stat:                      f32: 451  |    q5_K: 180  |    q6_K: 255  |    bf16: 7    
[INFO ] stable-diffusion.cpp:445  - Conditioner weight type stat:          f32: 53   |    q6_K: 183  
[INFO ] stable-diffusion.cpp:446  - Diffusion model weight type stat:      f32: 150  |    q5_K: 180  |    q6_K: 72   |    bf16: 7    
[INFO ] stable-diffusion.cpp:447  - VAE weight type stat:                  f32: 248  
[INFO ] stable-diffusion.cpp:865  - using VAE for encoding / decoding
[INFO ] auto_encoder_kl.hpp:525  - vae decoder: ch = 128
[INFO ] stable-diffusion.cpp:952  - Using flash attention in the diffusion model
[INFO ] model_loader.cpp:806  - using mmap for '/models/ernie-image-Q5_K_M.gguf'
[INFO ] model_loader.cpp:806  - using mmap for '/models/ernie-image-abliterated-te-Q6_K.gguf'
[INFO ] model_loader.cpp:806  - using mmap for '/models/flux2-vae.safetensors'
[INFO ] model_loader.cpp:828  - model files processing completed in 0.00s
[INFO ] model_loader.cpp:913  - memory-mapped 743 tensors in 3 files (8092.02 MB), taking 0.00s
  |======================>                           | 409/893 - 5.00MB/s
  |====================================>             | 645/893 - 260.51MB/s
  |==================================================| 893/893 - 278.47MB/s
[INFO ] model_loader.cpp:1167 - loading tensors completed, taking 1.81s (read: 0.57s, memcpy: 0.00s, convert: 0.08s, copy_to_backend: 0.00s)
[INFO ] stable-diffusion.cpp:1158 - total params memory size = 1631.57MB (VRAM 0.00MB, RAM 1631.57MB): text_encoders 1536.00MB(RAM), diffusion_model 1.00MB(RAM), vae 94.57MB(RAM), controlnet 0.00MB(N/A), extensions 0.00MB(N/A)
[INFO ] stable-diffusion.cpp:1263 - running in FLOW mode
[INFO ] stable-diffusion.cpp:4416 - generate_image 1280x720
[INFO ] denoiser.hpp:579  - get_sigmas with discrete scheduler
[INFO ] stable-diffusion.cpp:3470 - sampling using Euler method
[INFO ] ggml_extend.hpp:2150 - ministral3.3b offload params (3904.04 MB, 236 tensors) to runtime backend (Vulkan0), taking 4.72s
[INFO ] ggml_extend.hpp:2150 - ministral3.3b offload params (3904.04 MB, 236 tensors) to runtime backend (Vulkan0), taking 0.99s
[INFO ] stable-diffusion.cpp:4173 - get_learned_condition completed, taking 6.10s
[INFO ] stable-diffusion.cpp:4450 - generating image: 1/1 - seed 1857686940
[INFO ] ggml_extend.hpp:2150 - ernie_image offload params (5724.83 MB, 409 tensors) to runtime backend (Vulkan0), taking 14.86s
  |==================================================| 50/50 - 5.32s/it
[INFO ] stable-diffusion.cpp:4482 - sampling completed, taking 266.07s
[INFO ] stable-diffusion.cpp:4500 - generating 1 latent images completed, taking 266.07s
[INFO ] stable-diffusion.cpp:4194 - decoding 1 latents
[INFO ] ggml_extend.hpp:2150 - vae offload params ( 94.72 MB, 140 tensors) to runtime backend (Vulkan0), taking 0.02s
[INFO ] stable-diffusion.cpp:4210 - latent 1 decoded, taking 0.89s
[INFO ] stable-diffusion.cpp:4214 - decode_first_stage completed, taking 0.89s
[INFO ] stable-diffusion.cpp:4634 - generate_image completed in 273.10s
[INFO ] main.cpp:463  - save result image 0 to '/output/forest6.png' (success)
[INFO ] main.cpp:535  - 1/1 images saved

[morti86]

Generally these do not alter the result. Sometimes, just sometimes I can see some extremely garbled outlines of what I would like generated. With 512x512 image resolution I get what I want, but it is insanely noisy.

Although when I asked about a black cat on a wooden table, something extremely generic, it succeeded.

Tried to run it with the official safetensors text encoder from official Ernie profile and it crashes.
Prompting it in Chinese makes it worse, results get completely random. Got an anime girl instead of trees.

The Ministral model does not work either:

[ERROR] model_loader.cpp:1240 - tensor 'text_encoders.llm.model.embed_tokens.weight' not in model file
[ERROR] model_loader.cpp:1240 - tensor 'text_encoders.llm.model.norm.weight' not in model file
[ERROR] stable-diffusion.cpp:1086 - load tensors from model loader failed

[Green-Sky]

yea chinese tokenization seems broken.

try preview with linear projection and see if the problems start at a specific step or something.

[morti86]

Gave up on Ernie, playing with Ideogram4 which seems to work, although requires long, extremely detailed prompts and tends to make painting-like pictures rather than photorealistic ones.

Generally Flux 2 Klein 9B seems the best at generating more abstract, fantasy stuff while some models are great only at pictures of cats and pretty girls.

[Green-Sky]

Gave up on Ernie, playing with Ideogram4 which seems to work, although requires long, extremely detailed prompts and tends to make painting-like pictures rather than photorealistic ones.

I think I saw a lora for that specific usecase.

ig4 should just be the best imo, if you use a prompt enhancer and some bounding box prompt editor.

[morti86]

I have found the answer for the painting-like thing with Ideogram4. Need to set --sampling-method lcm --steps 16 --cfg-scale 2.5 and it is way better, although some prompts can require as many as 40 steps to render well, but it works at least.

[morti86]

Full verbose:

[DEBUG] main.cpp:598  - version: stable-diffusion.cpp version unknown, commit 19bdfe2
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XT (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: KHR_coopmat
[DEBUG] main.cpp:599  - System Info: 
   SSE3 = 1 |    SSSE3 = 1 |    AVX = 1 |    AVX2 = 1 |    F16C = 1 |    FMA = 1 |    BMI2 = 1 |    OPENMP = 1 |    REPACK = 1 | 
[DEBUG] main.cpp:600  - SDCliParams {
  mode: img_gen,
  output_path: "/output/forest6.png",
  image_path: "",
  metadata_format: "text",
  verbose: true,
  color: false,
  canny_preprocess: false,
  convert_name: false,
  preview_method: none,
  preview_interval: 1,
  preview_path: "preview.png",
  preview_fps: 16,
  taesd_preview: false,
  preview_noisy: false,
  metadata_raw: false,
  metadata_brief: false,
  metadata_all: false
}
[DEBUG] main.cpp:601  - SDContextParams {
  n_threads: 8,
  model_path: "",
  clip_l_path: "",
  clip_g_path: "",
  clip_vision_path: "",
  t5xxl_path: "",
  llm_path: "/models/ernie-image-abliterated-te-Q6_K.gguf",
  llm_vision_path: "",
  diffusion_model_path: "/models/Ernie-Image-Flex-V1-Q8_0.gguf",
  high_noise_diffusion_model_path: "",
  uncond_diffusion_model_path: "",
  embeddings_connectors_path: "",
  vae_path: "/models/flux2-vae.safetensors",
  vae_format: "auto",
  audio_vae_path: "",
  taesd_path: "",
  esrgan_path: "",
  control_net_path: "",
  embedding_dir: "",
  embeddings: {
  }
  wtype: NONE,
  tensor_type_rules: "",
  lora_model_dir: ".",
  hires_upscalers_dir: "",
  photo_maker_path: "",
  rng_type: cuda,
  sampler_rng_type: NONE,
  offload_params_to_cpu: true,
  max_vram: 0,
  stream_layers: false,
  backend: "",
  params_backend: "",
  enable_mmap: true,
  control_net_cpu: false,
  clip_on_cpu: false,
  vae_on_cpu: false,
  flash_attn: false,
  diffusion_flash_attn: true,
  diffusion_conv_direct: false,
  vae_conv_direct: false,
  circular: false,
  circular_x: false,
  circular_y: false,
  chroma_use_dit_mask: true,
  qwen_image_zero_cond_t: false,
  chroma_use_t5_mask: false,
  chroma_t5_mask_pad: 1,
  prediction: NONE,
  lora_apply_mode: auto,
  force_sdxl_vae_conv_scale: false
}
[DEBUG] main.cpp:602  - SDGenerationParams {
  loras: "{
  }",
  high_noise_loras: "{
  }",
  prompt: "The sky is of amber color with a few small yellow clouds. The vegetation is all dark purple.

A narrow trail in a mountain forest.",
  negative_prompt: "",
  clip_skip: -1,
  width: 1280,
  height: 720,
  batch_count: 1,
  init_image_path: "",
  end_image_path: "",
  mask_image_path: "",
  control_image_path: "",
  ref_image_paths: [],
  control_video_path: "",
  auto_resize_ref_image: true,
  increase_ref_index: false,
  pm_id_images_dir: "",
  pm_id_embed_path: "",
  pm_style_strength: 20,
  skip_layers: [7, 8, 9],
  sample_params: (txt_cfg: 5.00, img_cfg: 5.00, distilled_guidance: 3.50, slg.layer_count: 0, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: NONE, sample_method: NONE, sample_steps: 50, eta: inf, shifted_timestep: 0, flow_shift: inf, extra_sample_args: ),
  high_noise_skip_layers: [7, 8, 9],
  high_noise_sample_params: (txt_cfg: 7.00, img_cfg: 7.00, distilled_guidance: 3.50, slg.layer_count: 0, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: NONE, sample_method: NONE, sample_steps: 20, eta: inf, shifted_timestep: 0, flow_shift: inf, extra_sample_args: ),
  custom_sigmas: [],
  cache_mode: "",
  cache_option: "",
  cache: disabled (threshold=inf, start=0.15, end=0.95),
  moe_boundary: 0.875,
  video_frames: 1,
  fps: 16,
  vace_strength: 1,
  strength: 0.75,
  control_strength: 0.9,
  seed: 1370584644,
  upscale_repeats: 1,
  upscale_tile_size: 128,
  hires: { enabled: false, upscaler: "Latent", model_path: "", scale: 2, target_width: 0, target_height: 0, steps: 0, denoising_strength: 0.7, custom_sigmas: [], upscale_tile_size: 128 },
  vae_tiling_params: { 0, 0, 0, 0, 0.5, 0, 0, "" },
}
[DEBUG] ggml_extend_backend.cpp:367  - Found 2 backend devices:
[DEBUG] ggml_extend_backend.cpp:370  - #0: Vulkan0
[DEBUG] ggml_extend_backend.cpp:370  - #1: CPU
[DEBUG] ggml_extend_backend.cpp:292  - Initializing backend: Vulkan0
[DEBUG] ggml_extend_backend.cpp:292  - Initializing backend: CPU
[INFO ] stable-diffusion.cpp:306  - loading diffusion model from '/models/Ernie-Image-Flex-V1-Q8_0.gguf'
[INFO ] model_loader.cpp:216  - load /models/Ernie-Image-Flex-V1-Q8_0.gguf using gguf format
[DEBUG] model_loader.cpp:265  - init from '/models/Ernie-Image-Flex-V1-Q8_0.gguf'
[INFO ] stable-diffusion.cpp:360  - loading llm from '/models/ernie-image-abliterated-te-Q6_K.gguf'
[INFO ] model_loader.cpp:216  - load /models/ernie-image-abliterated-te-Q6_K.gguf using gguf format
[DEBUG] model_loader.cpp:265  - init from '/models/ernie-image-abliterated-te-Q6_K.gguf'
[INFO ] stable-diffusion.cpp:374  - loading vae from '/models/flux2-vae.safetensors'
[INFO ] model_loader.cpp:219  - load /models/flux2-vae.safetensors using safetensors format
[DEBUG] model_loader.cpp:294  - init from '/models/flux2-vae.safetensors', prefix = 'vae.'
[INFO ] stable-diffusion.cpp:416  - Version: Ernie Image 
[INFO ] stable-diffusion.cpp:444  - Weight type stat:                      f32: 451  |     f16: 21   |    q8_0: 238  |    q6_K: 183  
[INFO ] stable-diffusion.cpp:445  - Conditioner weight type stat:          f32: 53   |    q6_K: 183  
[INFO ] stable-diffusion.cpp:446  - Diffusion model weight type stat:      f32: 150  |     f16: 21   |    q8_0: 238  
[INFO ] stable-diffusion.cpp:447  - VAE weight type stat:                  f32: 248  
[DEBUG] stable-diffusion.cpp:449  - ggml tensor size = 400 bytes
[DEBUG] mistral_tokenizer.cpp:23   - vocab size: 131072
[DEBUG] mistral_tokenizer.cpp:31   - merges size 269443
[DEBUG] llm.hpp:221  - llm: num_layers = 26, vocab_size = 131072, hidden_size = 3072, intermediate_size = 9216
[DEBUG] ernie_image.hpp:75   - ernie_image: num_layers = 36, hidden_size = 4096, num_heads = 32, ffn_hidden_size = 12288, in_channels = 128, out_channels = 128
[INFO ] stable-diffusion.cpp:865  - using VAE for encoding / decoding
[INFO ] auto_encoder_kl.hpp:525  - vae decoder: ch = 128
[INFO ] stable-diffusion.cpp:952  - Using flash attention in the diffusion model
[DEBUG] stable-diffusion.cpp:981  - loading weights
[DEBUG] model_loader.cpp:800  - using mmap for I/O
[INFO ] model_loader.cpp:806  - using mmap for '/models/Ernie-Image-Flex-V1-Q8_0.gguf'
[DEBUG] model_loader.cpp:800  - using mmap for I/O
[INFO ] model_loader.cpp:806  - using mmap for '/models/ernie-image-abliterated-te-Q6_K.gguf'
[DEBUG] model_loader.cpp:800  - using mmap for I/O
[INFO ] model_loader.cpp:806  - using mmap for '/models/flux2-vae.safetensors'
[INFO ] model_loader.cpp:828  - model files processing completed in 0.00s
[DEBUG] model_loader.cpp:840  - memory-mapping tensors...
[INFO ] model_loader.cpp:913  - memory-mapped 744 tensors in 3 files (11061.52 MB), taking 0.00s
[DEBUG] ggml_extend.hpp:3074 - ministral3.3b params backend buffer size =  1536.00 MB(RAM) (236 tensors)
[DEBUG] ggml_extend.hpp:3044 - ernie_image all params already mmap-allocated (no separate buffer needed)
[DEBUG] ggml_extend.hpp:3074 - vae params backend buffer size =  94.57 MB(RAM) (140 tensors)
[DEBUG] model_loader.cpp:932  - using 8 threads for model loading
[DEBUG] model_loader.cpp:948  - loading tensors from /models/Ernie-Image-Flex-V1-Q8_0.gguf
  |======================>                           | 409/893 - 0.00MB/s
[DEBUG] model_loader.cpp:948  - loading tensors from /models/ernie-image-abliterated-te-Q6_K.gguf
  |====================================>             | 645/893 - 432.10MB/s
[DEBUG] model_loader.cpp:948  - loading tensors from /models/flux2-vae.safetensors
  |==================================================| 893/893 - 542.67MB/s
[INFO ] model_loader.cpp:1167 - loading tensors completed, taking 0.93s (read: 0.01s, memcpy: 0.00s, convert: 0.07s, copy_to_backend: 0.00s)
[DEBUG] stable-diffusion.cpp:1091 - finished loaded file
[INFO ] stable-diffusion.cpp:1158 - total params memory size = 1630.57MB (VRAM 0.00MB, RAM 1630.57MB): text_encoders 1536.00MB(RAM), diffusion_model 0.00MB(N/A), vae 94.57MB(RAM), controlnet 0.00MB(N/A), extensions 0.00MB(N/A)
[INFO ] stable-diffusion.cpp:1263 - running in FLOW mode
[INFO ] stable-diffusion.cpp:4416 - generate_image 1280x720
[INFO ] denoiser.hpp:579  - get_sigmas with discrete scheduler
[INFO ] stable-diffusion.cpp:3470 - sampling using Euler method
[DEBUG] conditioner.hpp:1688 - parse 'The sky is of amber color with a few small yellow clouds. The vegetation is all dark purple.

A narrow trail in a mountain forest.' to [['The sky is of amber color with a few small yellow clouds. The vegetation is all dark purple.

A narrow trail in a mountain forest.', 1], ]
[DEBUG] bpe_tokenizer.cpp:207  - split prompt "The sky is of amber color with a few small yellow clouds. The vegetation is all dark purple.

A narrow trail in a mountain forest." to tokens ["The", "Ġsky", "Ġis", "Ġof", "Ġamber", "Ġcolor", "Ġwith", "Ġa", "Ġfew", "Ġsmall", "Ġyellow", "Ġclouds", ".", "ĠThe", "Ġvegetation", "Ġis", "Ġall", "Ġdark", "Ġpurple", ".ĊĊ", "A", "Ġnarrow", "Ġtrail", "Ġin", "Ġa", "Ġmountain", "Ġforest", ".", ]
[DEBUG] ggml_extend.hpp:1912 - ministral3.3b compute buffer size: 3.18 MB(VRAM)
[INFO ] ggml_extend.hpp:2150 - ministral3.3b offload params (3904.04 MB, 236 tensors) to runtime backend (Vulkan0), taking 0.94s
[DEBUG] conditioner.hpp:2098 - computing condition graph completed, taking 1265 ms
[DEBUG] conditioner.hpp:1688 - parse '' to [['', 1], ]
[DEBUG] bpe_tokenizer.cpp:207  - split prompt "" to tokens []
[DEBUG] ggml_extend.hpp:1912 - ministral3.3b compute buffer size: 0.09 MB(VRAM)
[INFO ] ggml_extend.hpp:2150 - ministral3.3b offload params (3904.04 MB, 236 tensors) to runtime backend (Vulkan0), taking 0.84s
[DEBUG] conditioner.hpp:2098 - computing condition graph completed, taking 914 ms
[INFO ] stable-diffusion.cpp:4173 - get_learned_condition completed, taking 2.18s
[INFO ] stable-diffusion.cpp:4450 - generating image: 1/1 - seed 1370584644
[DEBUG] ggml_extend.hpp:1912 - ernie_image compute buffer size: 572.22 MB(VRAM)
[INFO ] ggml_extend.hpp:2150 - ernie_image offload params (8693.33 MB, 409 tensors) to runtime backend (Vulkan0), taking 2.07s
  |==================================================| 50/50 - 4.81s/it
[INFO ] stable-diffusion.cpp:4482 - sampling completed, taking 240.84s
[INFO ] stable-diffusion.cpp:4500 - generating 1 latent images completed, taking 240.84s
[INFO ] stable-diffusion.cpp:4194 - decoding 1 latents
[DEBUG] ggml_extend.hpp:1912 - vae compute buffer size: 7008.40 MB(VRAM)
[INFO ] ggml_extend.hpp:2150 - vae offload params ( 94.72 MB, 140 tensors) to runtime backend (Vulkan0), taking 0.02s
[DEBUG] vae.hpp:210  - computing vae decode graph completed, taking 0.90s
[INFO ] stable-diffusion.cpp:4210 - latent 1 decoded, taking 0.90s
[INFO ] stable-diffusion.cpp:4214 - decode_first_stage completed, taking 0.90s
[INFO ] stable-diffusion.cpp:4634 - generate_image completed in 243.96s
[INFO ] main.cpp:463  - save result image 0 to '/output/forest6.png' (success)
[INFO ] main.cpp:535  - 1/1 images saved