[Bug] PiD 1.5: `--diffusion-fa` zeroes whole 8×8 patch tokens above 2560², giving a black frame from the second step on (ROCm / gfx1151)

[larochef]

Git commit

17860c0, using release master-881-17860c0 with rocm

Operating System & Version

CachyOs

GGML backends

HIP

Command-line arguments used

sd-cli -M img_gen --diffusion-model pid_1.5_flux2_1024_to_4096_4step_bf16.safetensors --llm gemma_2_2b_it_elm_bf16.safetensors --tokenizer tokenizer.json --vae flux2_ae.safetensors --vae-format flux2 --rng cpu --cfg-scale 1.0 --steps 4 --seed 42 -p "A woman in a red silk dress beside a window in a dim room at golden hour." --ref-image source-768.png -W 3072 -H 3072 --diffusion-fa -o out.png

Steps to reproduce

Reproduce

sd-cli -M img_gen \
  --diffusion-model pid_1.5_flux2_1024_to_4096_4step_bf16.safetensors \
  --llm gemma_2_2b_it_elm_bf16.safetensors \
  --tokenizer tokenizer.json \
  --vae flux2_ae.safetensors --vae-format flux2 \
  --rng cpu --cfg-scale 1.0 --steps 4 --seed 42 \
  -p "A woman in a red silk dress beside a window in a dim room at golden hour." \
  --ref-image source-768.png -W 3072 -H 3072 \
  --diffusion-fa -o out.png

--steps 1 on the same command shows the block corruption instead of a black
frame.

Where the threshold is

One denoising step, same reference and seed, share of pixels the model returns
as zero where the reference is not dark:

output	patch tokens (side/8)²	zeroed at --steps 1	--steps 4
1536	36 864	0.11 %	clean
2048	65 536	0.02 %	clean
2560	102 400	0.05 %	clean
2688	112 896	0.07 %	black
3072	147 456	5.98 %	black
4096	262 144	8.30 %	black

(The sub-0.1 % readings are the measurement's own floor — genuinely dark
pixels in the picture.) So one step is still clean at 2688 while four steps are
already black there, and by 3072 the very first step is visibly corrupt.

What you expected to happen

I'd expect a fine scaled image representing the base image

What actually happened

PiD is the one model where flash attention is not optional: without
--diffusion-fa a single 1024→4096 pass cannot run at all on this machine —

[WARN ] model_manager.cpp:1796 - model manager cannot make enough memory available on ROCm0:
        need 400225.12 MB device / 399713.12 MB budget, available 62954.86 MB device / unlimited budget
[ERROR] ggml_runner.cpp:871  - PiD segment 1/17 (pid.patch_blocks.0) failed during weight preparation

With --diffusion-fa the memory fits and the run completes normally (457 s for
1024→4096, exit 0, no error reported) — but the saved image is uniformly
black, mean 0 and standard deviation 0 on every channel.

Cutting the run to --steps 1 shows what is actually going wrong: the image
comes out, and it is peppered with 8×8-pixel blocks zeroed to black, i.e.
whole patch tokens returning zeros. They cluster in the brightest part of the
frame (a sunlit window), which suggests the failure is magnitude dependent.
With 2 steps or more, those zeros feed back into the next step and the whole
frame collapses to black.

attachment
pid-3072-1step-full.jpg	768→3072, --steps 1: the corruption across the frame
pid-3072-1step-crop-1to1.jpg	same image at 1:1 — the 8×8 block structure
pid-2560-1step-full-clean.jpg	640→2560, --steps 1: clean, for contrast

Measured over the whole frame, 98.4 % of 8×8 blocks are pure (either wholly
zeroed or wholly clean) and block boundaries fall exactly on multiples of 8,
so it is whole tokens being lost, not stray pixels.

Logs / error messages / stack trace

No response

Additional context / environment details

What it is not

• Not the overflow in docs/troubleshooting.md. At 768→3072 the frame is
  black for --attn-scale + --linear-scale at 0.00390625, at 0.000244140625,
  and for --attn-scale 1. The option is reaching the model — the single-step
  image does change, RMSE 0.027 against the unscaled run — but the corrupted
  share is identical: 5.98 % without it, 6.04 % with it.
• Not the sampler. lcm (the model's default) and euler both give the
  same black frame at 3072.
• Not a VRAM problem. Nothing is reported as failing; the run exits 0, and
  the same sizes without --diffusion-fa fail loudly with the 400 GB message
  above instead.
• Not reported anywhere. No [ERROR] line, and the process exits 0; the
  only sign is the image itself.

I could not compare against a non-flash-attention run at these sizes, because
without --diffusion-fa they do not run.

Environment

• sd.cpp: release master-881-17860c0 (prebuilt Linux ROCm archive), sd-cli
• Backend: ROCm / HIP, libggml-hip.so
• GPU: AMD Radeon 8060S Graphics, gfx1151 (Ryzen AI MAX+ 395), 124 GB unified
• ROCm: 7.14 (TheRock gfx1151 build)
• OS: Linux 7.2.6 (CachyOS)
• Model: Comfy-Org/PixelDiT → diffusion_models/pid_1.5_flux2_1024_to_4096_4step_bf16.safetensors
  (added in feat: add PiD 1.5 support #1790), text encoder text_encoders/gemma_2_2b_it_elm_bf16.safetensors,
  tokenizer unsloth/gemma-2-2b-it tokenizer.json, VAE nvidia/PiD
checkpoints/flux2_ae.safetensors, --vae-format flux2

(Issue written with the help of Claude)

[larochef]

After some more time debugging, here are what Claude found:

Cause: on AMD/HIP, ggml's flash-attention tile kernel keeps its softmax·V accumulator (VKQ) in f16. It ignores the GGML_PREC_F32 that sd.cpp requests.

and he also stated:

Verified fix: accumulating VKQ in f32 removes every black token at 3072² and 4096². It passes ggml's FA test suite and was ~10 % faster on gfx1151.

Is there anything I can do to help getting this fixed?

[larochef]

Another interesting fact he gave me:

Before ggml ec7bd5c9 (llama.cpp#15769, "CUDA: faster tile FA (Pascal/AMD)", 2025-09-07), the dispatcher picked a separate TILE_F32 kernel whenever prec == GGML_PREC_F32. That commit merged the f16 and f32 kernels and moved the choice to compile time.