* qwen4exp : build the QSA selection without the kq_mask
* clean
* lang
* qwen4exp : reshape the fused QSA selection to the kq_mask streams
ggml_top_k_mask with a single keep row returns [n_kv, n_tokens], while
build_attn_qsa expects the kq_mask layout [n_kv, n_tokens/n_stream, 1, n_stream].
With several streams the shape assert failed at graph reserve.
* test the QSA optimization
* reword
* update build_fused_top_k_mask instead of handwiring
* simplify
* keep il at the end for consistency
* hex-fusion: support for GELU+MUL, REPEAT+GET_ROWS and BINARY_NX (back-to-back ADD/MUL/...)
* hex-fusion: factor out fusion logic into separate .h and .cpp and update all related code
* hex-fusion: allow for RMS_NORM+MUL fusions for non-contig tensors
* hex-fusion: streamline fusion eligibility checks and cover missing corner cases
* hex-fusion: streamline fusion flow
* hex-opbatch: make n_tensors limit more scalable
* hex-bufs: adjust the trace buff sizing
* hex-glu: cleanup glu-ops interfaces and add kernel_params
* hex-kparams: clean up kparams handling in unary, ssm-conv, and other ops
* hex-kparams: update the rest of the ops PAD,DIAG,... to use kparams
* hex-unary: add support for collapsing dims
* hex-ops: cleanup redundant runtime checks
* hex-ops: consistent use of kernel_type in all kparams
* hex-fusion: remove redundant check in gdn+cpy
* hex-fusion: further streamlining of fusion rules and code
* hex-fusion: cleanup naming
* hex-opfmt: cleanup op formatting
* hex-fusion: bump fusion lookahead to 64
* hex-fusion: fuse SSM_CONV+SILU
* hex-fusion: fix hvx register spills in fused ssm_conv+silu
* hex-fusion: add more GLU fusions
* hex-fusion: remove duplicate checks and use ggml-common helpers
* hex-kparams: refactor kparams into htp-kparams.h/cpp
* hex-fusion: fix binary-nx fusion regression
* hex-kparams: another pass at cleaning up kparams (removing unused, etc)
* hex-kparams: streamling nameing and move _check_ wrappers into .h
* hex-kparams: templetize htp_kparams_check_<op> to reduce code bloat
* hex-ops: de-duplicate op dispatch switches
* hex-kparams: trim down exposed kparams calls
* hex-kparams: simplify type casting of kparams structs
* hex-fence: de-dup fence tensor construction
* hex-opbatch: simplify tensor shape init/checks
* hex-mm: cleanup leftovers of the original QK defines
* hex-fa: fix stack overflow on older npu firmware
* hex-ctx: move dma_caches into ops_ctx
* Multi-column on Intel XMX matrix engines refactored
Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0).
* Fixed regression bug
real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4
* Removed GGML_TYPE_Q8_0 activation
* rewritten xmx_mul_mat
64-byte alignment and 16-byte pitch check on the activations
* Implemented fused gate/up kernel
One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types.
* Removed the temporary XMX_NOGLU environment switch
* Q2_K min-sum optimization
Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group
* Added dispatch check and GRF sweep
tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%).
Q6_K is unchanged.
MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361.
Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without.
* Renamed variables
* Added 2 environment variables
GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16): the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it.
* Update esimd.hpp
1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces.
2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once.
The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.
* llama: create new fused op for QSA scoring
* enroll glm5.3-next too
* tests : initialize the remaining tensors in test_kpool_mask
The harness sentinels were left uninitialized, so leftover memory could read as NaN and fail the NaN check.
* ggml: generalize the k-pool mask op into ggml_top_k_mask
The k-pool mask op only covered the pooled QSA selection of qwen4exp
and glm5-next, while the DSA models and the DeepSeek V4 CSA build the
same top-k mask by hand with fill, set_rows and add.
ggml_top_k_mask(mask, top_k, score, blocks, extra) keeps the mask
cells of the live picks and the extra cells, and sets the rest to
-inf. Every row tensor is 4D in the layout of top_k, and the mask is
broadcast to those rows, so a selection can also be made per head.
score, blocks and extra are optional: without score every pick is
live, without blocks a pick is a cell, and extra holds the cells a
row keeps without a pick.
llm_graph_context::build_fused_top_k_mask views the row tensors in
the layout of kq_mask and registers the fused node. The k-pool
models, the DSA attention of build_attn and the DeepSeek V4 CSA mask
use it when the probe enables the op, and keep their current path on
backends without it.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table
* ggml-cpu : keep the GGML_GELU_QUICK_FP16 table path
* ggml-cpu : add the VXE gelu_quick path
* tests : add test-gelu-quick for the f32 CPU kernels
* tests : drop inf/nan inputs from test-gelu-quick
* metal : optimize argmax and pool_1d
Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : add pool_1d perf cases
Add representative POOL_1D max cases for the MoE down-projection amax shapes, reducing the whole intermediate dim.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : minor fix
* ggml: support non-contiguous SUM (CPU, CUDA)
* tests: add non-contiguous SUM cases
* tests: SUM over non-contiguous views
* vulkan: support non-contiguous SUM
* webgpu: support non-contiguous SUM
* fuse grouped experts
* add tests
* hip fix
* add to graph_optimize
* apply suggestion from am17an
* hip fix
* fix hip fix
* fix fix fix
* gate on hip version instead
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round
Add support for GET_ROWS with tiled Q5_K
Add support for raw k-quant tensors
* hex-get-rows: vectorize raw k-quant ukernels
* hex-binary: refactor dma-loop thread functions (no need for macros, etc)
* hex-quant: move/cleanup flat and tiled dequant ukernels
* hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights
* hex-get-rows: remove redundant cast
* hex-repack: proper filtering for BF16 that were leaking through during the tests
* hex-repack: streamling tiling rules (broken MM for now)
* hex-repack: fix mmid after tiling updates
* hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars
* hex-mm: fix valid_rows calc in various kernels
* hex-mm: remove unused hvx_mm_nx_2d_flat
* hex-mm: cleanup and optimize hvx-tiled vec_dot_f16
* hex-mm: another pass at hvx-tiled f16 matmul
* hex-get-rows: optimize small src1
* hex-get-rows: one more pass at recovering get-rows perf regression
* hex-repack: export GGML_HEXAGON_RT_REPACK via run.py
* hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul
* hex-mm: fold tiled_f16 checks into the general flow
* hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op
* hex-quant: proper tail handling in new dequant ukernels
* hex-get-rows: remove unused code and cleanup support checks
* hex-mm: cleanup ne1 padding/tiling checks
* hex-mm: cleanup fusion checks for tiled tensors
* hex-repack: add a little can_repack helper to reduce duplication
* hex-repack: never repack host buffers
* hex-repack: factor out repack functions into separate .h and .cpp
* hex-tensor: cleanup tensor ifaces for tiled/padded sizes
* hex-repack: update k-quant tiling comments and cleanup #defines
* hex-dcache: factor out l2/dcache dirty tracking and flushing
* hex-tensor: update docs and remove #defs duplication
* hex-tensor: remove unused/incorrect get_data interface
* hex-get-rows: cleanup vtcm layout build
* tests: disable huge MM tests
* hex-dcache: add missing header
* vulkan : copy strided f16 KV once so reads spread across memory channels
Assisted-by: Claude (Opus 4.8 / Opus 5)
* tests : cover the f16 KV copy to scratch
Assisted-by: Claude (Opus 4.8 / Opus 5)
* vulkan : gate the f16 KV scratch copy to AMD
Assisted-by: Claude (Opus 4.8 / Opus 5)
* vulkan : reuse the generic copy shader for the f16 KV scratch fill
The f16 clause needs only a strided copy, not a dequant, so drop the
dedicated shader and dispatch the existing copy pipeline instead. The
copy is described in source memory order so the strided access lands on
the writes, which stream, instead of the reads, which would concentrate
on a few memory channels.
Assisted-by: Claude (Opus 4.8 / Opus 5 / Fable 5)
* vulkan : extend the strided KV scratch copy to bf16
Assisted-by: Claude Opus 5
* vulkan : clean up comments
Assisted-by: Claude Opus 5
* consolidate copy shaders
* disable on RDNA4
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* cpu: add F16 SWIGLU_OAI reference implementation and F16 activation op test coverage
Adds ggml_compute_forward_swiglu_oai_f16() to the CPU backend, mirroring
the existing F32 path, and extends test-backend-ops with F16 test cases
for SWIGLU_OAI so downstream backends(e.g. Hexagon) can be CPU-diffed
for correctness.
* ggml-openvino: fix f16 SWIGLU_OAI type mismatch in alpha/one constants
translate_glu_swiglu_oai creates alpha_const and one as f32 constants
unconditionally, so f16 inputs hit a type mismatch in Swish/Add on GPU
("Swish inputs must have the same type but they are: f16 and f32").
translate_glu_swiglu_clamp already upcasts to f32 and converts back at
the end; apply the same pattern here.
---------
Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
* cuda, hip: GDN chunked kernel
Ported-from: https://github.com/tetherto/qvac-fabric-llm.cpp/pull/257
Source-commit: 53f0027d8070715fe9ad2131f3201bb0001d4eed
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
* fix older builds
* review fixes
* cuda: enable GDN MMA on tested Ampere and Ada GPUs
Enable SM86 and SM89 after RTX 3090 and RTX 4090 correctness and prefill benchmarks. Keep other untested architectures on the existing fallback.
Assisted-by: Codex
* cuda : use refactored swizzle API in GDN MMA
* ggml : GDN only guarantees the final state when n_tokens > K
* cuda : add short GDN MMA phase comments
---------
Co-authored-by: gianni-cor <gianfranco.cordella@tether.io>
Co-authored-by: Aman Gupta <aman.gupta@tether.io>
* vulkan : split matmul dispatch when it would exceed maxComputeWorkGroupCount
* vulkan : fix narrowing in chunked matmul dispatch
* vulkan : zero fill the A side of mul_mmq for k steps past the end
The A side skipped the shared memory write for the k steps past end_k, while the B side was already told the bound and zero filled. The A registers are read for all BK_STEP steps, so the stale scale went into the dot product and produced NaN at k = 32.
Pass the bound to block_a_to_shmem and write zeros when it is false, for every DATA_A type, mirroring the B side.
Also assert that the B and D subbuffer sizes are not VK_WHOLE_SIZE before they are adjusted per chunk, and add two batched cases of the chunked shape.
Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
* vulkan : remove redundant matmul tile adjustment
---------
Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
* tests: cover weighted scalar broadcasts
Assisted-by: OpenCode
* ggml : support extended scalar DMA binary ops
Assisted-by: OpenCode
* hex-binary: remove all non-dma paths
* hex-binary: add support for non-contig strides and cleanup dma preable a bit
* hex-binary: cleanup the gating and simplify dma retry
* hex-geglu-erf: add a gate for F32 only, we dont support F16 for this op
* tests: cover weighted repeat broadcast
Assisted-by: OpenCode
* hex-binary: update REPEAT and COMPLEX kernels to use dma as well (missed them last time)
* hex-add-id: add dma support for src2 (idx)
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* vulkan: compute every row of an expert in mul_mm_id when ids repeat
* vulkan: handle mul_mat_id duplicates in the prepass rather than looping
Extend the "hoist row ids" optimization to always be enabled and to emit a
compact list of tile descriptions that need to run, and to emit multiple tiles
when needed. Then we can launch a tighter upper bound on the number of
workgroups and the tail can trivially early exit.
---------
Co-authored-by: Anjielon <[EMAIL_REDACTED]>
* CUDA: fuse copy of updated state snapshots into recurrent cache with ssm_scan
* CUDA: remove redundant cuda copies with K==1 (non spec-dec) scenario as well
* hexagon: fix IM2COL patch-embed DMA ring overflow
The exact-tiling (stride == kernel, no pad/dilation) IM2COL DMA kernel
issues IC*KH DDR->VTCM descriptors per output row without checking the
return value of dma_queue_push(), and then pops IC*KH times. The per-thread
DMA ring holds 256 entries and a push into a full ring returns false
and drops the transfer, so for IC*KH > 255 the remaining rows of the
VTCM staging buffer were never written and stale data (often NaN/inf)
leaked into the output.
Solution is to retire the oldest descriptor when the ring is full, just as the blocked
kernel in the same file already does, and wait with dma_queue_flush().
For testing, added exact-tiling test cases with IC*KH > 256 (2D 1x1, 2D 2x2 patch
embed and 1D, F16 and F32 dst), which fail on HTP without this fix.
* Apply suggestion from @max-krasnyansky
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* CUDA: radix top-k for large row counts
Replaces CUB's per-row DeviceTopKKernel with a grid-over-rows radix select,
gated on GGML_CUDA_TOPK_RADIX_MIN_ROWS. On qwen4exp at 34,816 tokens this cuts
top-k from 1,671,253 launches / 5,761.8 ms to 2,329 / 941.8 ms.
* CUDA: select the TOP_K implementation by shape
Replace the nrows/ncols special case with the decision boundary from #28547
(as implemented in #29278): bitonic for short rows, radix select for several
long rows, and DeviceTopK or CUB argsort for a single long row. The
thresholds stay overridable at build time.
Two refinements on top of that boundary:
- bitonic stays in use for rows up to a padded 1024 while the rows fit in one
wave of blocks (nrows <= number of SMs); radix select pays a fixed cost of
about a dozen launches that only amortizes over more rows
- with DeviceTopK available, it handles up to two rows
Radix select now processes rows in chunks so its scratch memory stays bounded,
and the bitonic path keeps its chunking. HIP and MUSA keep their previous
thresholds.
Add perf cases around the bitonic/radix crossover to test-backend-ops.
* CUDA: make top-k comments less verbose
* CUDA: remove the TOP_K width limit from supports_op
* CUDA: use DeviceTopK for single-row TOP_K if available
* CUDA: avoid ncols overflow in the TOP_K bitonic check
* CUDA: share the row chunking helper between argsort and top-k
* CUDA: do the TOP_K radix blocks_per_row math in int64_t
* CUDA: rename GGML_CUDA_TOP_K_NROWS_THRESHOLD_DEVICETOPK to GGML_CUDA_TOP_K_NROWS_THRESHOLD
* CUDA: share one sort helper between the bitonic and CUB TOP_K paths
* CUDA: update the TOP_K TODO, threshold and chunking comments
* tests: add TOP_K cases that span several row chunks
* CUDA: use int64_t col in the TOP_K radix loops, fix threshold comment
* CUDA: limit TOP_K and ARGSORT support to ne[0] <= INT_MAX
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
The bucket search in topk_nary_search.comp started from the range
[0, 0xFF800000), which ends just below the ordered-uint mapping of +inf,
so +inf and NaN were never counted. A workgroup block with fewer than k
countable values left the ballot empty and the shader read uninitialized
shared state (hang/device lost on NVIDIA, wrong indices on AMD), and a few
+inf in a block were selected without being counted, dropping real top
values.
Map NaN to -inf on input, start from [0, 0xFFFFFFFF) so every value is
counted, and clamp the top bucket's end (2^32) instead of wrapping to 0.
The k = 1 path compared float bits as signed integers, which orders
negative values backwards; compare floats instead.
Add test_top_k_inf to test-backend-ops: negative values, fewer than k
+inf and many -inf, for k = 1, 10, 40.
Assisted-by: Claude Opus 5.5
* sycl: fuse the delta-net alpha gate (add + unary + mul)
* tests: cover the fused add + unary + mul chain
* sycl: give the fused alpha gate a flat path and pin the node skip
The generic few-row MMA kernel works for any type with a 16-weight
dequantizer, so it now also takes BF16, Q1_0, Q2_0, MXFP4, Q2_K, Q3_K,
TQ2_0 and the IQ types. Each type starts at the row count where it beats
the current kernels on an M3 Ultra: 5 rows for TQ2_0, 4 for BF16, 3
for MXFP4, Q2_0, Q2_K and IQ4_NL, and 2 for the others.
test-backend-ops perf -o MUL_MAT, m=4096, k=14336, M3 Ultra, time of this
change over master (mean of two interleaved runs each): 0.23 to 0.98 from
the threshold to 8 rows, 0.24 to 0.33 at 9 to 16 rows, and 0.99 to 1.01 at
1 and 512 rows.
* metal : fix MUL_MAT+ADD fusion when the residual is itself a MUL_MAT
ggml_metal_op_mul_mat_mma picks the residual of a fused MUL_MAT+ADD as
"the ADD operand whose op is not MUL_MAT". When both operands of the ADD
are mat-mul outputs (x = W1 @ u + W2 @ v), that test is true for both, so
the residual resolves to the fused mat-mul's own, never-written output and
the kernel adds whatever that buffer holds.
The fusion check (ggml_metal_mul_mat_add_operand) already selects the
operand by identity; make the encoder do the same.
Clef decision models hit this in their head (proj_option_context @ ctx +
proj_option_lexical @ lex, 9 option rows): on Metal, /v1/systemone
probabilities collapse toward uniform (billing 0.28 where the CPU backend
gives 0.977, Cloudflare_clef-flash Q8_0), deterministic per memory layout,
correct with GGML_METAL_FUSION_DISABLE=1. Not a quantization issue: the
same file is right on CPU.
Add a MUL_MAT_ADD mode to test-backend-ops where the residual is a second
mat-mul; on Metal it fails 27 of 28 cases before this change (the one pass
is f16 n=2, under the MMA row threshold, so nothing fuses).
* Update tests/test-backend-ops.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
The XIELU CUDA kernel template is already generic over the element
type; only the F32/F16 type assertion and the else-if dispatch were
missing. Add the nv_bfloat16 branch to the launcher, and drop the
temporary supports_op gate in ggml-cuda.cu that rejected BF16+XIELU.
test-backend-ops gains two BF16 cases ([10,5,4,3] and [512,16,1,1]).
docs/ops/CUDA.csv and docs/ops.md are regenerated; the F32 xIELU row
flips from no to yes as well, i.e. the previous record was stale.
Tested:
- Mac CPU: xIELU F32/F16/BF16, 6/6
- Mac Metal: existing F32/F16, 4/4; BF16 still unsupported
- RTX 4090 CUDA: xIELU F32/F16/BF16, 6/6
- RTX 4090 CUDA BF16-only: 2/2
- git diff --check passes
* ggml: fix CLAMP on non-contiguous views (CPU, CUDA)
CUDA clamped ggml_nelements values flat and ignored the view strides.
CPU addressed row j as j*nb01 and ignored nb02/nb03. Both now follow the
strides of dims 1..3; CUDA supports_op requires contiguous rows, like
Metal. test_clamp gains a non-contiguous view case.
* cuda: clamp kernel uses fastdiv for the view strides
* vulkan: sparse flash attention for quantized K/V
Assisted-by: Claude
* vulkan: single-scan sparse FA index compaction
The compaction ran one workgroup per mask row and walked the row in
BLOCK_SIZE chunks, with a workgroup scan per chunk. For decode that is
one workgroup doing KV/1024 barrier-bound iterations, so at 128k cells
it cost more than the sparse attention it feeds.
Split the row into contiguous segments instead: one per subgroup with
ballot counting over coalesced loads, or one per thread without
subgroups. A single scan over the segment counts then gives each
segment its output offset. The index list stays ascending.
* cuda: tile the lightning indexer over keys and tokens for 4 heads
With too few heads for a wmma tile, a block scores 64 keys against 8
tokens: the keys are staged once in half precision, the queries one
head at a time, and each thread owns one key for two tokens, so no dot
product needs a cross thread reduction. Batches smaller than a token
tile keep the vector kernel. test-backend-ops measures 4 heads.
* cuda: multiply the lightning indexer tile in float
Address review from am17an: the half2 products overflow once a single
q * k exceeds the f16 range. The queries stay in float in shared memory
and each half2 of keys is widened once for both tokens, so every
product and sum is computed in float.
* cuda: widen each lightning indexer key once for all heads
The tile kernel stages the queries and weights of every head at once,
so each key element is widened from half once and feeds all heads,
with a single barrier. F16 keys are copied into the tile without a
float round trip. Keeping the keys in float in shared memory measures
slower, the occupancy drops.
* cuda: stop the lightning indexer tile from spilling registers on ROCm
Each thread of the tile kernel now scores two keys for a single token,
so a warp shares its token and the query reads are broadcasts: six
shared reads per element pair instead of nine for the same products.
The inner loop is unrolled by 8, which keeps gfx908 at 63 VGPRs with no
spill where the fully unrolled loop needed over a thousand, and makes
the kernel 36x faster on an R9700 and slightly faster on CUDA.
* metal : few-row MMA mat-mul and batched copies for speculative decoding
Speculative decoding verifies a few draft tokens per step. Without the tensor API, Metal ran these mat-muls with the mat-vec kernels, whose time grows with every src1 row, so DFlash2 decoding on an M3 Ultra was slower than serial decoding.
- add mat-mul kernels for 2..16 src1 rows on 8x8 simdgroup matrices: each weight is dequantized once for all rows, and the simdgroups of a threadgroup split K. Q4_0, Q8_0 and Q5_K have their own kernels, F32, F16, Q4_1, Q5_0, Q5_1, Q4_K and Q6_K use a generic path over the 16-weight dequantizers, and Q4_0 at 2 rows uses a 2-row variant of the mat-vec kernel
- use them only on MTLGPUFamilyApple7+ without the tensor API, from the row count at which they beat the mat-vec kernels on an M3 Ultra (F32: 6, F16, Q4_K, Q5_0, Q5_1: 3, other types: 2)
- fusion table: MUL_MAT + ADD adds a same-shape residual in the MMA store, and up to 16 adjacent same-layout f32 copies between the same two tensors run as one dispatch
- the fusion checks and ggml_graph_optimize take the device props, so the reorder packs MUL_MAT + ADD only on devices that can fuse it, at every src1 row count
- views do not count toward GGML_METAL_FUSION_MAX when the reorder packs a group, so 16 recurrent state snapshot copies with views between them stay one group
- the encoder checks the inner nodes of a fused group for concurrency, tracks written views by their extent, and does not count the destination of a CPY as a read
- CONCAT splits long rows across threadgroups when there are few rows
- tests: few-row MUL_MAT, MUL_MAT_ADD, CPY_BATCH and CONCAT cases in test-backend-ops (with a prepare_graph hook for the copy order), test-metal-graph-optimize, test-metal-cpy-batch-alias
* metal : remove the CPY_BATCH fusion and the memory range changes
Remove the batched copy fusion with its kernel and tests, and revert the
memory range changes, as suggested in review. The memory ranges, the
graph reorder and the CPY encoder are again the same as on master.
* cont : clean-up
* cont : drop has_tensor gate
* cont : clean-up operand/residual logic
* cont : drop Q4_0 ne11=2 special-case
* cont : add kernels/mul_mv_mma.metal
* cont : consolidate mma pipeline selection logic
* cont : decouple fusion logic from device props
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml-cpu: vectorize BF16 K tails in tinyBLAS
* tests: Skip tinyBLAS when use_ref is enabled so CPU tests compare against the vec_dot path.
* ggml-cpu: vectorize tinyBLAS F16/F32 tails
* qwen4exp : halve the indexer score memory
The indexer scored all heads in one product and rectified a copy of it,
so two [n_pool, n_idx_h, n_tokens] f32 tensors were live at once, the
largest buffers of the graph at long context. Each head now gets its
own product, rectified and summed in place into one [n_pool, n_tokens]
score.
* qwen4exp: let the allocator reuse the indexer score buffers
Address review from CISC: use plain ggml_add and ggml_relu in the
indexer head loop. The graph allocator already runs them in place when
their source has no other consumer, so the _inplace variants are not
needed. The compute buffer and the speed are unchanged.
* cuda: support 4 heads in the lightning indexer
Dispatch 4 heads to the vector kernel, too few for a wmma tile, and
accept them in supports_op. test-backend-ops covers 4 heads.
* metal: take the lightning indexer head count as a function constant
The kernel reads the head count from a function constant and zero fills
the last head tile, so any head count runs and 64 heads is unchanged.
* qwen4exp: compute the indexer score with the lightning indexer
Address review from am17an: the unweighted sum of the rectified head
scores scaled by 1/sqrt(head_dim) is the lightning indexer with every
head weight set to that scale, so the indexer calls
ggml_lightning_indexer on the pooled keys with an f16 pool mask. The
keys are read once for all heads and no per head score is
materialized.
* vulkan: tile the lightning indexer over keys and tokens
A workgroup scores 64 keys against 8 tokens: the keys are staged once
in shared memory, the queries one head at a time, and each invocation
owns one key for two tokens, so no dot product needs a cross invocation
reduction. The subgroup variant and the flat dispatch are gone, the grid
is keys x tokens x streams.
* vectorize vulkan loads and use fp16 dot product
---------
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
* metal : add tensor API flash attention kernel for F16 KV
* cont : add tensor FA kernels for DK=DV=512 and DK=576, DV=512
* cont : support attention sinks, ALiBi and logit softcap in the tensor FA kernel
* cont : add tensor FA kernel for DK=192, DV=128
