[feat](sft): support Qwen VLM MoE fine-tuning (#2156)

doc/en/SFT/KTransformers-VLM-LoRA-Fine-Tuning-Guide.md

@@ -0,0 +1,279 @@
+# Heterogeneous VLM LoRA Fine-Tuning with KTransformers and LLaMA-Factory
+
+Last updated: 2026-08-14
+
+This guide covers MoE VLM LoRA SFT with LLaMA-Factory for training and
+KTransformers for CPU/GPU heterogeneous expert execution. The workflow has been
+validated with Qwen3-VL-30B-A3B-Instruct and Qwen3.5-35B-A3B.
+
+KTransformers is optional. Use LLaMA-Factory alone for dense VLMs, pure-GPU
+training, or full-parameter fine-tuning.
+
+## Quick Start
+
+### 1. Prerequisites
+
+- Linux, NVIDIA GPUs, and a working CUDA driver
+- Python 3.11
+- An AVX-512 or AMX CPU recommended for KT expert execution
+- Sufficient GPU memory, CPU memory, and storage for the selected model
+- LLaMA-Factory with `requirements/ktransformers.txt` and the Qwen3-VL KT example
+- A KTransformers release with Qwen3-VL MoE and instance-scoped VLM Conv3D support
+
+### 2. Create the environment
+
+```bash
+conda create -n kt-vlm-lora python=3.11 -y
+conda activate kt-vlm-lora
+
+pip install \
+  --extra-index-url https://download.pytorch.org/whl/cu130 \
+  torch==2.9.1 \
+  torchvision==0.24.1 \
+  torchaudio==2.9.1
+```
+
+KT SFT pins torch 2.9.1. Do not mix the torch 2.10 pure-GPU stack into this
+environment.
+
+### 3. Install LLaMA-Factory and KT
+
+```bash
+git clone https://github.com/Illumination111/LlamaFactory.git
+cd LlamaFactory
+pip install -e .
+pip install -r requirements/ktransformers.txt
+pip check
+```
+
+`requirements/ktransformers.txt` should contain:
+
+```text
+ktransformers[sft]
+```
+
+Install in this order so the KT-enabled Transformers and Accelerate modules are
+applied after the base LLaMA-Factory dependencies.
+
+When building KTransformers from source, clone its submodules as well:
+
+```bash
+git clone --recursive https://github.com/kvcache-ai/ktransformers.git
+# For an existing checkout:
+git submodule update --init --recursive
+```
+
+Without the pinned `llama.cpp` and `pybind11` submodules, `kt-kernel` cannot be
+built.
+
+### 4. Verify the installation
+
+```bash
+python - <<'PY'
+import importlib.metadata as md
+import accelerate, kt_kernel, torch, transformers
+from accelerate.utils.dataclasses import KTransformersPlugin
+from kt_kernel.sft.conv3d_compat import is_vlm_conv3d_compatible, patch_vlm_conv3d
+from transformers.integrations.kt import is_kt_expert_loading_enabled
+
+print("torch                    =", torch.__version__)
+print("transformers dist/module =", md.version("transformers"), transformers.__version__)
+print("accelerate dist/module   =", md.version("accelerate"), accelerate.__version__)
+print("transformers-kt          =", md.version("transformers-kt"))
+print("accelerate-kt            =", md.version("accelerate-kt"))
+print("kt-kernel                =", kt_kernel.__version__)
+print("KT plugin                =", KTransformersPlugin.__name__)
+print("Conv3D API               =", patch_vlm_conv3d.__name__, is_vlm_conv3d_compatible.__name__)
+print("Transformers KT hook     =", is_kt_expert_loading_enabled.__name__)
+PY
+
+pip check
+```
+
+The KT-enabled modules may report different module and distribution versions
+because the current KT fork wheels provide the `transformers` and `accelerate`
+import packages under separate distribution names. The capability imports above
+are the authoritative installation check.
+
+### 5. Configure and launch training
+
+Start from:
+
+```text
+examples/ktransformers/train_lora/qwen3vlmoe_lora_sft_kt.yaml
+```
+
+Review at least these fields:
+
+```yaml
+model_name_or_path: Qwen/Qwen3-VL-30B-A3B-Instruct
+image_max_pixels: 262144
+video_max_pixels: 16384
+
+stage: sft
+do_train: true
+finetuning_type: lora
+lora_rank: 8
+lora_alpha: 16
+lora_target: all
+
+dataset: mllm_demo
+template: qwen3_vl
+cutoff_len: 512
+
+use_kt: true
+```
+
+For Qwen3.5 VLM, change the model path, use `template: qwen3_5`, and select a
+multimodal dataset. The existing Qwen3.5 KT YAML uses text data by default.
+
+Set `num_processes` in
+`examples/ktransformers/accelerate/fsdp2_kt_bf16.yaml` to the visible GPU count,
+then run:
+
+```bash
+CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch \
+  --config_file examples/ktransformers/accelerate/fsdp2_kt_bf16.yaml \
+  src/train.py \
+  examples/ktransformers/train_lora/qwen3vlmoe_lora_sft_kt.yaml
+```
+
+Use the BF16 configuration for original BF16 checkpoints. INT8/INT4 KT configs
+require matching converted expert weights and `kt_weight_path`.
+
+### 6. Check the result
+
+A successful smoke test should show:
+
+- finite loss and a finite, nonzero gradient norm;
+- `Wrapped ... MoE layers with KTMoEWrapper` in the log;
+- Conv3D compatibility activation under torch 2.9;
+- regular PEFT adapter files and `fused_expert_lora.safetensors`;
+- nonzero LoRA-B values after an optimizer step.
+
+## Full Documentation
+
+### Integration boundary
+
+LLaMA-Factory owns multimodal preprocessing, templates, FSDP, LoRA configuration,
+training, and checkpoints. With `use_kt: true`, KTransformers wraps supported MoE
+experts, runs them on CPU backends, and stores their fused LoRA tensors.
+
+The VLM-specific KT code only:
+
+1. Maps Qwen3-VL MoE experts and their checkpoint prefix.
+2. On torch 2.9.x, patches only the supported Conv3D instances on the loaded KT
+   VLM and marks them after validating the operator contract.
+
+It is not used when `use_kt: false`.
+
+### Models and LoRA scope
+
+| Model | Template | Notes |
+| --- | --- | --- |
+| Qwen3-VL-30B-A3B-Instruct | `qwen3_vl` | Reference KT VLM example |
+| Qwen3.5-35B-A3B | `qwen3_5` | Requires multimodal data settings |
+
+Use LLaMA-Factory's native VLM freeze flags:
+
+| LoRA scope | `freeze_vision_tower` | `freeze_multi_modal_projector` | `freeze_language_model` |
+| --- | ---: | ---: | ---: |
+| Text only | `true` | `true` | `false` |
+| Vision only | `false` | `true` | `true` |
+| Text and vision | `false` | `true` | `false` |
+
+With `lora_target: all`, LLaMA-Factory excludes the multimodal projector from
+automatic target discovery. Projector training requires explicit compatible
+targets and `freeze_multi_modal_projector: false`.
+
+The KT path supports `finetuning_type: lora`. Use pure-GPU LLaMA-Factory with
+`use_kt: false` for full-parameter VLM fine-tuning.
+
+### Data and resource settings
+
+`mllm_demo` is suitable for a smoke test. Register production multimodal data in
+`data/dataset_info.json` using a format supported by the selected LLaMA-Factory
+revision, and verify that image/video paths resolve correctly.
+
+In the FSDP config, keep `kt_config.lora_rank` equal to the training YAML's
+`lora_rank`. Tune `num_processes`, KT CPU thread counts, pixel limits,
+`cutoff_len`, batch size, and gradient accumulation for the host.
+
+The validation setup was eight RTX 4090 48 GB GPUs, an Intel Xeon Platinum 8488C,
+and 2 TiB RAM. It is not a minimum hardware requirement.
+
+### Checkpoints
+
+KT saves `fused_expert_lora.safetensors` alongside the normal PEFT adapter files.
+Keep both together when copying or resuming a checkpoint.
+
+### End-to-end validation
+
+One-step BF16 FSDP2 tests with LoRA rank 2 and real images completed on
+2026-08-13:
+
+| Model | Loss | Gradient norm | Runtime | KT evidence |
+| --- | ---: | ---: | ---: | --- |
+| Qwen3-VL-30B-A3B-Instruct | 13.6875 | 7.493 | 9.42 s | 48 MoE layers; 288 fused LoRA tensors |
+| Qwen3.5-35B-A3B | 1.6299 | 0.6675 | 14.10 s | 40 MoE layers; 240 fused LoRA tensors |
+
+Both runs completed multimodal preprocessing, distributed loading,
+forward/backward, optimizer update, and adapter saving. These are functional
+checks, not convergence or performance benchmarks.
+
+### Troubleshooting
+
+#### `ValueError: unknown keys (['kt_config'])`
+
+The KT-enabled Accelerate build is not active. Reinstall in order:
+
+```bash
+pip install -e .
+pip install -r requirements/ktransformers.txt
+```
+
+#### `kt_kernel.sft.conv3d_compat` is missing
+
+Reinstall `kt-kernel` from a KTransformers revision containing VLM support.
+
+#### The KT Conv3D compatibility check fails
+
+The instance-scoped KT fallback rejects Conv3D modules whose stride, padding,
+dilation, or groups are unsupported. No ms-swift installation is required.
+
+#### Qwen3.5 cannot import `causal_conv1d_cuda`
+
+Either uninstall the optional package or rebuild it after installing the final
+torch version:
+
+```bash
+MAX_JOBS=16 pip install \
+  causal-conv1d==1.6.2.post1 \
+  --force-reinstall \
+  --no-deps \
+  --no-build-isolation
+```
+
+#### CPU or GPU memory is exhausted
+
+Reduce pixel limits, `cutoff_len`, and batch size. Quantized KT configurations
+also reduce memory use but require correctly converted expert weights.
+
+### Pure-GPU alternative
+
+In a separate clean environment, install LLaMA-Factory using its normal
+procedure, do not install `requirements/ktransformers.txt`, and set:
+
+```yaml
+use_kt: false
+```
+
+This uses LLaMA-Factory's existing VLM LoRA and full-parameter training paths
+without importing KTransformers.
+
+## Related documentation
+
+- [KT SFT Quick Start](./KTransformers-Fine-Tuning_Quick-Start.md)
+- [KT SFT User Guide](./KTransformers-Fine-Tuning_User-Guide.md)
+- [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory)
+- [KTransformers](https://github.com/kvcache-ai/ktransformers)

kt-kernel/python/sft/__init__.py

@@ -25,6 +25,7 @@ from .arch import (
     KTAMXConfigError,
 )
 from .autograd import KTMoEFunction
+from .conv3d_compat import is_vlm_conv3d_compatible, patch_vlm_conv3d
 from .layer import KTMoELayerWrapper
 from .weights import (
     extract_moe_weights,
@@ -69,6 +70,8 @@ __all__ = [
     "KTAMXModelNotSupportedError",
     "KTAMXConfigError",
     "KTMoEFunction",
+    "patch_vlm_conv3d",
+    "is_vlm_conv3d_compatible",
     "KTMoELayerWrapper",
     "extract_moe_weights",
     "load_experts_from_checkpoint_files",

kt-kernel/python/sft/arch.py

@@ -98,7 +98,7 @@ def get_moe_arch_config(config) -> MOEArchConfig:
             has_shared_experts=getattr(config, "n_shared_experts", 0) > 0,
             router_type="deepseek_gate",
         )
-    if "Qwen2Moe" in arch or "Qwen3Moe" in arch or "Qwen3_5Moe" in arch:
+    if "Qwen2Moe" in arch or "Qwen3Moe" in arch or "Qwen3VLMoe" in arch or "Qwen3_5Moe" in arch:
         cfg = getattr(config, "text_config", config)
         return MOEArchConfig(
             moe_layer_attr="mlp",
@@ -136,7 +136,8 @@ def get_moe_arch_config(config) -> MOEArchConfig:
 
     raise KTAMXModelNotSupportedError(
         f"Model architecture {arch} not supported for KT AMX. "
-        "Supported architectures: DeepseekV2, DeepseekV3, Qwen2Moe, Qwen3Moe, Qwen3_5Moe, Glm4Moe, Mixtral"
+        "Supported architectures: DeepseekV2, DeepseekV3, Qwen2Moe, Qwen3Moe, Qwen3VLMoe, "
+        "Qwen3_5Moe, Glm4Moe, Mixtral"
     )
 
 
@@ -167,7 +168,7 @@ def detect_fused_experts(experts: nn.Module) -> bool:
 
 def _get_layers_prefix(config) -> str:
     arch = config.architectures[0] if getattr(config, "architectures", None) else ""
-    if "Qwen3_5Moe" in arch:
+    if "Qwen3_5Moe" in arch or "Qwen3VLMoe" in arch:
         return "model.language_model.layers"
     return "model.layers"
 

kt-kernel/python/sft/conv3d_compat.py

@@ -0,0 +1,150 @@
+# SPDX-License-Identifier: Apache-2.0
+
+"""Instance-scoped Qwen VLM compatibility for the KT SFT stack."""
+
+from __future__ import annotations
+
+from types import MethodType
+from typing import Any
+
+from packaging.version import Version
+
+
+_SUPPORTED_MODEL_TYPES = {
+    "qwen2_vl",
+    "qwen2_5_vl",
+    "qwen3_vl",
+    "qwen3_vl_moe",
+    "qwen3_5",
+    "qwen3_5_moe",
+}
+_COMPATIBLE_ATTR = "_kt_conv3d_compatible"
+_ORIGINAL_FORWARD_ATTR = "_kt_original_conv3d_forward"
+
+
+def _canonicalize_qwen3_vl_fused_expert_weights(
+    gate_up: Any,
+    down: Any,
+    moe_config: Any,
+) -> tuple[Any, Any, Any] | None:
+    """Convert only the transposed Qwen3-VL fused expert layout to KT layout."""
+    expert_num = moe_config.expert_num
+    intermediate_size = moe_config.intermediate_size
+    if gate_up.dim() != 3 or down.dim() != 3:
+        return None
+
+    hidden_size = gate_up.shape[1]
+    if tuple(gate_up.shape) != (expert_num, hidden_size, 2 * intermediate_size):
+        return None
+    if tuple(down.shape) != (expert_num, intermediate_size, hidden_size):
+        return None
+
+    gate_proj, up_proj = (
+        tensor.transpose(1, 2).contiguous()
+        for tensor in gate_up.split(intermediate_size, dim=2)
+    )
+    return gate_proj, up_proj, down.transpose(1, 2).contiguous()
+
+
+def _requires_conv3d_patch() -> bool:
+    import torch
+
+    version = Version(torch.__version__.split("+", 1)[0])
+    return Version("2.9.0") <= version < Version("2.10.0")
+
+
+def _is_supported_vlm(config: Any) -> bool:
+    return (
+        getattr(config, "model_type", None) in _SUPPORTED_MODEL_TYPES
+        and getattr(config, "vision_config", None) is not None
+    )
+
+
+def _contract_errors(module: Any) -> list[str]:
+    errors = []
+    if tuple(module.stride) != tuple(module.kernel_size):
+        errors.append(f"stride={module.stride} != kernel_size={module.kernel_size}")
+    if any(value != 0 for value in module.padding):
+        errors.append(f"padding={module.padding} != 0")
+    if any(value != 1 for value in module.dilation):
+        errors.append(f"dilation={module.dilation} != 1")
+    if module.groups != 1:
+        errors.append(f"groups={module.groups} != 1")
+    return errors
+
+
+def _kt_conv3d_forward(module: Any, inputs: Any) -> Any:
+    """Evaluate the non-overlapping Conv3D contract using autograd-native ops."""
+    import torch.nn.functional as functional
+
+    batch_size = inputs.shape[0]
+    kernel = tuple(module.kernel_size)
+    inputs = (
+        inputs.unfold(2, kernel[0], kernel[0])
+        .unfold(3, kernel[1], kernel[1])
+        .unfold(4, kernel[2], kernel[2])
+    )
+    depth, height, width = inputs.shape[2:5]
+    inputs = inputs.permute(0, 2, 3, 4, 1, 5, 6, 7).reshape(
+        -1, module.in_channels * kernel[0] * kernel[1] * kernel[2]
+    )
+    outputs = functional.linear(
+        inputs,
+        module.weight.reshape(module.out_channels, -1),
+        module.bias,
+    )
+    return outputs.view(batch_size, depth, height, width, module.out_channels).permute(
+        0, 4, 1, 2, 3
+    )
+
+
+def patch_vlm_conv3d(model: Any) -> list[str]:
+    """Patch supported Conv3D instances without changing ``torch.nn.Conv3d`` globally."""
+    if not _requires_conv3d_patch() or not _is_supported_vlm(
+        getattr(model, "config", None)
+    ):
+        return []
+
+    import torch
+
+    modules = [
+        (name or "<root>", module)
+        for name, module in model.named_modules()
+        if isinstance(module, torch.nn.Conv3d)
+    ]
+    unsupported = []
+    for name, module in modules:
+        errors = _contract_errors(module)
+        if errors:
+            unsupported.append(f"{name}: {', '.join(errors)}")
+
+    if unsupported:
+        raise RuntimeError(
+            "the KT Conv3D fallback does not support: " + "; ".join(unsupported)
+        )
+
+    patched_names = []
+    for name, module in modules:
+        if not getattr(module, _COMPATIBLE_ATTR, False):
+            setattr(module, _ORIGINAL_FORWARD_ATTR, module.forward)
+            module.forward = MethodType(_kt_conv3d_forward, module)
+            setattr(module, _COMPATIBLE_ATTR, True)
+        patched_names.append(name)
+
+    setattr(model, "_kt_vlm_conv3d_compatible", bool(modules))
+    return patched_names
+
+
+def is_vlm_conv3d_compatible(model: Any) -> bool:
+    """Return whether every Conv3D instance on a torch 2.9 KT VLM was patched."""
+    if not _requires_conv3d_patch():
+        return True
+
+    import torch
+
+    modules = [
+        module for module in model.modules() if isinstance(module, torch.nn.Conv3d)
+    ]
+    return bool(modules) and all(
+        getattr(module, _COMPATIBLE_ATTR, False) for module in modules
+    )

kt-kernel/python/sft/weights.py

@@ -14,6 +14,7 @@ import torch
 import torch.nn as nn
 
 from .arch import MOEArchConfig
+from .conv3d_compat import _canonicalize_qwen3_vl_fused_expert_weights
 from .dist_utils import _maybe_zero3_gathered_parameters
 
 logger = logging.getLogger(__name__)
@@ -54,6 +55,9 @@ def extract_moe_weights(
     if detect_fused_experts(experts):
         gate_up = getattr(experts, "gate_up_proj").data
         down_fused = getattr(experts, "down_proj").data
+        vlm_weights = _canonicalize_qwen3_vl_fused_expert_weights(gate_up, down_fused, moe_config)
+        if vlm_weights is not None:
+            return vlm_weights
         # gate_up_proj is [E, 2*I, H], split into gate [E, I, H] and up [E, I, H]
         intermediate = gate_up.shape[1] // 2
         gate_proj = gate_up[:, :intermediate, :].contiguous()
@@ -393,10 +397,15 @@ def load_experts_from_checkpoint_files(
         if gate_up is None or down is None:
             raise FileNotFoundError(f"Missing fused expert weights for layer {layer_idx}")
         gate_up = gate_up.cpu().to(torch.bfloat16).contiguous()
-        I = gate_up.shape[1] // 2
-        gate_proj = gate_up[:, :I, :].contiguous()
-        up_proj = gate_up[:, I:, :].contiguous()
-        down_proj = down.cpu().to(torch.bfloat16).contiguous()
+        down = down.cpu().to(torch.bfloat16)
+        vlm_weights = _canonicalize_qwen3_vl_fused_expert_weights(gate_up, down, moe_config)
+        if vlm_weights is not None:
+            gate_proj, up_proj, down_proj = vlm_weights
+        else:
+            I = gate_up.shape[1] // 2
+            gate_proj = gate_up[:, :I, :].contiguous()
+            up_proj = gate_up[:, I:, :].contiguous()
+            down_proj = down.contiguous()
         del gate_up
         print(
             f"[kt_moe] Layer {layer_idx}: fused expert format — "

kt-kernel/python/sft/wrapper.py

@@ -25,6 +25,7 @@ from .layer import KTMoELayerWrapper
 from .lora import LoRAExperts
 from .base import _supports_authoritative_optimizer_grads
 from .checkpoint import load_full_weight_layer, resolve_full_weight_checkpoint
+from .conv3d_compat import patch_vlm_conv3d
 from .dist_utils import _distributed_rank_world_size
 from .weights import (
     _clear_original_expert_weights,
@@ -160,6 +161,9 @@ def wrap_moe_layers_with_kt_wrapper(model: nn.Module, kt_plugin: Any) -> list[KT
     is_rank_0 = distributed_rank == 0
 
     moe_config = get_moe_arch_config(model.config)
+    patched_conv3d = patch_vlm_conv3d(model)
+    if patched_conv3d:
+        logger.info(f"Patched KT VLM Conv3D modules: {patched_conv3d}")
     _text_cfg = getattr(model.config, "text_config", model.config)
     hidden_size = _text_cfg.hidden_size
 

kt-kernel/test/per_commit/test_sft_conv3d_compat.py

@@ -0,0 +1,68 @@
+# SPDX-License-Identifier: Apache-2.0
+
+import copy
+import importlib.util
+import sys
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+
+MODULE_PATH = (
+    Path(__file__).resolve().parents[2] / "python" / "sft" / "conv3d_compat.py"
+)
+SPEC = importlib.util.spec_from_file_location(
+    "kt_sft_conv3d_compat_under_test", MODULE_PATH
+)
+assert SPEC is not None and SPEC.loader is not None
+conv3d_compat = importlib.util.module_from_spec(SPEC)
+sys.modules[SPEC.name] = conv3d_compat
+SPEC.loader.exec_module(conv3d_compat)
+
+
+def _vlm(conv3d):
+    model = torch.nn.Sequential(conv3d)
+    model.config = SimpleNamespace(model_type="qwen3_vl_moe", vision_config=object())
+    return model
+
+
+def test_patch_vlm_conv3d_matches_native_forward_and_backward(monkeypatch):
+    monkeypatch.setattr(conv3d_compat, "_requires_conv3d_patch", lambda: True)
+    native_forward = torch.nn.Conv3d.forward
+    reference = _vlm(
+        torch.nn.Conv3d(
+            3, 4, kernel_size=(2, 4, 4), stride=(2, 4, 4), bias=True
+        ).double()
+    )
+    patched = copy.deepcopy(reference)
+    reference_input = torch.randn(
+        2, 3, 4, 8, 8, dtype=torch.float64, requires_grad=True
+    )
+    patched_input = reference_input.detach().clone().requires_grad_(True)
+
+    assert conv3d_compat.patch_vlm_conv3d(patched) == ["0"]
+    assert conv3d_compat.patch_vlm_conv3d(patched) == ["0"]
+    assert torch.nn.Conv3d.forward is native_forward
+    assert conv3d_compat.is_vlm_conv3d_compatible(patched)
+
+    expected = reference(reference_input)
+    actual = patched(patched_input)
+    expected.square().sum().backward()
+    actual.square().sum().backward()
+
+    torch.testing.assert_close(actual, expected)
+    torch.testing.assert_close(patched_input.grad, reference_input.grad)
+    torch.testing.assert_close(patched[0].weight.grad, reference[0].weight.grad)
+    torch.testing.assert_close(patched[0].bias.grad, reference[0].bias.grad)
+
+
+def test_patch_vlm_conv3d_rejects_unsupported_contract_atomically(monkeypatch):
+    monkeypatch.setattr(conv3d_compat, "_requires_conv3d_patch", lambda: True)
+    model = _vlm(torch.nn.Conv3d(3, 4, kernel_size=(2, 4, 4), stride=(1, 4, 4)))
+
+    with pytest.raises(RuntimeError, match="stride="):
+        conv3d_compat.patch_vlm_conv3d(model)
+
+    assert not hasattr(model[0], "_kt_conv3d_compatible")

kt-kernel/test/per_commit/test_sft_qwen3_vl_moe.py

@@ -0,0 +1,104 @@
+# SPDX-License-Identifier: Apache-2.0
+
+import importlib.util
+import sys
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+from kt_kernel.sft.layer import KTMoELayerWrapper
+from kt_kernel.sft.weights import extract_moe_weights
+
+
+MODULE_PATH = Path(__file__).resolve().parents[2] / "python" / "sft" / "arch.py"
+SPEC = importlib.util.spec_from_file_location("kt_sft_arch_under_test", MODULE_PATH)
+assert SPEC is not None and SPEC.loader is not None
+arch = importlib.util.module_from_spec(SPEC)
+sys.modules[SPEC.name] = arch
+SPEC.loader.exec_module(arch)
+
+
+def _qwen3_vl_moe_modules():
+    configuration = pytest.importorskip(
+        "transformers.models.qwen3_vl_moe.configuration_qwen3_vl_moe",
+        reason="Qwen3-VL-MoE support requires a recent transformers build",
+    )
+    modeling = pytest.importorskip(
+        "transformers.models.qwen3_vl_moe.modeling_qwen3_vl_moe",
+        reason="Qwen3-VL-MoE support requires a recent transformers build",
+    )
+    return configuration, modeling
+
+
+def _text_config():
+    configuration, _ = _qwen3_vl_moe_modules()
+    return configuration.Qwen3VLMoeTextConfig(
+        hidden_size=8,
+        intermediate_size=16,
+        moe_intermediate_size=4,
+        num_experts=2,
+        num_experts_per_tok=1,
+        num_hidden_layers=1,
+        num_attention_heads=1,
+        num_key_value_heads=1,
+        head_dim=8,
+    )
+
+
+def test_qwen3_vl_moe_architecture_and_checkpoint_prefix():
+    config = SimpleNamespace(
+        architectures=["Qwen3VLMoeForConditionalGeneration"],
+        text_config=_text_config(),
+    )
+
+    moe_config = arch.get_moe_arch_config(config)
+
+    assert moe_config.expert_num == 2
+    assert moe_config.intermediate_size == 4
+    assert moe_config.num_experts_per_tok == 1
+    assert moe_config.has_shared_experts is False
+    assert arch._get_layers_prefix(config) == "model.language_model.layers"
+
+
+def test_qwen3_vl_moe_fused_weights_and_router_match_transformers():
+    _, modeling = _qwen3_vl_moe_modules()
+    text_config = _text_config()
+    original_moe = modeling.Qwen3VLMoeTextSparseMoeBlock(text_config)
+    moe_config = arch.MOEArchConfig(
+        moe_layer_attr="mlp",
+        router_attr="gate",
+        experts_attr="experts",
+        weight_names=("gate_proj", "up_proj", "down_proj"),
+        expert_num=2,
+        intermediate_size=4,
+        num_experts_per_tok=1,
+    )
+
+    gate_proj, up_proj, down_proj = extract_moe_weights(original_moe, moe_config)
+    torch.testing.assert_close(gate_proj, original_moe.experts.gate_up_proj[:, :4])
+    torch.testing.assert_close(up_proj, original_moe.experts.gate_up_proj[:, 4:])
+    torch.testing.assert_close(down_proj, original_moe.experts.down_proj)
+
+    hidden_states = torch.randn(2, 3, text_config.hidden_size)
+    with torch.no_grad():
+        _, expected_weights, expected_ids = original_moe.gate(
+            hidden_states.reshape(-1, text_config.hidden_size)
+        )
+
+    wrapped_moe = KTMoELayerWrapper(
+        original_moe=original_moe,
+        wrapper=None,
+        lora_params=None,
+        moe_config=moe_config,
+        hidden_size=text_config.hidden_size,
+        layer_idx=0,
+        full_weight_grad=False,
+    )
+    actual_ids, actual_weights = wrapped_moe._compute_routing(hidden_states)
+
+    assert torch.equal(actual_ids, expected_ids)
+    torch.testing.assert_close(
+        actual_weights, expected_weights.to(torch.bfloat16), rtol=2e-2, atol=2e-2
+    )