mirror of
https://github.com/unslothai/unsloth.git
synced 2026-10-21 14:00:32 +00:00
7f4736d383
With TRL 0.28.0 and `padding_free=True`, DPOTrainer flattens chosen/rejected inputs and passes reset `position_ids` without `packed_seq_lengths`. Fast Llama/Qwen2 attention currently sees no packed metadata, so rejected tokens can attend to earlier chosen tokens. Selecting FlashAttention 2 does not prevent this: the missing metadata selects the dense rather than varlen backend. Recover lengths from reset positions at the shared LlamaModel entry, before overlength truncation and decoder layers. Reuse the existing position-boundary helper and pass the inferred `packed_seq_lengths` through the existing packed attention path. Inference is opt-in for a single row without a padding mask or incoming KV cache; explicit lengths take precedence. Doing this at model entry also preserves the original padding-mask distinction before the training path may discard that mask and avoids repeating boundary inference in each layer. Scope: the confirmed issue is the older TRL 0.28.0 padding-free DPO path. [Current TRL documents that DPO padding-free is disabled and falls back to padding](https://github.com/huggingface/trl/blob/main/docs/source/reducing_memory_usage.md); this is not a claim about its default behavior. Validation: 14 local CPU regression cases passed using real TRL 0.28.0 `DPOTrainer.concatenated_forward`, source-extracted production model-entry/fast-attention functions, and production attention dispatch with PyTorch SDPA. Identity RoPE isolates sample-boundary behavior. Checks include log-prob and gradient agreement with independent/padded inputs, zero rejected-to-chosen gradients, a control reproducing leakage without the fix, MHA/GQA, both packed SDPA routes, explicit metadata precedence, ordinary positions, padding masks, KV cache, and preserving a flattened batch past the length check. FA2 dispatch is checked to select `FLASH_VARLEN`; GPU FA2 kernel execution was not tested. Repository-pinned formatting, Ruff lint, and `git diff --check` passed. This PR contains only the two production files; regression scripts remain local. --------- Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| aime_eval.md | ||
| aime_eval.py | ||
| cleanup_utils.py | ||
| data_utils.py | ||
| generate_dataset_with_none.py | ||
| hf_utils.py | ||
| ocr_eval.md | ||
| ocr_eval.py | ||
| os_utils.py | ||
| perplexity_eval.md | ||
| perplexity_eval.py | ||
| run_none_detect_tests.py | ||
| test_attention_dispatch_bidirectional.py | ||
| test_attention_dispatch_dora_dtype.py | ||
| test_attention_dispatch_eval_backward.py | ||
| test_attention_dispatch_mixed_devices.py | ||
| test_attention_masks.py | ||
| test_attention_native_routing.py | ||
| test_attn_mask_compat.py | ||
| test_attn_mask_upstream_drift.py | ||
| test_batched_leftpad_generation_gpu.py | ||
| test_create_optimizer_model_arg.py | ||
| test_dataset_num_proc.py | ||
| test_flash_adamw_optim.py | ||
| test_flash_impl.py | ||
| test_hybrid_packing_default.py | ||
| test_init_lora_validation.py | ||
| test_lora_ga_save_conversion.py | ||
| test_packed_flash_documents.py | ||
| test_packing.py | ||
| test_packing_overflow_guard.py | ||
| test_prepare_inputs_leftpad.py | ||
| test_q_galore.py | ||
| test_qat.py | ||
| test_qwen4_exp_ple_packing.py | ||
| test_rope_scaling_drift.py | ||
| test_rope_scaling_user_arg.py | ||
| test_sdpa_packed_segments.py | ||
| test_single_row_cached_attention.py | ||
| test_torch_varlen_packed.py | ||
| test_train_on_responses_only_num_proc.py | ||
| test_trunc_normal_patch.py | ||
| test_truncation_attestation.py | ||
| test_unsloth_optimizer_embedding_optim_bits.py | ||
| test_unsloth_optimizer_full_finetune_embedding_lr.py | ||
| test_unsloth_optimizer_loraplus.py | ||
| test_unsloth_optimizer_weight_decay.py | ||
| test_varlen_int32_overflow_guard.py | ||
| test_xformers_broken_message.py | ||
| test_xformers_capability_gate.py | ||