自我蒸馏：大模型性能恢复的新机制 Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting

精读摘要

大语言模型在监督微调、量化与剪枝过程中常因灾难性遗忘等原因性能下降。这项研究提出基于自蒸馏微调（SDFT）的性能恢复框架，让模型以自身为教师找回丢失的能力，并为恢复机制给出了严格的理论解释。这对模型压缩与微调后「掉点」这一普遍痛点，提供了一个可操作的补救方案。 LLMs often suffer performance degradation from catastrophic forgetting during supervised fine-tuning, quantization, and pruning. This work introduces a performance recovery framework based on Self-Distillation Fine-Tuning (SDFT) that restores model capabilities, complemented by a rigorous theoretical explanation of the recovery mechanism. It offers a practical remedy for post-compression performance drops.