选择性重要性采样：让离策略 token 变在策略 Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment

精读摘要

LLM 强化学习后训练采用「先采样后更新」的高效范式，必然产生离策略训练数据。经典的重要性采样（IS）修正方案在长序列上 token 级比率连乘，方差爆炸严重。这项研究提出选择性重要性采样（SIS），思路是把离策略 token「转移」为在策略 token，从而无需修正分数。它作为一个即插即用的方案，旨在提升对齐训练的稳定性。 RL post-training for LLMs follows a rollout-then-update paradigm, inevitably producing off-policy data. Classic importance sampling corrects this, but token-level ratios compound over long sequences, causing severe variance explosion. The proposed Selective Importance Sampling transfers off-policy tokens on-policy so correction scores become unnecessary, offered as a plug-in for improving alignment.