殊途同归：不同模型学到了相似的数字表征 Convergent Evolution: How Different Language Models Learn Similar Number Representations

精读摘要

在自然文本上训练的语言模型会用周期为 2、5、10 的周期特征表示数字。这项研究进一步发现了一个两级层次结构：Transformer、线性 RNN、LSTM 与经典词嵌入虽然训练方式各异，都学到了傅里叶域中周期 T 尖峰的特征，但只有部分模型学到几何可分的特征——可用于线性分类一个数 mod T。作者证明傅里叶域稀疏性是必要条件，为「数字表征如何涌现」提供了理论解释。 Language models trained on natural text represent numbers with periodic features at periods T of 2, 5, and 10. This work identifies a two-tiered hierarchy: Transformers, linear RNNs, LSTMs, and classical word embeddings all learn features with period-T spikes in the Fourier domain, but only some learn geometrically separable features usable for linear classification mod-T. The authors prove Fourier domain sparsity is necessary, offering a theoretical account of numeric representation.