https://arxiv.org/abs/2607.05316
The authors took great care to separate decodability from causal use. However, their central contrast of a plan-like representation rather than a downstream regularity of decoding may not be exclusive. If the length signal rides on the geometry of the model’s representational trajectory, then it is both a real structure and a byproduct of next-token training.
Recent work on transformer representation geometry makes this non-exclusive reading plausible. Hosseini and Fedorenko show that hidden-state trajectories straighten through the middle layers of trained models, with the effect growing based on model size and training corpus. Nothing explicitly optimizes for straightening; it is a natural byproduct of next-token prediction. Yom Din et al. add that successive layers are related by fitted linear maps. Along a partially straightened trajectory, position becomes linearly available from h_t, and remaining length follows arithmetically from a prompt-conditional marginal.
This ontology is both representational and downstream, which invalidates the rather than.
The paper’s results corroborate. The prompt-end probe shows that the prompt constrains the trajectory before generation starts, which the geometric account agrees with. The cross-dataset transfer is more interesting. The natural language corpora explore more representation space than the synthetic set, thus the result is trivial. Additionally, the classification probes of A.8 establish that the computational progress is decodable, which means that remaining length can be a readout of this progress.
This is an interesting place to explore. Give the probe [h_t, t] and see if the residual stream still adds anything when position is handed directly. Then compare the trained probe directions to one another and to a probe fit on the A.8 progress target. If they’re collinear, then one global direction explains the transfer results the paper attributes to length tracking.
References
Hosseini, Eghbal A., and Evelina Fedorenko. “Large Language Models Implicitly Learn to Straighten Neural Sentence Trajectories to Construct a Predictive Representation of Natural Language.” Advances in Neural Information Processing Systems 36 (NeurIPS 2023), https://doi.org/10.48550/arXiv.2311.04930
Yom Din, Alexander, Taelin Karidi, Leshem Choshen, and Mor Geva. “Jump to Conclusions: Short-Cutting Transformers with Linear Transformations.” Proceedings of LREC-COLING 2024, 2024, pp. 9615–9625, https://aclanthology.org/2024.lrec-main.840/
