LoRA

Why Transformers Stop Following References Early, and How a Rank-8 LoRA at One Layer Pushes Them Further

A September 29 arXiv paper reports that pretrained transformers follow only a few in-context references, and that a tiny LoRA at one…