Knowing the Questions before the Exam: Enabling Selective Memory in Linear Attention Models

Abstract

Linear-attention language models compress their input into a fixed-size state and while processing tokens must decide what information to retain. Such compressed attention models are particularly attractive for long form generation as in agentic workloads. An advance hint or question, as is available in many practical agentic tasks, should facilitate this selection. We test this hypothesis on multi-document question answering by placing the question before or after contexts containing relevant and irrelevant information. Contrary to expectation, both a recurrent model (RWKV-7 7.2B) and a hybrid model (Qwen3.5 0.8B) recall less when the question appears first. Low-rank adaptation on question-first data reverses this result, enabling selective retention at lengths far beyond fine-tuning and even beyond the pretrained context window. Fine-tuning makes state writing, erasing, and decay depend on question relevance, a behavior we call selective memory. Varying question reliability during training controls the trade-off between selectivity and breadth. Thus, linear-attention architectures support selective memory, but targeted adaptation is required to enable it.