Difan Jiao

Generative models can adapt their outputs to different skill levels, yet the mechanism underlying this adaptation remains unexplored. We address this gap using chess as a model system, leveraging its well-defined decision space, precise skill metrics, and formally measurable strategic concepts. Taking Maia-2, a unified chess transformer that generates human-like moves calibrated to a wide skill spectrum, we formalize two competing hypotheses: the model dynamically adjusts its internal knowledge acquisition of chess concepts across skill levels, or it maintains consistent internal knowledge acquisition while varying its knowledge externalization for move predictions. Linear probes on 173 chess concepts reveal a nearly constant level of internal knowledge acquisition across all conditioning skill levels, in sharp contrast to the large gap in move prediction quality, directly refuting the first hypothesis. We then validate the second by modulating knowledge externalization on our curated Blundered Transitional Dataset: fine-tuning only the policy head, just 8.2% of model parameters, with the entire backbone frozen, almost perfectly bridges the skill gap from beginner to advanced play. Sparse Autoencoder feature steering provides complementary evidence, independently improving lower-skill move predictions while leaving knowledge acquisition level unchanged as verified by probes. Post-hoc transfer analyses further confirm that the externalization modulation is concept-specific rather than generic. Our findings indicate that skill-adaptive transformers maintain consistent knowledge acquisition across all skill levels and adapt by selectively gating what they externalize, carrying implications for both pedagogical applications and AI oversight. Read less