Open up a trained large language model and you will not find a drawer labeled "dog" or a tidy compartment for "negation." What you find instead are distributed statistical associations: patterns spread across many numbers, none of which is the concept by itself. Researchers have spent years developing techniques to pull concept-like structure back out of these systems. Probing trains a small classifier on a model's internal activations to test whether some notion is encoded there. Dictionary learning tries to decompose those tangled activations into a vocabulary of interpretable pieces.
Both work after the fact. That is the point Chen Shani presses in a position paper posted to arXiv on 29 July 2026. Concept-level structure in today's models, Shani writes, is found rather than designed. It is recovered from a system that was never built to have it.
The difference matters more than it might sound. If you discover a concept by probing a finished model, you have no guarantee that the concept is stable, meaning it stays put rather than shifting between training runs or layers. You have no guarantee it is compositional, meaning smaller concepts combine into larger ones in predictable ways. You have no guarantee it is controllable, meaning you can reach in and adjust it. And you have no guarantee it lines up with how people organize ideas. Nothing in the architecture promises any of that. The structure is a discovery, not a commitment.
Two questions that organize a scattered field
Rather than propose a new model, Shani does something more modest and arguably more useful: she lays out a map. The paper organizes existing work along two dimensions.
The first is when. At what stage of the pipeline does concept structure enter? It might be built into the training objective, so the model is rewarded for learning concepts. It might live in the core architecture itself. It might be introduced at inference time, when the model is actually running and producing text. Or it might arrive only at the post-hoc interpretation stage, which is where probing and dictionary learning sit.
The second dimension is where the structure comes from. Is it internally derived, meaning it emerges from the model's own representations? Or is it externally grounded, meaning it is anchored in outside resources such as human-built knowledge bases and ontologies?
Cross those two axes and you get a grid. Shani reports three patterns that fall out of it.
Inference-time approaches, first, are comparatively underexplored. Compared with work on training objectives and post-hoc analysis, relatively little effort has gone into shaping concept structure at the moment a model runs.
Second, closely related ideas have grown up in isolation from one another. Researchers working on training objectives and researchers working on interpretability have been circling similar questions from different corners of the pipeline, largely without talking.
Third, externally grounded methods turn up at every stage of the pipeline, but they are frequently described under different names. The same underlying move, anchoring a model's concepts in an external human-curated resource, appears repeatedly in the literature wearing different vocabulary. Shani's taxonomy makes those family resemblances visible.
Why it matters
Much of the current push for AI transparency runs through interpretability: peer inside a finished model and work out what it is doing. That effort is real and valuable, and Shani is not dismissing it. But her framing points at a structural limitation. Reading concepts out of a system that was not built to hold them is inference. It can be wrong, and it offers no guarantee that what you found today will still be there tomorrow.
Designing concepts in from the start would change what you can promise. If a model's conceptual representations were explicit rather than implicit, you could in principle check whether they are stable, verify that they compose, edit them directly, and hold them against how humans actually carve up the world. Those four properties, which Shani names as the ones current approaches cannot guarantee, are close to what anyone auditing or steering a model would want.
A caveat worth keeping in view: this is a position paper, not an experiment. Shani is making an argument and organizing a literature, not reporting that a concept-designed model outperforms anything. The paper does not claim a working system. It claims a research direction, and offers a map of who is already, perhaps without realizing it, heading in that direction.
The piece has not yet been through peer review. Its value is in the framing: a compact way to see a scattered body of work as parts of one question, and a clear statement of what the field gives up by treating concepts as something to be excavated rather than built.