Language models map some word relationships more cleanly than others, and asymmetry helps

Six relations. Three kinds of language model. One question that sounds simple until you try to answer it: when a model turns words into numbers, does the arrangement of those numbers actually know that a wheel is part of a car?

That is the question Zhihan Cao, Hiroaki Yamada, Simone Teufel and four colleagues set out to test in a preprint posted to arXiv on 29 July 2026. Their subject is what the field calls semantic space, the high-dimensional coordinate system a language model builds so that every word it knows sits somewhere in it. Words with similar meanings tend to land near each other. That much has been established for years. What the authors wanted to know is subtler: whether the relations between words leave their own geometric footprint in that space, and whether some relations leave clearer footprints than others.

The team approached it from three angles. First they asked whether relata, their term for the set of words that stand in a given relation to a target word, cluster together in one region of the space. If you take "car" as the target and collect all the things that are parts of a car, do those words occupy a coherent patch? And is that patch distinct from the patch occupied by, say, the car's category memberships? Second, they checked whether the geometry respects properties that logicians take for granted about relations. Symmetry means the relation runs both ways: if A is a synonym of B, B is a synonym of A. Asymmetry means it runs one way only: a finger is part of a hand, but a hand is not part of a finger. Transitivity means it chains: if A is a kind of B and B is a kind of C, then A is a kind of C. Third, they asked where the geometry gets its information from, the bare surface forms of the words themselves or the contexts those words appear in.

They ran all of this across six semantic relations and three model families that generate text in genuinely different ways: causal models, which predict each word from the words before it; masked models, which fill in blanks with the whole sentence available on both sides; and diffusion models, which refine text from noise.

What the geometry showed

The clearest pattern involved asymmetry. Relata in asymmetric relations, the lopsided ones, carve out reasonably distinct regions of semantic space. The authors describe this as "relatively clearly" the case, and that hedge is theirs, not a softening added here. Asymmetric relations also fared better on the logical-properties test: their symmetry, asymmetry and transitivity behaviour was encoded only moderately well, but still better than the equivalent behaviour for symmetric relations. The mutual relations, in other words, were the harder case. A model that has learned language purely by watching which words co-occur may struggle to distinguish "A relates to B" from "B relates to A" precisely when the honest answer is that both hold.

The information-source question split the models apart. For the causal model, lexical information, meaning the word forms themselves, mattered more. For the masked and diffusion models, context mattered more. The authors are careful to scope this to the specific models they evaluated rather than to causal or masked architectures in general, and it is worth holding that caution. Still, the direction makes a certain intuitive sense: a model trained to look only leftward has less context to lean on than one that reads a sentence from both ends at once.