Take a sentence from a TED talk and ask four people what kind of evaluation it carries. Is the speaker expressing a feeling, passing moral judgment on someone, or appraising the worth of a thing? The distinctions sound crisp until you try to apply them to a few hundred real sentences of spoken science talk. Then the boundaries blur, and trained readers start disagreeing with each other.
That disagreement is the subject of a new preprint by Mirela Imamovic, Aenne Cecilia Kristine Knierim, Khushi Pitroda, and Ekaterina Lapshinova-Koltunski, posted to arXiv on 30 July 2026. The team set out to compare three kinds of annotator on the same difficult job: linguists in training, one trained linguist, and large language models. The question was not simply who scores highest. It was whether humans and machines stumble in the same places.
Their test material comes from a corpus of English TED talk transcripts, which is popular science discourse in its spoken form: a speaker on a stage, working to make an audience care about something technical. Talks like that are dense with evaluation. The framework the researchers used to describe it is Appraisal theory, a well established approach in linguistics for mapping how speakers signal stance. Within Appraisal sits a subsystem called Attitude, which splits into three classes. Affect covers emotion, the speaker's own feelings. Judgement covers assessments of people and their behaviour. Appreciation covers evaluations of things, ideas, and phenomena.
The authors are explicit that this is a highly subjective annotation task, and they chose it for exactly that reason. A task where everyone agrees teaches you nothing about the limits of annotation. A task where careful people diverge is where the interesting problems live.
What they did
The work came in two halves. First the team assessed human annotations at the sentence level, within specific scientific domains, to see how consistently people applied the three Attitude classes. Then they turned to the models. They wrote three different prompts, which are the instructions given to a language model describing what to label and how, and compared how well each one drove the classification. Having identified the best-performing prompt, they used it to test three different large language models. Finally they fine-tuned a model, meaning they trained it further on labelled examples rather than relying on instructions alone.
That fine-tuned model reached an F1 score of 0.77. F1 is a single number combining two things a classifier can get wrong: labelling sentences it should have left alone, and missing ones it should have caught. It runs from 0 to 1. For a task this contested, 0.77 is a respectable figure, though the paper does not claim it as a solved problem.
The comparison is where the result gets interesting. The models performed best when measured against the annotations produced by the trained linguist, the most experienced human in the study. The linguists in training, meanwhile, did not reach high agreement scores among themselves. In other words, the machine output aligned more closely with expert human labelling than novice humans aligned with each other.
It is worth being careful about what that does and does not mean. It does not mean the models understand Appraisal theory. It means that on this corpus, with this three-way classification, model output tracked one expert's decisions more reliably than a group of students tracked their own collective sense of the categories. The authors' framing is measured: they conclude that LLMs can aid in complex annotation task resolution, not that they replace the annotator.
Why it matters
Annotation is the unglamorous infrastructure of a great deal of research in linguistics and digital humanities. Before anyone can analyse how evaluative language works across thousands of talks, someone has to label the sentences. That labour is slow, expensive, and, on subjective tasks, unreliable in ways that are hard to see from the outside. Low agreement between annotators does not just add noise; it raises a genuine question about whether the categories are stable enough to build findings on.
So a tool that produces expert-comparable labels at scale changes the arithmetic of what is worth attempting. The authors suggest this opens new pathways for the complex theories annotated and analysed in digital humanities studies, which is to say that frameworks previously too laborious to apply across a large corpus might now be testable.
There is a quieter finding underneath, too. If novice annotators disagree substantially on Appraisal categories, that tells us something about the theory itself, or at least about how it is taught and applied. The difficulty is not merely a training gap to be closed. Some of it appears to be built into the distinctions.
This is a preprint, not yet peer reviewed, and it rests on a single corpus in a single language. The findings describe English TED talks, not evaluative language in general. Still, the design is unusually honest about where the hard part sits: not in the machine's performance, but in the shakiness of the human ground truth it is being measured against.