What determines inter-coder agreement in manual annotations? A meta-analytic investigation

Saskia Bayerl, KI Paul

Research output: Contribution to journalArticleAcademicpeer-review

14 Citations (Scopus)


This article presents our work on constructing a corpus of news articles in which events are annotated for estimated bounds on their duration, and automatically learning from this corpus. We describe the annotation guidelines, the event classes we categorized to reduce gross discrepancies in inter-annotator judgments, and our use of normal distributions to model vague and implicit temporal information and to measure inter-annotator agreement for these event duration distributions. We then show that machine learning techniques applied to this data can produce coarse-grained event duration information automatically, considerably outperforming a baseline and approaching human performance. The methods described here should be applicable to other kinds of vague but substantive information in texts.
Original languageEnglish
Pages (from-to)727-752
Number of pages26
JournalComputational Linguistics
Issue number4
Publication statusPublished - 2011


Dive into the research topics of 'What determines inter-coder agreement in manual annotations? A meta-analytic investigation'. Together they form a unique fingerprint.

Cite this