问题描述
如何计算多标签注释的 Krippendorff Alpha? 在多类注释的情况下(假设 3 个编码员用 3 个标签注释了 4 个文本:a、b、c),我首先构建可靠性数据矩阵,然后是巧合,然后根据巧合我可以计算 Alpha:
问题是如何在多标签分类问题(如下例)的情况下准备巧合并计算 alpha?
Python 实现甚至 excel 将不胜感激。
解决方法
在寻找类似信息时遇到了您的问题。我们使用了以下代码,其中 nltk.agreement
表示指标,pandas_ods_reader
从 LibreOffice 电子表格中读取数据。我们的数据有两个注释者,对于某些项目可以有两个标签(例如,一个编码员只注释了一个标签,另一个编码员注释了两个标签)。
下面的电子表格屏幕截图显示了输入数据的结构。注释项的列称为annotItems
,注释列称为coder1
和coder2
。有多个标签时的分隔符是管道,与示例中的逗号不同。
代码的灵感来自这篇 SO 帖子:Low alpha for NLTK agreement using MASI distance
from nltk import agreement
from nltk.metrics.distance import masi_distance
from nltk.metrics.distance import jaccard_distance
import pandas_ods_reader as pdreader
annotfile = "test-iaa-so.ods"
df = pdreader.read_ods(annotfile,"Sheet1")
annots = []
def create_annot(an):
"""
Create frozensets with the unique label
or with both labels splitting on pipe.
Unique label has to go in a list so that
frozenset does not split it into characters.
"""
if "|" in str(an):
an = frozenset(an.split("|"))
else:
# single label has to go in a list
# need to cast or not depends on your data
an = frozenset([str(int(an))])
return an
for idx,row in df.iterrows():
annot_id = row.annotItem + str.zfill(str(idx),3)
annot_coder1 = ['coder1',annot_id,create_annot(row.coder1)]
annot_coder2 = ['coder2',create_annot(row.coder2)]
annots.append(annot_coder1)
annots.append(annot_coder2)
# based on https://stackoverflow.com/questions/45741934/
jaccard_task = agreement.AnnotationTask(distance=jaccard_distance)
masi_task = agreement.AnnotationTask(distance=masi_distance)
tasks = [jaccard_task,masi_task]
for task in tasks:
task.load_array(annots)
print("Statistics for dataset using {}".format(task.distance))
print("C: {}\nI: {}\nK: {}".format(task.C,task.I,task.K))
print("Pi: {}".format(task.pi()))
print("Kappa: {}".format(task.kappa()))
print("Multi-Kappa: {}".format(task.multi_kappa()))
print("Alpha: {}".format(task.alpha()))
对于从此答案链接的屏幕截图中的数据,将打印:
Statistics for dataset using <function jaccard_distance at 0x7fa1464b6050>
C: {'coder1','coder2'}
I: {'item3002','item1000','item6005','item5004','item2001','item4003'}
K: {frozenset({'1'}),frozenset({'0'}),frozenset({'0','1'})}
Pi: 0.1818181818181818
Kappa: 0.35714285714285715
Multi-Kappa: 0.35714285714285715
Alpha: 0.02941176470588236
Statistics for dataset using <function masi_distance at 0x7fa1464b60e0>
C: {'coder1','1'})}
Pi: 0.09181818181818181
Kappa: 0.2864285714285714
Multi-Kappa: 0.2864285714285714
Alpha: 0.017962466487935425