问题描述
我想在我的 Pandas 数据帧上使用 Spacy 的依赖解析器为基于方面的情感分析提取名词-形容词对。我在来自 kaggle 的亚马逊美食评论数据集上尝试使用此代码:Named Entity Recognition in aspect-opinion extraction using dependency rule matching
但是,我将熊猫数据框提供给 spacy 的方式似乎有问题。我的结果不是我期望的那样。有人可以帮我调试这个吗?非常感谢。
!python -m spacy download en_core_web_lg
import nltk
nltk.download('vader_lexicon')
import spacy
nlp = spacy.load("en_core_web_lg")
from nltk.sentiment.vader import SentimentIntensityAnalyzer
sid = SentimentIntensityAnalyzer()
def find_sentiment(doc):
# find roots of all entities in the text
for i in df['Text'].tolist():
doc = nlp(i)
ner_heads = {ent.root.idx: ent for ent in doc.ents}
rule3_pairs = []
for token in doc:
children = token.children
A = "999999"
M = "999999"
add_neg_pfx = False
for child in children:
if(child.dep_ == "nsubj" and not child.is_stop): # nsubj is nominal subject
if child.idx in ner_heads:
A = ner_heads[child.idx].text
else:
A = child.text
if(child.dep_ == "acomp" and not child.is_stop): # acomp is adjectival complement
M = child.text
# example - 'this Could have been better' -> (this,not better)
if(child.dep_ == "aux" and child.tag_ == "MD"): # MD is modal auxiliary
neg_prefix = "not"
add_neg_pfx = True
if(child.dep_ == "neg"): # neg is negation
neg_prefix = child.text
add_neg_pfx = True
if (add_neg_pfx and M != "999999"):
M = neg_prefix + " " + M
if(A != "999999" and M != "999999"):
rule3_pairs.append((A,M,sid.polarity_scores(M)['compound']))
return rule3_pairs
df['three_tuples'] = df['Text'].apply(find_sentiment)
df.head()
解决方法
如果您在 apply
上调用 df['Text']
,那么您实际上是在遍历该列中的每个值并将该值传递给函数。
然而,在这里,您的函数本身会迭代您正在应用该函数的同一数据帧列,同时还会覆盖在函数早期传递给它的值。
所以我会从如下重写函数开始,看看它是否产生了预期的结果。我不能肯定,因为您没有发布任何示例数据,但这至少应该使球向前移动:
def find_sentiment(text):
doc = nlp(text)
ner_heads = {ent.root.idx: ent for ent in doc.ents}
rule3_pairs = []
for token in doc:
children = token.children
A = "999999"
M = "999999"
add_neg_pfx = False
for child in children:
if(child.dep_ == "nsubj" and not child.is_stop): # nsubj is nominal subject
if child.idx in ner_heads:
A = ner_heads[child.idx].text
else:
A = child.text
if(child.dep_ == "acomp" and not child.is_stop): # acomp is adjectival complement
M = child.text
# example - 'this could have been better' -> (this,not better)
if(child.dep_ == "aux" and child.tag_ == "MD"): # MD is modal auxiliary
neg_prefix = "not"
add_neg_pfx = True
if(child.dep_ == "neg"): # neg is negation
neg_prefix = child.text
add_neg_pfx = True
if (add_neg_pfx and M != "999999"):
M = neg_prefix + " " + M
if(A != "999999" and M != "999999"):
rule3_pairs.append((A,M,sid.polarity_scores(M)['compound']))
return rule3_pairs