使用Quanteda计算术语特定项和逆项frq

问题描述

示例测试数据集：

library(quanteda)

dataset1 <- data.frame( anumber = c(1,2,3),text = c("Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s,when an unkNown printer took a galley of type and scrambled it to make a type specimen book.","It has survived not only five centuries,but also the leap into electronic typesetting,remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages,and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum","Contrary to popular belief,Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC,making it over 2000 years old. Richard Mcclintock,a Latin professor at Hampden-Sydney College in Virginia,looked up one of the more obscure Latin words,consectetur,from a Lorem Ipsum passage,and going through the cites of the word in classical literature,discovered the undoubtable source."))

myDfm <- dataset1 %>%
corpus() %>%
tokens(remove_punct = TRUE,remove_numbers = TRUE,remove_symbols = TRUE) %>%
tokens_ngrams(n = 1:3) %>%
dfm()

如何计算所有文档中的tf，并将其乘以特定于术语的idf并将结果再次作为dfm？

解决方法

这里的诀窍是计算所有文档中的特征（项）频率组合（通常在文档中计算），然后乘以文档频率（所有特征中的特征频率始终由特征计算）。

然后，您可以为整个集合的每个功能计算一个“ tf-idf”得分。您可以使用对dfm对象起作用的两个功能来完成此操作：featfreq()表示术语/功能频率，而docfreq()表示文档频率。

library(quanteda)
## Package version: 2.1.1

# your tokenization and dfm code
myDfm <- dataset1 %>%
  corpus() %>%
  tokens(remove_punct = TRUE,remove_numbers = TRUE,remove_symbols = TRUE) %>%
  tokens_ngrams(n = 1:3) %>%
  dfm()

这将计算整个语料库中每个功能的tf-idf，就像dfm_tfidf()在文档中每个功能的计算一样。还将它们按降序排序。

result <- featfreq(myDfm) * log(ndoc(myDfm) / docfreq(myDfm),base = 10) %>%
  sort(decreasing = TRUE)

这些是您的最佳名词：

head(result,10)
##     lorem     ipsum        is    simply     dummy      text        of       the 
## 2.8627275 2.8627275 0.9542425 0.9542425 0.9542425 1.4313638 3.3398488 4.7712125 
##  printing       and 
## 0.4771213 1.9084850

以及您的最低要求：

tail(result,10)
##                    the_cites_of                    cites_of_the 
##                       0.1760913                       0.1760913 
##                     of_the_word                     the_word_in 
##                       0.0000000                       0.0000000 
##               word_in_classical         in_classical_literature 
##                       0.0000000                       0.0000000 
## classical_literature_discovered       literature_discovered_the 
##                       0.0000000                       0.0000000 
##      discovered_the_undoubtable          the_undoubtable_source 
##                       0.0000000                       0.0000000