在R中同时输出有单字和双字的文本

问题描述

我正在尝试弄清楚如何在R中的文本中识别单字组和双字组，然后根据阈值将两者都保留在最终输出中。我已经使用gensim的Phraser模型在Python中完成了此操作，但是还没有弄清楚如何在R中执行此操作。

例如：

strings <- data.frame(text = 'This is a great movie from yesterday','I went to the movies','Great movie time at the theater','I went to the theater yesterday')
#Pseudocode below
bigs <- tokenize_uni_bi(strings,n = 1:2,threshold = 2)
print(bigs)
[['this','great_movie','yesterday'],['went','movies'],['great_movie','theater'],'theater','yesterday']]

谢谢！

解决方法

您可以为此使用Quanteda框架：

library(quanteda)
# tokenize,tolower,remove stopwords and create ngrams
my_toks <- tokens(strings$text) 
my_toks <- tokens_tolower(my_toks)
my_toks <- tokens_remove(my_toks,stopwords("english"))
bigs <- tokens_ngrams(my_toks,n = 1:2)

# turn into document feature matrix and filter on minimum frequency of 2 and more
my_dfm <- dfm(bigs)
dfm_trim(my_dfm,min_termfreq = 2)

Document-feature matrix of: 4 documents,6 features (50.0% sparse).
       features
docs    great movie yesterday great_movie went theater
  text1     1     1         1           1    0       0
  text2     0     0         0           0    1       0
  text3     1     1         0           1    0       1
  text4     0     0         1           0    1       1

# use convert function to turn this into a data.frame

或者，您可以使用tidytext包，tm，tokenizers等。这全都取决于您期望的输出。

使用tidytext / dplyr的示例如下：

library(tidytext)
library(dplyr)
strings %>% 
  unnest_ngrams(bigs,text,n = 2,n_min = 1,ngram_delim = "_",stopwords = stopwords::stopwords()) %>% 
  count(bigs) %>% 
  filter(n >= 2)

         bigs n
1       great 2
2 great_movie 2
3       movie 2
4     theater 2
5        went 2
6   yesterday 2

quanteda和tidytext都有大量在线帮助。参见使用cran上的两个软件包的小插图。