在.txt文件中查找热门单词计数-while循环进行得非常慢,无法正常工作

问题描述

我试图从本质上遍历.txt文件中的每个单词,当我发现一个单词(从我的单词映射中)的单词超过maxwordcount变量时,我将其添加到topwords向量的前面

int main(int argc,char** argv) {
    fstream txtfile;
    string filename = argv[1];
    string word,tempword;
    int maxwordcount = 0;
    int wordcount = 0;
    int uniquewordcount = 0;
    vector<pair <string,int> > topwords;
    map<string,int> words;

if (argc != 2) {
    cout << "Incorrect number of arguments on the command line bud" << endl;
}else{
    txtfile.open(filename.c_str());
if (txtfile.is_open()) {
        while (txtfile >> word){
            //removePunctuation(word);
            //transform(word.begin(),word.end(),word.begin(),[](unsigned char c){ return::tolower(c); });     //makes string lowercase using iterator
            if (words.find(word) == words.end()) {   
                words[word] = 1;                                //adds word into the map as a pair starting with a word count of 1
                if (words[word] > maxwordcount) {            //For case if word is the first word added to the map
                    maxwordcount = words[word];              //change maxwordcount
                    topwords.insert( topwords.begin(),make_pair(word,words[word]) );    //insert word into the front of the top words vector
                    cout << "word: '" << word << "'  word-count: " << words[word] << endl;
                }
                uniquewordcount++;              
            }else{                                          //the word is found
                words[word]++;                              //increment count for word by 1
                if (words[word] > maxwordcount) {           //check if wordcount > maxwordcount
                    topwords.insert( topwords.begin(),words[word]) );      //insert word into the front of the top words vector       
                }                                           
            }
            wordcount++;
        }

在程序结束时,我想显示txt文件中的前10个字。我通过显示实时单词计数(cout)测试了while循环是否正在运行。这个数字上升了,但是上升得非常慢。另外,我正在为txt文件使用大量书籍。

Image of results when running

我也不完全了解在地图和向量中插入变量,因此那里可能出了问题。

我已经走到了尽头,所以这时任何事情都会有所帮助。

我也使用了一个较小的文本文件进行测试:

This is a small sentence to test test test
hey hey

结果:

word: 'This'  word-count: 1
1
2
3
4
5
6
7
7
7
8
8
There were 11 words in the file.
There were 8 unique words in the file.
Top 20 words in little.txt:
   hey 2
   test 3
   test 2
   This 1
Segmentation fault

我知道我做错了什么,但是我不知道下一步该做什么或要测试什么。还是C ++和C的业余爱好者。

解决方法

您应该逐行读取文件,逐行处理

按行读取文件文件:https://www.systutorials.com/how-to-process-a-file-line-by-line-in-c/

https://www.geeksforgeeks.org/split-a-sentence-into-words-in-cpp/

https://www.w3schools.com/cpp/cpp_functions.asp

https://www.w3schools.com/cpp/cpp_function_param.asp

https://www.w3schools.com/cpp/cpp_function_return.asp

https://www.w3schools.com/cpp/cpp_pointers.asp

https://www.w3schools.com/cpp/cpp_references.asp

#include <bits/stdc++.h>
#include <iostream>
#include <fstream>
#include <string>
#include <vector>
#include <string>
#include <algorithm>
#include <sstream>
#include <regex>
#include <iterator>

using namespace std;


vector<std::pair <string,int> > topwords;

void store(vector<pair <string,int> > &topwords,string str){
    auto pos = std::find_if(topwords.begin(),topwords.end(),[str](std::pair<string,int> const &b) {
            return b.first == str;
});
    
    //std::cout<< pos->first << endl;
    
    if(pos != topwords.end()){
        std::cout << "word: " << pos->first << " " << pos->second << " found" << endl;
        pos->second++;
    }
    else{
        std::cout << "not found" << endl;
        topwords.push_back( make_pair(str,1) );
    }
}

void removeDupWord(string str)
{
    // Used to split string around spaces.
    istringstream ss(str);
    
    // Traverse through all words
    /*
    do {
        // Read a word
        string word;
        ss >> word;
        
        // Print the read word
        cout << word << endl;
        store(topwords,word );
        
        // While there is more to read
    } while (ss);
    */
    string word;
    while (ss >> word) {
        //cout << word << endl;
        const std::regex sanitized{ R"([-[\]{}()*+?.,\^$|#\s])" };
        
        std::stringstream result;
        std::regex_replace(std::ostream_iterator<char>(result),word.begin(),word.end(),sanitized,"");
        
        //store(topwords,word );
        store(topwords,result.str() );
    }
}

void readReadFile(string &fileName){
    std::cout << "fileName" << fileName << endl;
    std::ifstream file(fileName);
    std::string str;
    while (std::getline(file,str)) {
        //std::cout << str << "\n";
        removeDupWord(str);
        //store(topwords,str);
    }
}

bool compareFunction (const std::pair<std::string,int> &a,const std::pair<std::string,int> &b) {
    
    return a.first<b.first; // sort by letter
}

bool compareFunction2 (const std::pair<std::string,int> &b) {
    
    return a.second>b.second; // sort by count
}

bool cmp(pair<string,int> &A,pair<string,int> &B) {
    return A.second < B.second;
}

void check(vector<pair <string,int> > &topwords){
    std::pair<string,int> mostUsedWord = make_pair("",0);
    for(auto ii : topwords){
        std::cout << "word: " << ii.first << " count: " << ii.second << endl;
        if(ii.second > mostUsedWord.second){
            mostUsedWord.first = ii.first;
            mostUsedWord.second = ii.second;
        }
    }
    std::cout << "most used Word: " << mostUsedWord.first << " x " << mostUsedWord.second << " Times." << endl;
           
}

void get_higestTopTenValues(vector<pair <string,int> > &topwords){
    std::sort(topwords.begin(),compareFunction2);//sort the vector by count
    int MAX = std::max_element(topwords.begin(),cmp)->second;
    std::cout << "max: " << MAX << endl;
    for(auto ii : topwords){
        //std::cout << "word: " << ii.first << " count: " << ii.second << endl;
        if(ii.second >= (MAX - 10)){
            std::cout << ii.first << " " << ii.second << endl;
            
        }
    }
}

void get_LowestTopTenValues(vector<pair <string,compareFunction2);//sort the vector by count
    int MIN = std::min_element(topwords.begin(),cmp)->second;
    std::cout << "min: " << MIN << endl;
    for(auto ii : topwords){
        //std::cout << "word: " << ii.first << " count: " << ii.second << endl;
        if(ii.second <= (MIN + 9)){
            std::cout << ii.first << " " << ii.second << endl;
            
        }
    }
}

int main ()
{
    std::string word,fileName;
    
    fileName = "input.txt";
    readReadFile(fileName);
    
    topwords.push_back( make_pair("ba",1) );
    topwords.push_back( make_pair("bu",1) );
    topwords.push_back( make_pair("hmmm",1) );
    topwords.push_back( make_pair("what",1) );
    topwords.push_back( make_pair("and",1) );
    topwords.push_back( make_pair("hello",1) );
    
    word = "hellos";
    
    store(topwords,word);
    store(topwords,word);
    
    word = "hello";
    
    store(topwords,word);
    
    std::sort(topwords.begin(),compareFunction);//sort the vector by letter
    // or
    //std::sort(topwords.begin(),compareFunction2);//sort the vector by count
    
    
    std::cout << "---------------------------------------" << endl;
    std::cout << " get all values" << endl;
    std::cout << "---------------------------------------" << endl;
    check(topwords);
    
    
    std::cout << "---------------------------------------" << endl;
    std::cout << " get the top 10 highest values" << endl;
    std::cout << "---------------------------------------" << endl;
    get_higestTopTenValues(topwords);
    
    std::cout << "---------------------------------------" << endl;
    std::cout << " get the top 10 lowest values" << endl;
    std::cout << "---------------------------------------" << endl;
    get_LowestTopTenValues(topwords);
    
}
,

这个问题很久以前就有人回答了。我偶然发现了这个问题并回答了,我发现一切都过于复杂。

因此,我想添加一个使用现有 STL 元素的更现代的 C++ 解决方案。

这使得代码更加紧凑。

请看下面:

#include <iostream>
#include <utility>
#include <unordered_map>
#include <vector>
#include <algorithm>
#include <string>
#include <fstream>

const std::string fileName{"r:\\loremipsum.txt"};

int main() {

    if (std::ifstream textFileStream{ fileName }; textFileStream) {

        // Here we store the count of all words
        std::unordered_map<std::string,size_t> counter{};

        size_t countOfOverallWords{}; // Counter for the number of all words

        // Read all words from file,remove punctuation,and count teh occurence
        for (std::string word; textFileStream >> word; counter[word]++) {
            word.erase(std::remove_if(word.begin(),ispunct),word.end());
            ++countOfOverallWords;
        }
        // For storing the top 10
        std::vector<std::pair<std::string,size_t>> top(10);

        // Get top 10
        std::partial_sort_copy(counter.begin(),counter.end(),top.begin(),top.end(),[](const std::pair<std::string,size_t >& p1,size_t>& p2) { return p1.second > p2.second; });

        // Now show result
        std::cout << "Count of overall words:\t " << countOfOverallWords << "\nCount of unique words:\t " << counter.size() << "\n\nTop 10:\n";
        for (const auto& t : top) std::cout << "Value: " << t.first << "\t Count: " << t.second << '\n';
    }
    else std::cerr << "\n\nError: Could not open source file '" << fileName << "'\n\n";

    return 0;
}

使用 Microsoft Visual Studio Community 2019 版本 16.8.2 进行开发和测试。

使用带有标志 --std=c++17 -Wall -Wextra -Wpedantic

的 clang11.0 和 gcc10.2 额外编译和测试

语言:C++17

相关问答

依赖报错 idea导入项目后依赖报错,解决方案:https://blog....
错误1:代码生成器依赖和mybatis依赖冲突 启动项目时报错如下...
错误1:gradle项目控制台输出为乱码 # 解决方案:https://bl...
错误还原:在查询的过程中,传入的workType为0时,该条件不起...
报错如下,gcc版本太低 ^ server.c:5346:31: 错误:‘struct...