问题描述
|
我正在尝试运行一个程序来分析一堆包含数字的文本文件。文本文件的总大小约为12 MB，我从360个文本文件中分别提取了1000倍，并将其放入向量中。我的问题是我进入了文本文件列表的一半，然后我的计算机速度变慢，直到不再处理任何文件。该程序不是无限循环，但我认为使用过多内存存在问题。有没有更好的方法来存储不会占用太多内存的数据？
其他可能相关的系统信息：
运行Linux
8 GB内存
已安装Cern ROOT框架（不过，我不知道如何减少内存占用）
英特尔至强四核处理器
如果您需要其他信息，我将更新此列表
编辑：我跑了顶，我的程序使用更多的内存，并且一旦它超过80％我就杀死了它。有很多代码，因此我将挑选出分配内存并共享的位。
编辑2：我的代码：
void FileAnalysis::doWork(std::string opath,std::string oName)
{
//sets the ouput filepath and the name of the file to contain the results
outpath = opath;
outname = oName;
//Reads the data source and writes it to a text file before pushing the filenames into a vector
setinput();
//Goes through the files queue and analyzes each file
while(!files.empty())
{
//Puts all of the data points from the next file onto the points vector then deletes the file from the files queue
readNext();
//Places all of the min or max points into their respective vectors
analyze();
//Calculates the averages and the offset and pushes those into their respective vectors
calcAvg();
}
makeGraph();
}
//Creates the vector of files to be read
void FileAnalysis::setinput()
{
string sysCall = \"\",filepath=\"\",temp;
filepath = outpath+\"filenames.txt\";
sysCall = \"ls \"+dataFolder+\" > \"+filepath;
system(sysCall.c_str());
ifstream allfiles(filepath.c_str());
while (!allfiles.eof())
{
getline(allfiles,temp);
files.push(temp);
}
}
//Places the data from the next filename into the files vector,then deletes the filename from the vector
void FileAnalysis::readNext()
{
cout<<\"Reading from \"<<dataFolder<<files.front()<<endl;
ifstream curfile((dataFolder+files.front()).c_str());
string temp,temptodouble;
double tempval;
getline(curfile,temp);
while (!curfile.eof())
{
if (temp.size()>0)
{
unsigned long pos = temp.find_first_of(\"\\t\");
temptodouble = temp.substr(pos,pos);
tempval = atof(temptodouble.c_str());
points.push_back(tempval);
}
getline(curfile,temp);
}
setTime();
files.pop();
}
//Sets the maxpoints and minpoints vectors from the points vector and adds the vectors to the allmax and allmin vectors
void FileAnalysis::analyze()
{
for (unsigned int i = 1; i<points.size()-1; i++)
{
if (points[i]>points[i-1]&&points[i]>points[i+1])
{
maxpoints.push_back(points[i]);
}
if (points[i]<points[i-1]&&points[i]<points[i+1])
{
minpoints.push_back(points[i]);
}
}
allmax.push_back(maxpoints);
allmin.push_back(minpoints);
}
//Calculates the average max and min points from the maxpoints and minpoints vector and adds those averages to the avgmax and avgmin vectors,and adds the offset to the offset vector
void FileAnalysis::calcAvg()
{
double maxtotal = 0,mintotal = 0;
for (unsigned int i = 0; i<maxpoints.size(); i++)
{
maxtotal+=maxpoints[i];
}
for (unsigned int i = 0; i<minpoints.size(); i++)
{
mintotal+=minpoints[i];
}
avgmax.push_back(maxtotal/maxpoints.size());
avgmin.push_back(mintotal/minpoints.size());
offset.push_back((maxtotal+mintotal)/2);
}
解决方法
可以无休止地进行优化，但是我的直接反应是使用除vector之外的其他容器。请记住，向量的存储是在内存中按顺序分配的，这意味着如果没有足够的当前空间来容纳新元素，则添加其他元素会导致整个向量的重新分配。
请尝试针对常量插入进行优化的容器，例如队列或列表。
或者，如果需要向量，则可以尝试预先分配预期的内存占用以避免连续重新分配。参见
vector.reserve()
int numberOfItems = 1000;
int numberOfFiles = 360;
size_type totalExpectedSize = (numberOfItems) * (numberOfFiles);
myVector.reserve( totalExpectedSize );
for (unsigned int i = 1; i<points.size()-1; i++)
{
if (points[i]>points[i-1]&&points[i]>points[i+1])
{
maxpoints.push_back(points[i]);
}
if (points[i]<points[i-1]&&points[i]<points[i+1])
{
minpoints.push_back(points[i]);
}
}
allmax.push_back(maxpoints);
allmin.push_back(minpoints);
File 1: 2 1 2 1 2 1 2 1 2 1 2
minpoints: [1 1 1 1 1]
allmin: [1 1 1 1 1]
File 2: 3 2 3 2 3 2 3 2 3 2 3
minpoints: [1 1 1 1 1 2 2 2 2 2]
allmin: [1 1 1 1 1 1 1 1 1 1 2 2 2 2 2]
File 3: 4 3 4 3 4 3 4 3 4 3 4
minpoints: [1 1 1 1 1 2 2 2 2 2 3 3 3 3 3]
allmin: [1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3]
makeGraph()
valgrind
readNext()
eof()
getline()