使用内存映射文件在 C++ 中解析二进制文件太慢

编程问答 2022-06-02

问题描述

我正在尝试以整数方式解析二进制文件，以检查整数值是否满足特定条件，但循环非常慢。

此外，我发现 memory-mapped file 是将文件快速读入内存的最快速度，因此我使用了以下基于 Boost 的代码：

unsigned long long int get_file_size(const char *file_path) {
    const filesystem::path file{file_path};
    const auto generic_path = file.generic_path();
    return filesystem::file_size(generic_path);
}

boost::iostreams::mapped_file_source read_bytes(const char *file_path,const unsigned long long int offset,const unsigned long long int length) {
    boost::iostreams::mapped_file_params parameters;
    parameters.path = file_path;
    parameters.length = static_cast<size_t>(length);
    parameters.flags = boost::iostreams::mapped_file::mapmode::readonly;
    parameters.offset = static_cast<boost::iostreams::stream_offset>(offset);

    boost::iostreams::mapped_file_source file;

    file.open(parameters);
    return file;
}

boost::iostreams::mapped_file_source read_bytes(const char *file_path) {
    const auto file_size = get_file_size(file_path);
    const auto mapped_file_source = read_bytes(file_path,file_size);
    return mapped_file_source;
}

我的测试用例大致如下：

inline auto test_parsing_binary_file_performance() {
    const auto start_time = get_time();
    const std::filesystem::path input_file_path = "...";
    const auto mapped_file_source = read_bytes(input_file_path.string().c_str());
    const auto file_buffer = mapped_file_source.data();
    const auto file_buffer_size = mapped_file_source.size();
    LOG_S(INFO) << "File buffer size: " << file_buffer_size;
    auto printed_lap = (long) (file_buffer_size / (double) 1000);
    printed_lap = round_to_nearest_multiple(printed_lap,sizeof(int));
    LOG_S(INFO) << "Printed lap: " << printed_lap;
    std::vector<int> values;
    values.reserve(file_buffer_size / sizeof(int)); // Pre-allocate a large enough vector
    // Iterate over every integer
    for (auto file_buffer_index = 0; file_buffer_index < file_buffer_size; file_buffer_index += sizeof(int)) {
        const auto value = *(int *) &file_buffer[file_buffer_index];
        if (value >= 0x30000000 && value < 0x49000000 - sizeof(int) + 1) {
            values.push_back(value);
        }

        if (file_buffer_index % printed_lap == 0) {
            LOG_S(INFO) << std::setprecision(4) << file_buffer_index / (double) file_buffer_size * 100 << "%";
        }
    }

    LOG_S(INFO) << "Values found count: " << values.size();

    print_time_taken(start_time,false,"Parsing binary file");
}

memory-mapped file 读取几乎按预期立即完成，但尽管硬件出色（SSD 等），但在我的机器上按整数迭代它太慢了：

2020-12-20 13:04:35.124 (   0.019s) [main thread     ]Tests.hpp:387   INFO| File buffer size: 419430400
2020-12-20 13:04:35.124 (   0.019s) [main thread     ]Tests.hpp:390   INFO| Printed lap: 419432
2020-12-20 13:04:35.135 (   0.029s) [main thread     ]Tests.hpp:405   INFO| 0%
2020-12-20 13:04:35.171 (   0.065s) [main thread     ]Tests.hpp:405   INFO| 0.1%
2020-12-20 13:04:35.196 (   0.091s) [main thread     ]Tests.hpp:405   INFO| 0.2%
2020-12-20 13:04:35.216 (   0.111s) [main thread     ]Tests.hpp:405   INFO| 0.3%
2020-12-20 13:04:35.241 (   0.136s) [main thread     ]Tests.hpp:405   INFO| 0.4%
2020-12-20 13:04:35.272 (   0.167s) [main thread     ]Tests.hpp:405   INFO| 0.5%
2020-12-20 13:04:35.293 (   0.188s) [main thread     ]Tests.hpp:405   INFO| 0.6%
2020-12-20 13:04:35.314 (   0.209s) [main thread     ]Tests.hpp:405   INFO| 0.7%
2020-12-20 13:04:35.343 (   0.237s) [main thread     ]Tests.hpp:405   INFO| 0.8%
2020-12-20 13:04:35.366 (   0.261s) [main thread     ]Tests.hpp:405   INFO| 0.9%
2020-12-20 13:04:35.399 (   0.293s) [main thread     ]Tests.hpp:405   INFO| 1%
2020-12-20 13:04:35.421 (   0.315s) [main thread     ]Tests.hpp:405   INFO| 1.1%
2020-12-20 13:04:35.447 (   0.341s) [main thread     ]Tests.hpp:405   INFO| 1.2%
2020-12-20 13:04:35.468 (   0.362s) [main thread     ]Tests.hpp:405   INFO| 1.3%
2020-12-20 13:04:35.487 (   0.382s) [main thread     ]Tests.hpp:405   INFO| 1.4%
2020-12-20 13:04:35.520 (   0.414s) [main thread     ]Tests.hpp:405   INFO| 1.5%
2020-12-20 13:04:35.540 (   0.435s) [main thread     ]Tests.hpp:405   INFO| 1.6%
2020-12-20 13:04:35.564 (   0.458s) [main thread     ]Tests.hpp:405   INFO| 1.7%
2020-12-20 13:04:35.586 (   0.480s) [main thread     ]Tests.hpp:405   INFO| 1.8%
2020-12-20 13:04:35.608 (   0.503s) [main thread     ]Tests.hpp:405   INFO| 1.9%
2020-12-20 13:04:35.636 (   0.531s) [main thread     ]Tests.hpp:405   INFO| 2%
2020-12-20 13:04:35.658 (   0.552s) [main thread     ]Tests.hpp:405   INFO| 2.1%
2020-12-20 13:04:35.679 (   0.574s) [main thread     ]Tests.hpp:405   INFO| 2.2%
2020-12-20 13:04:35.702 (   0.597s) [main thread     ]Tests.hpp:405   INFO| 2.3%
2020-12-20 13:04:35.727 (   0.622s) [main thread     ]Tests.hpp:405   INFO| 2.4%
2020-12-20 13:04:35.769 (   0.664s) [main thread     ]Tests.hpp:405   INFO| 2.5%
2020-12-20 13:04:35.802 (   0.697s) [main thread     ]Tests.hpp:405   INFO| 2.6%
2020-12-20 13:04:35.831 (   0.726s) [main thread     ]Tests.hpp:405   INFO| 2.7%
2020-12-20 13:04:35.860 (   0.754s) [main thread     ]Tests.hpp:405   INFO| 2.8%
2020-12-20 13:04:35.887 (   0.781s) [main thread     ]Tests.hpp:405   INFO| 2.9%
2020-12-20 13:04:35.924 (   0.818s) [main thread     ]Tests.hpp:405   INFO| 3%
2020-12-20 13:04:35.956 (   0.850s) [main thread     ]Tests.hpp:405   INFO| 3.1%
2020-12-20 13:04:35.998 (   0.893s) [main thread     ]Tests.hpp:405   INFO| 3.2%
2020-12-20 13:04:36.033 (   0.928s) [main thread     ]Tests.hpp:405   INFO| 3.3%
2020-12-20 13:04:36.060 (   0.955s) [main thread     ]Tests.hpp:405   INFO| 3.4%
2020-12-20 13:04:36.102 (   0.997s) [main thread     ]Tests.hpp:405   INFO| 3.5%
2020-12-20 13:04:36.132 (   1.026s) [main thread     ]Tests.hpp:405   INFO| 3.6%
...
2020-12-20 13:05:03.456 (  28.351s) [main thread     ]Tests.hpp:410   INFO| Values found count: 10650389
2020-12-20 13:05:03.456 (  28.351s) [main thread     ]          benchmark.cpp:31    INFO| Parsing binary file took 28.341 second(s)

解析这些 419 MB 总是需要大约 28 - 70 秒。即使在 Release 模式下编译也无济于事。有什么办法可以减少这个时间吗？我正在执行的操作似乎没有那么低效。

请注意，我正在使用 Linux 64-bit 编译 GCC 10。

编辑：
正如评论中所建议的，将 memory-mapped file 与 advise() 一起使用也无助于性能：

boost::interprocess::file_mapping file_mapping(input_file_path.string().data(),boost::interprocess::read_only);
boost::interprocess::mapped_region mapped_region(file_mapping,boost::interprocess::read_only);
mapped_region.advise(boost::interprocess::mapped_region::advice_sequential);
const auto file_buffer = (char *) mapped_region.get_address();
const auto file_buffer_size = mapped_region.get_size();
...

迄今为止通过考虑评论/答案而获得的经验教训：

  • 使用 advise(boost::interprocess::mapped_region::advice_sequential) 没有帮助
  • 不调用 reserve() 或以完全正确的大小调用它可以使性能翻倍
  • 直接在 int * 上迭代比在 char * 上迭代要慢一些
  • 使用 std::set 收集结果比使用 std::vector 慢一些
  • 进度日志对性能来说无关紧要

解决方法

正如 xanatos View view = LayoutInflater.from(context).inflate(R.layout.row_users,(android.view.ViewGroup) ViewGroup,false); 所暗示的那样，它们在性能上具有欺骗性，因为它们并没有真正将整个文件瞬间读入内存。在处理过程中，页面丢失导致多次磁盘访问，严重降低了性能。

在这种情况下，先将整个文件读入内存，然后遍历内存，效率更高：

String userImage = usersList.get(position).getImage();

现在性能更容易接受，总共大约 for (DataSnapshot ds: dataSnapshot.getChildren())。

,

这让我想起了大约 40 年前我第一次遇到缓慢。由衡量进度的百分比条引起。注释掉那部分并再次测量。 还要测量容量储备，并检查所需的实际容量 - 如果是 1%，那么您就是在浪费空间和时间。

  • unsigned long long 可能很昂贵。 unsignedlong 还不够吗？
  • 取模，除法可能会额外花费。
  • 进度记录可能很慢，最好是一个单独的线程，然后检查 刷新（违反直觉）是否可能不会更快。

所以：

const auto pct_factor = file_buffer_size == 0 ? 0.0 : 100 / (double)file_buffer_size;
values.reserve(file_buffer_size / sizeof(int));
for (auto file_buffer_index = 0,long pct_countdown = 0; file_buffer_index < file_buffer_size; file_buffer_index += sizeof(int)) {
    const auto value = *(int *) &file_buffer[file_buffer_index];
    if (value >= 0x30000000 && value < 0x49000000 - sizeof(int) + 1) {
        values.push_back(value);
    }

    if (pct_countdown-- < 0) {
        pct_countdown = printed_lap;
        const auto pct = file_buffer_index * pct_factor;
        LOG_S(INFO) << std::setprecision(4) << pct << "%";
    }
}
  • 整数百分比会更好。有点舍弃精度。
  • 批量数据 values - 是否需要这样的数据。一套就足够了。

我承认我对 *(int *) 有疑问。使用 int* 指针并增加它似乎也更直接。

c++memory-mapped-filesmicro-optimizationoptimization