Skip to content

Improve memory usage of the CSV reader

Wiebel, Alexander requested to merge stringViewCSV into main

The csv-reader stores a lot of duplicated strings. By replacing std::string with std::string_view only necessary strings are stored in memory, while other copies are only links to the original data. This works very good, because the WDataSetCSV also stores each row. Now, loading a file of 3 GB only results in roughly 3 GB. Before that change, the memory usage was up to 10 times the file size.

We still need to check, if other modules work as intended. I changed several usages of the csv-reader in the proton modules, but did not commit this, yet.

Merge request reports

Loading