Improve memory usage of the CSV reader
The csv-reader stores a lot of duplicated strings. By replacing std::string
with std::string_view
only necessary strings are stored in memory, while other copies are only links to the original data. This works very good, because the WDataSetCSV
also stores each row. Now, loading a file of 3 GB only results in roughly 3 GB. Before that change, the memory usage was up to 10 times the file size.
We still need to check, if other modules work as intended. I changed several usages of the csv-reader in the proton modules, but did not commit this, yet.