On Thursday, 16 April 2015 at 17:13:25 UTC, Laeeth Isharc wrote:
For CSV files, what I found is that parsing is quite slow (and
memory intensive).
If your sure that CSV reading is the culprit, writing a custom
parser could help. It's possible to load a CSV file with almost
no memory overhead. What I would do:
- Use std.mmfile with Mode.readCopyOnWrite to map the file into
memory.
- Iterate over the lines, and then over the fields using
std.algorithm.splitter.
- Don't copy, but return slices into the mapped memory.
- If a field needs to be unescaped, this can be done in-place.
Unescaping never makes a string longer, and the original file
won't be modified thanks to COW (private mapping).
