On Tuesday, 4 February 2014 at 00:04:23 UTC, Rene Zwanenburg
wrote:
On Monday, 3 February 2014 at 23:50:54 UTC, bearophile wrote:
Rene Zwanenburg:
The problem is speed. I'm using LockingTextReader in
std.stdio, but it't not nearly fast enough. On my system it
only reads about 3 MB/s with one core spending all it's time
in IO calls.
Are you reading the text by lines? In Bugzilla there is a
byLineFast:
https://d.puremagic.com/issues/show_bug.cgi?id=11810
Bye,
bearophile
Nope, I'm feeding it to csvReader which uses an input range of
characters. Come to think of it..
Well this is embarassing, I've been sloppy with my profiling
:). It appears the time is actually spent converting strings to
doubles, done by csvReader to read a row into my Record struct.
No way to speed that up I suppose. Still I find it surprising
that parsing doubles is so slow.
Parsing should be faster than I/O. Set up two buffers and have
one thread reading into buffer A while you parse buffer B with a
second thread.
