Re: Performant method for reading huge text files

2014-02-06 Thread Marco Leise
Am Tue, 04 Feb 2014 00:04:22 + schrieb Rene Zwanenburg renezwanenb...@gmail.com: On Monday, 3 February 2014 at 23:50:54 UTC, bearophile wrote: Rene Zwanenburg: The problem is speed. I'm using LockingTextReader in std.stdio, but it't not nearly fast enough. On my system it only

Re: Performant method for reading huge text files

2014-02-05 Thread Kagamin
You can also try a BufferedRange. http://forum.dlang.org/thread/l9q66g$2he3$1...@digitalmars.com

Re: Performant method for reading huge text files

2014-02-04 Thread Chris Williams
On Tuesday, 4 February 2014 at 00:04:23 UTC, Rene Zwanenburg wrote: On Monday, 3 February 2014 at 23:50:54 UTC, bearophile wrote: Rene Zwanenburg: The problem is speed. I'm using LockingTextReader in std.stdio, but it't not nearly fast enough. On my system it only reads about 3 MB/s with one

Re: Performant method for reading huge text files

2014-02-04 Thread Chris Williams
Parsing should be faster than I/O. Set up two buffers and have one thread reading into buffer A while you parse buffer B with a second thread. ...and then flip buffers whenever the slower of the two has completed.

Re: Performant method for reading huge text files

2014-02-03 Thread bearophile
Rene Zwanenburg: The problem is speed. I'm using LockingTextReader in std.stdio, but it't not nearly fast enough. On my system it only reads about 3 MB/s with one core spending all it's time in IO calls. Are you reading the text by lines? In Bugzilla there is a byLineFast:

Re: Performant method for reading huge text files

2014-02-03 Thread Rene Zwanenburg
On Monday, 3 February 2014 at 23:50:54 UTC, bearophile wrote: Rene Zwanenburg: The problem is speed. I'm using LockingTextReader in std.stdio, but it't not nearly fast enough. On my system it only reads about 3 MB/s with one core spending all it's time in IO calls. Are you reading the text