On Tuesday, 12 February 2013 at 12:45:26 UTC, monarch_dodra wrote:
On Tuesday, 12 February 2013 at 12:02:59 UTC, bioinfornatics
wrote:
instead to use memcpy I try with slicing ~ lines 136 :
_hardBuffer[ 0 .. moveSize] = _hardBuffer[_bufPosition ..
moveSize + _bufPosition];
I get same perf
I think I figured out why I'm getting different results than
you guys are, on my windows machine.
AFAIK, file reads in windows are done natively asynchronously.
I wrote a multi-threaded version of the parser, with a thread
dedicated to reading the file, while the main thread parses the
read buffers.
I'm getting EXACTLY 0% performance improvement. Not better, not
worst, just 0%.
I'd have to try again on my SSD. Right now, I'm parsing the
file 6 Gig file in 60 seconds, which is the limit of my HDD. As
a matter of fact, just *reading* the files takes the EXACT same
amount of time as parsing it...
This takes 60 seconds.
//----
auto input = File(args[1], "rb");
ubyte[] buffer = new ubyte[](BufferSize);
do{
buffer = input.rawRead(buffer);
}while(buffer.length);
//----
This takes 60 seconds too.
//----
Parser parser = new Parser(args[1]);
foreach(q; parser)
foreach(char c; q.sequence)
globalNucleic.collect(c);
}
//----
So at this point, I'd need to test on my Linux box, or publish
the code so you can tell me how I'm doing.
I'm still tweaking the code to publish something readable, as
there is a lot of sketchy code right now.
I'm also implementing a correct exception handling, so that if
there is an erroneous entry, an exception is thrown. However,
all the erroneous data is parsed out of the file, and placed
inside the exception. This means that:
a) You can inspect the erroneous data
b) You can skip the erroneous data, and parse the rest of the
file.
Once I deliver the code with the multi-threaded code activated,
you should get some better performance on Linux.
When "1.0" is ready, I'll create a github project for it, so
work can be done parallel on it.
about threaded version is possible to use get file size function
to split it in several thread.
Use fseek read end of section return it to detect end of split to
used
