Re: Reading bzipped files

[via Digitalmars-d-learn]

On Monday, 4 May 2015 at 20:53:27 UTC, monty wrote:
cool. btw (at least GzipByLine) its slw if you compare it 
to gzcat and pipe it into stdin and use ByLineFast.

i think its mainly the buffer appending that  suboptimal.


Could anyone please point out if this is possible to fix somehow? 
Is the solution to avoid the GC and do buffer reuse somehow?

Re: Reading bzipped files

[via Digitalmars-d-learn]

On Tuesday, 5 May 2015 at 06:48:36 UTC, Per Nordlöw wrote:
Could anyone please point out if this is possible to fix 
somehow? Is the solution to avoid the GC and do buffer reuse 
somehow?


I *really* need this for reading Gigabytes of DBpedia data...

Re: Reading bzipped files

[monty via Digitalmars-d-learn]

On Tuesday, 5 May 2015 at 06:50:12 UTC, Per Nordlöw wrote:

On Tuesday, 5 May 2015 at 06:48:36 UTC, Per Nordlöw wrote:
Could anyone please point out if this is possible to fix 
somehow? Is the solution to avoid the GC and do buffer reuse 
somehow?


I *really* need this for reading Gigabytes of DBpedia data...


my approach would be to more or less port the ByLineFast approach.
i just haven't had any time to do this yet.

maybe this could work too though:
https://github.com/biod/BioD/tree/master/bio/core/bgzf


also check out:
https://github.com/agordon/fileslurp
https://github.com/biod/BioD/blob/master/bio/core/utils/bylinefast.d

Re: Reading bzipped files

[via Digitalmars-d-learn]

On Tuesday, 5 May 2015 at 07:11:36 UTC, monty wrote:

also check out:
https://github.com/agordon/fileslurp
https://github.com/biod/BioD/blob/master/bio/core/utils/bylinefast.d


Thanks!

Re: Reading bzipped files

[via Digitalmars-d-learn]

On Saturday, 2 May 2015 at 13:50:10 UTC, Per Nordlöw wrote:

Should probably be built on top of

http://code.dlang.org/packages/bzip2


Docs here:

http://www.bzip.org/1.0.3/html/hl-interface.html

Re: Reading bzipped files

[monty via Digitalmars-d-learn]

On Sunday, 3 May 2015 at 14:37:32 UTC, Per Nordlöw wrote:

On Saturday, 2 May 2015 at 20:37:44 UTC, tom wrote:

i use Stephan Schiffels code from
http://forum.dlang.org/thread/djhteyhpcnaskpabx...@forum.dlang.org?page=2


See polished version at:

https://github.com/nordlow/justd/blob/master/zio.d


cool. btw (at least GzipByLine) its slw if you compare it to 
gzcat and pipe it into stdin and use ByLineFast.

i think its mainly the buffer appending that  suboptimal.

Re: Reading bzipped files

[via Digitalmars-d-learn]

On Saturday, 2 May 2015 at 20:37:44 UTC, tom wrote:

i use Stephan Schiffels code from
http://forum.dlang.org/thread/djhteyhpcnaskpabx...@forum.dlang.org?page=2


See polished version at:

https://github.com/nordlow/justd/blob/master/zio.d

Reading bzipped files

[via Digitalmars-d-learn]

Have anybody cooked up any range adaptors for on the fly decoding 
of bzipped files? Preferable compatible with phobos standard 
interfaces for file io.


Should probably be built on top of

http://code.dlang.org/packages/bzip2

Re: Reading bzipped files

[tom via Digitalmars-d-learn]

On Saturday, 2 May 2015 at 13:50:10 UTC, Per Nordlöw wrote:
Have anybody cooked up any range adaptors for on the fly 
decoding of bzipped files? Preferable compatible with phobos 
standard interfaces for file io.


Should probably be built on top of

http://code.dlang.org/packages/bzip2


i use Stephan Schiffels code from

http://forum.dlang.org/thread/djhteyhpcnaskpabx...@forum.dlang.org?page=2