On Thursday, 4 January 2018 at 15:48:21 UTC, Steven Schveighoffer
wrote:
It's now been updated, see version 0.0.3.
Note, the performance isn't something I focused on. I'll note
that gzcat | wc -l is 2x faster than your simple example on
that file.
I can think of a couple reasons for this:
On 1/4/18 7:23 AM, Andrew wrote:
On Thursday, 4 January 2018 at 12:15:27 UTC, Steven Schveighoffer wrote:
In any case, I'll figure out how to deal with concatenated gzip file,
and update iopipe. Next version will focus on a bunch of stuff
relating to the 2 zip threads recently posted here.
On Thursday, 4 January 2018 at 12:15:27 UTC, Steven Schveighoffer
wrote:
On 1/4/18 7:01 AM, Andrew wrote:
Ah thank you, that makes sense. These types of files are
compressed using the bgzip utility so that the file can be
indexed meaning specific rows extracted quickly (there's more
details
On 1/4/18 7:01 AM, Andrew wrote:
Ah thank you, that makes sense. These types of files are compressed
using the bgzip utility so that the file can be indexed meaning specific
rows extracted quickly (there's more details of this here
http://www.htslib.org/doc/tabix.html and the code can be
On Thursday, 4 January 2018 at 02:44:09 UTC, Steven Schveighoffer
wrote:
On 1/3/18 12:03 PM, Andrew wrote:
Thanks for looking into this.
So it looks like the file you have is a concatenated gzip file.
If I gunzip the file and recompress it, it works properly.
Looking at the docs of zlib
On 1/3/18 12:03 PM, Andrew wrote:
Thanks for looking into this.
So it looks like the file you have is a concatenated gzip file. If I
gunzip the file and recompress it, it works properly.
Looking at the docs of zlib inflate [1]:
" Unlike the gunzip utility and gzread() ..., inflate() will
On Wednesday, 3 January 2018 at 16:09:19 UTC, Steven
Schveighoffer wrote:
On 1/3/18 9:45 AM, Andrew wrote:
Hi,
I have a very large gziped text file (all ASCII characters and
~500GB) that I want to stream and process line-by-line, and I
thought the iopipe library would be perfect for this,
On 1/3/18 9:45 AM, Andrew wrote:
Hi,
I have a very large gziped text file (all ASCII characters and ~500GB)
that I want to stream and process line-by-line, and I thought the iopipe
library would be perfect for this, but I can't seem to get it to work.
So far, this is the closest I have to
Hi,
I have a very large gziped text file (all ASCII characters and
~500GB) that I want to stream and process line-by-line, and I
thought the iopipe library would be perfect for this, but I can't
seem to get it to work. So far, this is the closest I have to
getting it to work:
import
