[Python-ideas] Re: Accelerate zlib and gzip libraries with Intel's Storage Acceleration Libraries

2020-08-17 Thread David Mertz
Testing myself on a large fasta file, I find far faster performance with igzip, but also far lower compression rate (than gzip -6, the default). My gzipped fasta is nearly half the size of igzip at its -3 maximum. I haven't tried 'gzip -3' to compare that size. This limitation seems to limit this

[Python-ideas] Re: Accelerate zlib and gzip libraries with Intel's Storage Acceleration Libraries

2020-08-17 Thread Ben Rudiak-Gould
On Mon, Aug 17, 2020 at 8:02 AM Steven D'Aprano wrote: > Perhaps I have misunderstood, but isn't this a pure implementation > change, with no user visible API changes and backward compatible output? > According to the documentation [1], it only supports compression levels 0-3. They're supposed t

[Python-ideas] Re: Accelerate zlib and gzip libraries with Intel's Storage Acceleration Libraries

2020-08-17 Thread Chris Angelico
On Tue, Aug 18, 2020 at 3:00 AM Christopher Barker wrote: > > On Mon, Aug 17, 2020 at 8:42 AM Chris Angelico > >> but does it have a compatible API? If it does - if you can drop it in >> >> and everything behaves equivalently - then it sounds like the sort of >> >> thing that can be included in Py

[Python-ideas] Re: Accelerate zlib and gzip libraries with Intel's Storage Acceleration Libraries

2020-08-17 Thread Christopher Barker
On Mon, Aug 17, 2020 at 8:42 AM Chris Angelico but does it have a compatible API? If it does - if you can drop it in > > and everything behaves equivalently - then it sounds like the sort of > > thing that can be included in Python 3.10 with minimal fuss. I’m confused - if it is fully compatible

[Python-ideas] Re: Accelerate zlib and gzip libraries with Intel's Storage Acceleration Libraries

2020-08-17 Thread Chris Angelico
On Tue, Aug 18, 2020 at 1:05 AM Steven D'Aprano wrote: > > On Mon, Aug 17, 2020 at 04:08:54PM +0200, Stefan Behnel wrote: > > > I re-opened the ticket to allow for some discussion over there in order to > > understand the implications better. But I agree that a third-party package > > on PyPI seem

[Python-ideas] Re: Accelerate zlib and gzip libraries with Intel's Storage Acceleration Libraries

2020-08-17 Thread Stefan Behnel
Steven D'Aprano schrieb am 17.08.20 um 17:00: > On Mon, Aug 17, 2020 at 04:08:54PM +0200, Stefan Behnel wrote: > >> I re-opened the ticket to allow for some discussion over there in order to >> understand the implications better. But I agree that a third-party package >> on PyPI seems like a good

[Python-ideas] Re: Accelerate zlib and gzip libraries with Intel's Storage Acceleration Libraries

2020-08-17 Thread Steven D'Aprano
On Mon, Aug 17, 2020 at 04:08:54PM +0200, Stefan Behnel wrote: > I re-opened the ticket to allow for some discussion over there in order to > understand the implications better. But I agree that a third-party package > on PyPI seems like a good first step, also as a backport. Perhaps I have misun

[Python-ideas] Re: Accelerate zlib and gzip libraries with Intel's Storage Acceleration Libraries

2020-08-17 Thread Stefan Behnel
Antoine Pitrou schrieb am 17.08.20 um 15:00: > On Mon, 17 Aug 2020 08:49:23 - > "Ruben Vorderman" wrote: >> Dear python developers, >> >> As a bioinformatician I work a lot with gzip-compressed data. Recently I >> discovered Intel's Storage Acceleration Libraries at >> https://github.com/inte

[Python-ideas] Re: Accelerate zlib and gzip libraries with Intel's Storage Acceleration Libraries

2020-08-17 Thread Chris Angelico
On Mon, Aug 17, 2020 at 10:48 PM Ruben Vorderman wrote: > > Dear python developers, > > As a bioinformatician I work a lot with gzip-compressed data. Recently I > discovered Intel's Storage Acceleration Libraries at > https://github.com/intel/isa-l. These people implemented the DEFLATE and > IN

[Python-ideas] Re: Accelerate zlib and gzip libraries with Intel's Storage Acceleration Libraries

2020-08-17 Thread Antoine Pitrou
On Mon, 17 Aug 2020 08:49:23 - "Ruben Vorderman" wrote: > Dear python developers, > > As a bioinformatician I work a lot with gzip-compressed data. Recently I > discovered Intel's Storage Acceleration Libraries at > https://github.com/intel/isa-l. These people implemented the DEFLATE and >