Re: Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-08 Thread Laurent Rineau
Le Friday 04 April 2014 16:15:59 Mikolaj Izdebski a écrit : > CPU: Haswell B0, Genuine Intel(R) CPU @ 2.20GHz > bogomips: 4389.60 > Processors:56 > NUMA Nodes:2 > Memory:31966 MB It would be fair to post also a bench that ran on a more usual machine, like quad-core

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-08 Thread Zbigniew Jędrzejewski-Szmek
On Fri, Apr 04, 2014 at 12:49:25PM -0400, Matthew Miller wrote: > On Fri, Apr 04, 2014 at 04:15:59PM +0200, Mikolaj Izdebski wrote: > > "lbzip2" was the fastest compressor and decompressor in all tests. > > It the best command for interactive use. > > > > "lbzip2 -u" always produced smallest files

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-08 Thread Zbigniew Jędrzejewski-Szmek
On Fri, Apr 04, 2014 at 04:15:59PM +0200, Mikolaj Izdebski wrote: > As I promised, I prepared a benchmark of lbzip2 and bzip2. > Decompression of linux-3.12.6.tar.bz2 > - > > command| real | user | sys | memory > ---+++--+---

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-04 Thread Mikolaj Izdebski
On 04/04/2014 07:01 PM, Susi Lehtola wrote: > On Fri, 4 Apr 2014 12:49:25 -0400 > Matthew Miller wrote: >> On Fri, Apr 04, 2014 at 04:15:59PM +0200, Mikolaj Izdebski wrote: >>> "lbzip2 -u" always produced smallest files (even smaller than bzip2) >>> while consuming the least amount of resources (C

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-04 Thread Susi Lehtola
On Fri, 4 Apr 2014 12:49:25 -0400 Matthew Miller wrote: > On Fri, Apr 04, 2014 at 04:15:59PM +0200, Mikolaj Izdebski wrote: > > "lbzip2 -u" always produced smallest files (even smaller than bzip2) > > while consuming the least amount of resources (CPU power and memory). > > This directly translate

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-04 Thread Matthew Miller
On Fri, Apr 04, 2014 at 04:15:59PM +0200, Mikolaj Izdebski wrote: > "lbzip2" was the fastest compressor and decompressor in all tests. > It the best command for interactive use. > > "lbzip2 -u" always produced smallest files (even smaller than bzip2) > while consuming the least amount of resources

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-04 Thread Mikolaj Izdebski
On 04/04/2014 05:26 PM, Mikolaj Izdebski wrote: > On 04/04/2014 05:16 PM, Michal Schmidt wrote: >> On 04/04/2014 04:15 PM, Mikolaj Izdebski wrote: >>> Compression of payload.tar >>> -- >>> >>> command| real | user | sys | memory | compr. size >>> ---+---

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-04 Thread Mikolaj Izdebski
On 04/04/2014 05:16 PM, Michal Schmidt wrote: > On 04/04/2014 04:15 PM, Mikolaj Izdebski wrote: >> Compression of payload.tar >> -- >> >> command| real | user | sys | memory | compr. size >> ---+++--++ >> lbzip2 |

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-04 Thread Michal Schmidt
On 04/04/2014 04:15 PM, Mikolaj Izdebski wrote: > Compression of payload.tar > -- > > command| real | user | sys | memory | compr. size > ---+++--++ > lbzip2 | 3.36 | 170.07 | 6.38 | 380448 | 424676188 > lbzip2

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-04 Thread Mikolaj Izdebski
On 04/02/2014 08:03 PM, Bill Nottingham wrote: > A quick check shows lbzip2 doesn't provide a library interface, much less > one compatible with libbz2. Is that ever intended? > > If it's not, saying lbzip2 is the default bzip2 *implementation* may be a > bit of a stretch. Perhaps s/implementation

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-04 Thread Mikolaj Izdebski
As I promised, I prepared a benchmark of lbzip2 and bzip2. I also added pbzip2 for comparison. Basic information = Test date: 2014-04-04 Tester:Mikolaj Izdebski Test subjects: lbzip2 2.5 bzip2 1.0.6 pbzip2 1.1.6 Test purpose: compare per

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-03 Thread Mikolaj Izdebski
On 04/03/2014 06:08 PM, Miloslav Trmač wrote: > Looking at http://lbzip2.org/news , lbzip2 is still fixing crashes during > compression and decompression. That's rather troubling: we need the bzip2 > implementation to be roughly as stable as file system*.* They say that every non-trivial piece of

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-03 Thread Miloslav Trmač
2014-04-02 19:24 GMT+02:00 Jaroslav Reznik : > = Proposed System Wide Change: lbzip2 as default bzip2 implementation = > https://fedoraproject.org/wiki/Changes/lbzip2 > While the speedup is desirable, it's not really obvious that this is the right time to do the change. Looking at http://lbzip2

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-03 Thread Ville Skyttä
On Thu, Apr 3, 2014 at 1:03 PM, Mikolaj Izdebski wrote: > rpm does use bzip2 *command* To be more precise, I believe only rpmbuild does. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/cod

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-03 Thread Mikolaj Izdebski
On 04/03/2014 03:47 AM, Chris Adams wrote: > Many of the common users (such as rpm) are linked > against the library and don't use the command, so they won't be > impacted. rpm does use bzip2 *command* and it would be impacted by this change. rpm uses libbz2 only for compression and decompression

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-03 Thread Ville Skyttä
On Wed, Apr 2, 2014 at 11:27 PM, Zbigniew Jędrzejewski-Szmek wrote: >> ** possibly adjust spec files to require or build-require lbzip2 instead of >> bzip2. > Is this necessary? I don't think so. A better way would be to change them to depend on the actual executables they use, /usr/bin/bzip2 etc

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Toshio Kuratomi
On Thu, Apr 03, 2014 at 04:48:03AM +0200, Zbigniew Jędrzejewski-Szmek wrote: > On Wed, Apr 02, 2014 at 07:26:59PM -0700, Toshio Kuratomi wrote: > > On Wed, Apr 02, 2014 at 08:47:11PM -0500, Chris Adams wrote: > > > > > > I think the "right" way to move forward is to make a library that is at > > >

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Zbigniew Jędrzejewski-Szmek
On Wed, Apr 02, 2014 at 07:26:59PM -0700, Toshio Kuratomi wrote: > On Wed, Apr 02, 2014 at 08:47:11PM -0500, Chris Adams wrote: > > > > I think the "right" way to move forward is to make a library that is at > > least API-compatible with the current libbz2.so.1, make all the tools > > use it, and

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Toshio Kuratomi
On Wed, Apr 02, 2014 at 08:47:11PM -0500, Chris Adams wrote: > > I think the "right" way to move forward is to make a library that is at > least API-compatible with the current libbz2.so.1, make all the tools > use it, and just replace bzip2 with lbzip2. > Although I'm still on the fence about whe

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Chris Adams
Once upon a time, drago01 said: > Well the change says " multi-threaded operation for both compression > and decompression, with almost linear scalability" linear scalability > means speed ups on the range of 2-8x on current desktop / laptop > systems. > Which I'd call a "significant gain" ;) Tha

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Mikolaj Izdebski
> This clarification is significant. The change proposal text needs to > be updated to reflect this. I will add a clarification tomorrow. > > As long as the encoding is guaranteed to be byte-for-byte identical to > that produced by the original bzip2 (and libbz2) implementation, the > risks ar

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Al Dunsmuir
On Wednesday, April 2, 2014, 2:03:38 PM, Bill Nottingham wrote: > Jaroslav Reznik (jrez...@redhat.com) said: >> = Proposed System Wide Change: lbzip2 as default bzip2 implementation = >> https://fedoraproject.org/wiki/Changes/lbzip2 >> >> Change owner(s): Mikolaj Izdebski >> >> This change aims

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Mikolaj Izdebski
> On Wed, Apr 2, 2014 at 11:39 PM, Chris Adams wrote: > > Once upon a time, Mikolaj Izdebski said: > >> lbzip2 does not (at least not yet) provide interfaces of libbz2 library, > >> only command line tools. This Change does *not* affect users of libbz2. > > > > Is there enough of a gain to the s

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Chris Adams
Once upon a time, Mikolaj Izdebski said: > lbzip2 does not (at least not yet) provide interfaces of libbz2 library, > only command line tools. This Change does *not* affect users of libbz2. Is there enough of a gain to the system to only partially replace a core program like this (especially wit

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread drago01
On Wed, Apr 2, 2014 at 11:39 PM, Chris Adams wrote: > Once upon a time, Mikolaj Izdebski said: >> lbzip2 does not (at least not yet) provide interfaces of libbz2 library, >> only command line tools. This Change does *not* affect users of libbz2. > > Is there enough of a gain to the system to onl

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Mikolaj Izdebski
> This implementation has been built and extensively tested using the > current release of the real bzip2 library. Substituting a completely > different library implementation without going through extensive and > explicit validating and testing is risky and unreasonable. At best, it > would

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Al Dunsmuir
Hello Al, On Wednesday, April 2, 2014, 5:14:53 PM, Al Dunsmuir wrote: > On Wednesday, April 2, 2014, 4:27:55 PM, Zbigniew Jędrzejewski-Szmek wrote: >>> ** possibly adjust spec files to require or build-require lbzip2 instead of >>> bzip2. >> Is this necessary? Wouldn't it be better to have lbzip2

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Matthew Garrett
On Wed, Apr 02, 2014 at 05:14:53PM -0400, Al Dunsmuir wrote: > This implementation has been built and extensively tested using the > current release of the real bzip2 library. Substituting a completely > different library implementation without going through extensive and > explicit validat

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Mikolaj Izdebski
> On 2 April 2014 21:46, drago01 wrote: > > On Wed, Apr 2, 2014 at 10:27 PM, Zbigniew Jędrzejewski-Szmek > > wrote: > >>> ** possibly adjust spec files to require or build-require lbzip2 instead > >>> of > >>> bzip2. > >> Is this necessary? Wouldn't it be better to have lbzip2 Provide bzip2 > >>

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Mikolaj Izdebski
> > ** possibly adjust spec files to require or build-require lbzip2 instead of > > bzip2. > Is this necessary? Wouldn't it be better to have lbzip2 Provide bzip2 > or something so that updating all those packages is not necessary, > and also that people who prefer normal bzip2 can still use it? Y

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Al Dunsmuir
On Wednesday, April 2, 2014, 4:27:55 PM, Zbigniew Jędrzejewski-Szmek wrote: >> ** possibly adjust spec files to require or build-require lbzip2 instead of >> bzip2. > Is this necessary? Wouldn't it be better to have lbzip2 Provide bzip2 > or something so that updating all those packages is not n

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Ian Malone
On 2 April 2014 21:46, drago01 wrote: > On Wed, Apr 2, 2014 at 10:27 PM, Zbigniew Jędrzejewski-Szmek > wrote: >>> ** possibly adjust spec files to require or build-require lbzip2 instead of >>> bzip2. >> Is this necessary? Wouldn't it be better to have lbzip2 Provide bzip2 >> or something so that

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread drago01
On Wed, Apr 2, 2014 at 10:27 PM, Zbigniew Jędrzejewski-Szmek wrote: >> ** possibly adjust spec files to require or build-require lbzip2 instead of >> bzip2. > Is this necessary? Wouldn't it be better to have lbzip2 Provide bzip2 > or something so that updating all those packages is not necessary,

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Zbigniew Jędrzejewski-Szmek
> ** possibly adjust spec files to require or build-require lbzip2 instead of > bzip2. Is this necessary? Wouldn't it be better to have lbzip2 Provide bzip2 or something so that updating all those packages is not necessary, and also that people who prefer normal bzip2 can still use it? Zbyszek -

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Zbigniew Jędrzejewski-Szmek
On Wed, Apr 02, 2014 at 03:10:23PM -0400, Mikolaj Izdebski wrote: > > On 04/02/2014 11:33 AM, Reindl Harald wrote: > > > packages using the library (output of yum remove > > > bzip2-libs-1.0.6-9.fc20.x86_64) > > > > Try: repoquery --whatrequires 'libbz2.so.1()(64bit)' > > > > We'll certainly need

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Mikolaj Izdebski
> On 04/02/2014 11:33 AM, Reindl Harald wrote: > > packages using the library (output of yum remove > > bzip2-libs-1.0.6-9.fc20.x86_64) > > Try: repoquery --whatrequires 'libbz2.so.1()(64bit)' > > We'll certainly need to keep the library around. Yes, but we can have /usr/bin/bzip2 (and other com

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Josh Stone
On 04/02/2014 11:33 AM, Reindl Harald wrote: > packages using the library (output of yum remove > bzip2-libs-1.0.6-9.fc20.x86_64) Try: repoquery --whatrequires 'libbz2.so.1()(64bit)' We'll certainly need to keep the library around. -- devel mailing list devel@lists.fedoraproject.org https://adm

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Tom Hughes
On 02/04/14 19:22, Mikolaj Izdebski wrote: lbzip2 creates only *one* stream per compressed file, even when using multiple threads. Such files can be decompressed with all versions of bzip2, libbz2 and other tools, such as Apache Commons Compress. This is a difference between lbzip2 and pbzip2,

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Reindl Harald
Am 02.04.2014 20:18, schrieb Mikolaj Izdebski: >>> lbzip2 is a mature project and it has been used in production for years. It >>> is >>> already packaged for Fedora and it is also available in EPEL. >> >> A quick check shows lbzip2 doesn't provide a library interface, much less >> one compatib

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Mikolaj Izdebski
On 04/02/2014 08:10 PM, Tom Hughes wrote: > On 02/04/14 18:24, Jaroslav Reznik wrote: > >> == Detailed Description == >> lbzip2 is an independent implementation of bzip2 compression tool. It >> provides >> interface strictly compatible with bzip2, but also adds several new >> features >> and impro

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Mikolaj Izdebski
On 04/02/2014 08:03 PM, Bill Nottingham wrote: > Jaroslav Reznik (jrez...@redhat.com) said: >> = Proposed System Wide Change: lbzip2 as default bzip2 implementation = >> https://fedoraproject.org/wiki/Changes/lbzip2 >> >> Change owner(s): Mikolaj Izdebski >> >> This change aims at making lbzip2

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Tom Hughes
On 02/04/14 18:24, Jaroslav Reznik wrote: == Detailed Description == lbzip2 is an independent implementation of bzip2 compression tool. It provides interface strictly compatible with bzip2, but also adds several new features and improvements, such as: * multi-threaded operation for both compres

Re: F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Bill Nottingham
Jaroslav Reznik (jrez...@redhat.com) said: > = Proposed System Wide Change: lbzip2 as default bzip2 implementation = > https://fedoraproject.org/wiki/Changes/lbzip2 > > Change owner(s): Mikolaj Izdebski > > This change aims at making lbzip2 [1] default bzip2 implementation used in > Fedora.

F21 System Wide Change: lbzip2 as default bzip2 implementation

2014-04-02 Thread Jaroslav Reznik
= Proposed System Wide Change: lbzip2 as default bzip2 implementation = https://fedoraproject.org/wiki/Changes/lbzip2 Change owner(s): Mikolaj Izdebski This change aims at making lbzip2 [1] default bzip2 implementation used in Fedora. == Detailed Description == lbzip2 is an independent imple