Dear Laszlo,
On Jan 28 2010, Ersek, Laszlo wrote:
> On Wed, 27 Jan 2010, Rogério Brito wrote:
> >I have been using lbzip2 and being very happy with it.
>
> Thanks :)
Thank you for writing it in the first place. I think that I will read
the source code so that I can abstract its operation and pass on the
knowledge.
> >,----
> >| /tmp$ xz -9ev debian-testing-powerpc-businesscard.iso
> >| debian-testing-powerpc-businesscard.iso (1/1)
> >| ^C 44.1 % 14.5 MiB / 28.7 MiB = 0.507 866 KiB/s
> > 0:33
> >|
> >| /tmp$
> >`----
>
> Please consider using the great cstream or pv tools in a pipeline
> with lbzip2. (The Debian packages have the same names as the
> binaries.) For example:
Thank you very much for the hint of those pipe viewers (always nice to
have them in the toolkit).
> If you find this answer unsatisfactory, please tell me. In that
> case, I'll try to think about how to implement the feature *cleanly*
> within lbzip2. (Please don't take this as a promise, though.)
I experimented a little bit with lbzip2 and pv and it works fine. So, I
guess that there are no problems with the delegation that task to
another tool.
> Finally, I can't help but recommend a multi-threaded LZMA compressor
> to you: you may want to check out plzip [0] [1] [2], originally
> based on the lzlib-0.7 compression library [3] by Antonio Diaz Diaz,
> and the compressor module of lbzip2-0.17 by yours truly.
Ah, thank you for that. I also didn't know about it. I was here waiting
for a multi-threaded version of xz to be implemented. I just took the
package created by Daniel Baumann (still sitting in NEW) and compiled it
here.
> After the initial bare-bones, featureless implementation by me
> (modestly called llzip at that time), plzip continues to be developed
> / maintained by Antonio Diaz Diaz exclusively. He just released 0.3 a
> few days ago [4].
That's the version that I'm using here. With it, the pv trick doesn't
work so well: it seems that plzip works here by outputting large chunks
of data at a time and having many large-spaced periods without any data
output, which means that, for a long time, the user doesn't see the
progress of the compression (well, I suspect that this happens when the
user has "big" amounts of RAM---in my case, I have an amd64 system with
2GB of RAM).
> Comparing, on my dual core,
(...)
> (Needless to say, this was a completely non-tuned "benchmark".)
Thanks. Here is the data that I got, in a completely non-scientific,
non-tuned benchmark, with "stock" software (sorry for the long lines):
,----
| /tmp$ lbzip2 -9 < debian-testing-i386-netinst.iso - | pv -c >
debian-testing-i386-netinst.iso.bz2
| 158MB 0:01:11 [ 2.2MB/s] [ <=>
]
| /tmp$ plzip -9 < debian-testing-i386-netinst.iso - | pv -c >
debian-testing-i386-netinst.iso.lz
| 156MB 0:05:14 [ 506kB/s] [ <=>
]
| /tmp$ xz -9e < debian-testing-i386-netinst.iso - | pv -c >
debian-testing-i386-netinst.iso.xz
| 155MB 0:03:56 [ 669kB/s] [ <=>
]
`----
Here, plzip uses both of my cores (a Pentium D 805, running sid/amd64),
and xz uses only one. Despite that, for that particular file (taken from
the daily builds of debian-installer, from 2010-01-27), xz offers a
smaller file in less time.
Is there anything that I may be doing wrong? (Well, I guess that this is
becoming off-topic as a message meant as a feature request of lbzip2). :-)
Here is the environment:
,----
| /tmp$ xz --version
| xz (XZ Utils) 4.999.9beta
| liblzma 4.999.9beta
| /tmp$ lbzip2 --version
| lbzip2: Parallel bzip2 utility.
| Copyright (C) 2008, 2009 Laszlo Ersek. Released under the GNU GPLv2+.
| Version 0.20.
| (...)
| /tmp$ plzip --version
| Plzip 0.3
| Copyright (C) 2009 Laszlo Ersek.
| Copyright (C) 2010 Antonio Diaz Diaz.
| License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
| /tmp$ ls -log debian-testing-i386-netinst.iso*
| -rw-r--r-- 1 169531392 Jan 27 01:47 debian-testing-i386-netinst.iso
| -rw-r--r-- 1 165869114 Jan 29 02:50 debian-testing-i386-netinst.iso.bz2
| -rw-r--r-- 1 163139186 Jan 29 02:56 debian-testing-i386-netinst.iso.lz
| -rw-r--r-- 1 162206084 Jan 29 03:00 debian-testing-i386-netinst.iso.xz
| /tmp$ dpkg-query -W lbzip2 xz-utils liblzma2 plzip
| lbzip2 0.20-1
| liblzma2 4.999.9beta+20100117-1
| plzip 0.3-1
| xz-utils 4.999.9beta+20100117-1
| /tmp$
`----
I'm including Daniel's ITP bug as a CC.
Thanks, Rogério Brito.
--
Rogério Brito : rbr...@{ime.usp.br,gmail.com} : GPG key 1024D/7C2CAEB8
http://rb.doesntexist.org : Packages for LaTeX : algorithms.berlios.de
DebianQA: http://qa.debian.org/developer.php?login=rbrito%40ime.usp.br
--
To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
