Re: dist-xz compression level
* Andreas Jellinghaus wrote on Mon, Apr 12, 2010 at 09:41:44PM CEST: > Am Montag 12 April 2010 20:31:22 schrieb Ralf Wildenhues: > > My idea would be to implement this for master, and revert the recent > > xz -9 patch on branch-1.11. Comments, criticism? > > if the environment variables can be used to set the compression, > that is perfect for me. why write your own code, when the environment > will be accepted automatically? Because you cannot easily set an environment variable in the Makefile.am without another 'make' indirection, at least not without resorting to GNU make features. 'make dist' is fairly well-known, a package-specific mydist rule or so would be decidedly worse in terms of ease of use. > but some documentation would help users I think. Well, yes, amending automake.texi was an implicit part of my proposal. > (also some man pages don't document these environment options, > at least in the versions I have, but strings indicate they exist...) Are you speaking about Automake or the compressors? I found each of those variables in a manpage. Cheers, Ralf
Re: dist-xz compression level
Am Montag 12 April 2010 20:31:22 schrieb Ralf Wildenhues: > My idea would be to implement this for master, and revert the recent > xz -9 patch on branch-1.11. Comments, criticism? if the environment variables can be used to set the compression, that is perfect for me. why write your own code, when the environment will be accepted automatically? but some documentation would help users I think. (also some man pages don't document these environment options, at least in the versions I have, but strings indicate they exist...) Regards, Andreas
Re: dist-xz compression level
Am Sonntag 11 April 2010 21:37:13 schrieb Andreas Jellinghaus: > isn't xz extremely slw with -9? > maybe it wasn't a bug, bit intentionally not used, > as that huge extra amount of time doesn't result in > that many bytes saved. > > is the compression level configureable somehow? I did two tests with medium size sources and xz -6 vs. xz -9 is not a big difference in speed or compression size. so please ignore my comment. Regards, Andreas
Re: dist-xz compression level
Hello, * Jim Meyering wrote on Mon, Apr 12, 2010 at 11:44:01AM CEST: > Pavel Sanda wrote: > >> For my use, xz -9 is far too slow for anything except the > >> final "make dist" I run just prior to a release. > >> > >> For a release, I run this, via one of the > >> alpha, beta or stable targets in gnulib's maint.mk: > >> > >> $(MAKE) dist XZ_OPT=-9ev > > > > Is it possible to setup this to be the default inside makefiles? > > (I use make dist only for the final release...) > > Sure. > Create a GNUmakefile containing > > export XZ_OPT=-9ev > > and just passing all other rules to the default Makefile. This is fine for projects where the developers use GNU make anyway. Automake strives for more portable solutions where possible. I looked at a couple of manpages now. gzip, xz, bzip2, and zip allow optional arguments through environment variables GZIP, XZ_OPT, BZIP2, and ZIPOPT, respectively. compress can safely be ignored, and lzip improved I guess. AFAICS all give the environment variables lower priority than command-line arguments. Automake rules currently use GZIP by setting GZIP=$(GZIP_ENV) before calling gzip, but the other compression rules do not do anything similar. We could thus do the same for the other compressors. This would break backward compatibility and require adjusting maint.mk. An alternative would be something like if test -n "$(GZIP_ENV)"; then GZIP="$(GZIP_ENV)"; export GZIP; fi; \ ... which would be more backward-compatible. Next question is which variable names we should use. XZ_ENV BZIP2_ENV ZIP_ENV would be consistent. My idea would be to implement this for master, and revert the recent xz -9 patch on branch-1.11. Comments, criticism? Thanks, Ralf
Re: dist-xz compression level
On Mon, 12 Apr 2010, Pavel Sanda wrote: Pavel Sanda wrote: It was deliberate. For my use, xz -9 is far too slow for anything except the final "make dist" I run just prior to a release. For a release, I run this, via one of the alpha, beta or stable targets in gnulib's maint.mk: $(MAKE) dist XZ_OPT=-9ev Is it possible to setup this to be the default inside makefiles? (I use make dist only for the final release...) Some of us use 'make distcheck' regularly as a test case, and for periodic development snapshots. Since Automake passes through makefile content, it is easy enough to add a project-specific 'make release' target which adds the XZ_OPT option and invokes 'make dist' or 'make distcheck'. Overriding an already set makefile variable is likely not portable. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: dist-xz compression level
On 11 April 2010 23:37, Bob Friesenhahn wrote: > Yes, compression is useful. However, the cost of pushing the algorithm > close to the limit does incur costs as well. For many packages, getting 99% > of the max in 1/2 the time is a worthy tradeoff. This is similar to the > decision to use -O2 as the default GCC compiler optimization rather than > -O3. -O2 vs -O3 is a rather different case, as it's not a straightforward time-space tradeoff. The last time I checked, -O3 still had significant bugs, so it was only worth using for time-critical code where it was worth the testing required to show that they hadn't been triggered. Assuming that has been fixed, then there's the problem that -O3 often produces code that takes fewer clock cycles to execute, but is bigger than -O2's code, so in fact it is still only worth using for critical regions. Indeed, when Apple switched to Intel, they went one further, as their system profiling showed that it was better to use -O2 only for the kernel and system libraries, and to use -Os for everything else (i.e. optimize for space), because for application code cache impact was more important than raw speed. I can find little on research for GNU/Linux systems, except a supporting instance where someone tried -O2 vs -Os for the kernel and found -O2 to be a bit better. -- http://rrt.sc3d.org
Re: dist-xz compression level
Pavel Sanda wrote: >> Pavel Sanda wrote: >> It was deliberate. >> For my use, xz -9 is far too slow for anything except the >> final "make dist" I run just prior to a release. >> >> For a release, I run this, via one of the >> alpha, beta or stable targets in gnulib's maint.mk: >> >> $(MAKE) dist XZ_OPT=-9ev > > Is it possible to setup this to be the default inside makefiles? > (I use make dist only for the final release...) Sure. Create a GNUmakefile containing export XZ_OPT=-9ev and just passing all other rules to the default Makefile.
Re: dist-xz compression level
> Pavel Sanda wrote: > It was deliberate. > For my use, xz -9 is far too slow for anything except the > final "make dist" I run just prior to a release. > > For a release, I run this, via one of the > alpha, beta or stable targets in gnulib's maint.mk: > > $(MAKE) dist XZ_OPT=-9ev Is it possible to setup this to be the default inside makefiles? (I use make dist only for the final release...) Pavel
Re: dist-xz compression level
Pavel Sanda wrote: > the newly added dist-xz target produce worse compressed archives > than lzma-dist. The reason is that automake call lzma with > best compression while it won't use -9 level for xz. > Is this intention or bug? It was deliberate. For my use, xz -9 is far too slow for anything except the final "make dist" I run just prior to a release. For a release, I run this, via one of the alpha, beta or stable targets in gnulib's maint.mk: $(MAKE) dist XZ_OPT=-9ev
Re: dist-xz compression level
>> The same can be said about currently used -9 for lzma, no? > > Yes. The argument is that it should be possible to optionally set the > compression level. In most cases, the compression default should be the > tool's compression default. I have no problem with such solution. Pavel
Re: dist-xz compression level
On Sun, 11 Apr 2010, Pavel Sanda wrote: Are you assuming 'make dist' after 'make' or 'make dist' from scratch? Other than the time spent compressing data, 'make dist' after 'make' should be quite fast. Yep, I mean the make dist from the scratch; i.e. what one usually does when creating new release. The compression is used very rarely compared to the number of tarballs uncompression, disk usage multiplied by all the mirros, net usage etc... Yes, compression is useful. However, the cost of pushing the algorithm close to the limit does incur costs as well. For many packages, getting 99% of the max in 1/2 the time is a worthy tradeoff. This is similar to the decision to use -O2 as the default GCC compiler optimization rather than -O3. It may be that dimished returns become extreme as the compression level reaches -9. The same can be said about currently used -9 for lzma, no? Yes. The argument is that it should be possible to optionally set the compression level. In most cases, the compression default should be the tool's compression default. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: dist-xz compression level
> Are you assuming 'make dist' after 'make' or 'make dist' from scratch? > Other than the time spent compressing data, 'make dist' after 'make' > should be quite fast. Yep, I mean the make dist from the scratch; i.e. what one usually does when creating new release. The compression is used very rarely compared to the number of tarballs uncompression, disk usage multiplied by all the mirros, net usage etc... > It may be that dimished returns become extreme as the compression level > reaches -9. The same can be said about currently used -9 for lzma, no? Pavel
Re: dist-xz compression level
On Sun, 11 Apr 2010, Pavel Sanda wrote: isn't xz extremely slw with -9? maybe it wasn't a bug, bit intentionally not used, as that huge extra amount of time doesn't result in that many bytes saved. Compared to the total time of make dist its IMHO acceptable. But configurability won't hurt of course. Are you assuming 'make dist' after 'make' or 'make dist' from scratch? Other than the time spent compressing data, 'make dist' after 'make' should be quite fast. It may be that dimished returns become extreme as the compression level reaches -9. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: dist-xz compression level
> Well, does somebody have numbers (memory, time, compression) as to what > is reasonable? I didn't make any testing, but the report came from the observation that result was +300kb on 9 mb. The compression was slow, but decompression is not affected. pavel
Re: dist-xz compression level
> isn't xz extremely slw with -9? > maybe it wasn't a bug, bit intentionally not used, > as that huge extra amount of time doesn't result in > that many bytes saved. Compared to the total time of make dist its IMHO acceptable. But configurability won't hurt of course. Pavel
Re: dist-xz compression level
Hello Andreas, * Andreas Jellinghaus wrote on Sun, Apr 11, 2010 at 09:37:13PM CEST: > isn't xz extremely slw with -9? > maybe it wasn't a bug, bit intentionally not used, > as that huge extra amount of time doesn't result in > that many bytes saved. Well, does somebody have numbers (memory, time, compression) as to what is reasonable? > is the compression level configureable somehow? Not ATM, but if necessary we could change that. Thanks, Ralf
Re: dist-xz compression level
isn't xz extremely slw with -9? maybe it wasn't a bug, bit intentionally not used, as that huge extra amount of time doesn't result in that many bytes saved. is the compression level configureable somehow? Regards, Andreas
Re: dist-xz compression level
Hello Pavel, * Pavel Sanda wrote on Wed, Apr 07, 2010 at 01:22:06PM CEST: > the newly added dist-xz target produce worse compressed archives > than lzma-dist. The reason is that automake call lzma with > best compression while it won't use -9 level for xz. > Is this intention or bug? Bug, I guess. I'm applying this patch to maint and merging it to branch-1.11 and master, and adding you to THANKS. Thanks for the report, Ralf Use -9 for maximum xz compression with dist-xz. * lib/am/distdir.am (dist-xz, dist, dist-all): Pass -9 to xz. * NEWS, THANKS: Update. Report by Pavel Sanda. diff --git a/NEWS b/NEWS index a3ce191..5af2439 100644 --- a/NEWS +++ b/NEWS @@ -12,6 +12,8 @@ Bugs fixed in 1.11.0a: - The `parallel-tests' test driver works around a GNU make 3.80 bug with trailing white space in the test list (`TESTS = foo $(EMPTY)'). + - The `dist-xz' option now uses `xz -9' for maximum compression. + * Long standing bugs: - On Darwin 9, `pythondir' and `pyexecdir' pointed below `/Library/Python' diff --git a/lib/am/distdir.am b/lib/am/distdir.am index ec4d5e5..d88656c 100644 --- a/lib/am/distdir.am +++ b/lib/am/distdir.am @@ -1,6 +1,6 @@ ## automake - create Makefile.in from Makefile.am -## Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 -## Free Software Foundation, Inc. +## Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, +## 2010 Free Software Foundation, Inc. ## This program is free software; you can redistribute it and/or modify ## it under the terms of the GNU General Public License as published by @@ -353,7 +353,7 @@ dist-lzma: distdir ?XZ?DIST_ARCHIVES += $(distdir).tar.xz .PHONY: dist-xz dist-xz: distdir - tardir=$(distdir) && $(am__tar) | xz -c >$(distdir).tar.xz + tardir=$(distdir) && $(am__tar) | xz -9 -c >$(distdir).tar.xz $(am__remove_distdir) ?COMPRESS?DIST_ARCHIVES += $(distdir).tar.Z @@ -397,7 +397,7 @@ dist dist-all: distdir ?GZIP? tardir=$(distdir) && $(am__tar) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).tar.gz ?BZIP2?tardir=$(distdir) && $(am__tar) | bzip2 -9 -c >$(distdir).tar.bz2 ?LZMA? tardir=$(distdir) && $(am__tar) | lzma -9 -c >$(distdir).tar.lzma -?XZ? tardir=$(distdir) && $(am__tar) | xz -c >$(distdir).tar.xz +?XZ? tardir=$(distdir) && $(am__tar) | xz -9 -c >$(distdir).tar.xz ?COMPRESS? tardir=$(distdir) && $(am__tar) | compress -c >$(distdir).tar.Z ?SHAR? shar $(distdir) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).shar.gz ?ZIP? -rm -f $(distdir).zip
dist-xz compression level
Hi, the newly added dist-xz target produce worse compressed archives than lzma-dist. The reason is that automake call lzma with best compression while it won't use -9 level for xz. Is this intention or bug? Pavel
