Re: Speeding up libtool
* Ralf Wildenhues wrote on Mon, Apr 04, 2005 at 09:06:25AM CEST: * Robert Ögren wrote on Mon, Apr 04, 2005 at 01:30:26AM CEST: With ash and lt_ECHO=printf...: I think the disappointing speedup is caused by a lot of spawning of printf (printf isn't a builtin in ash, right?) Setting ECHO=echo in config.status makes it take 5.7 seconds after the patch (8.2 before). ash's `echo' is bad for libtool, as it interprets backslashes by default. The ash I use on linux has printf builtin. I looked on cygwin.com, they seem to ship an older ash variant without printf builtin. You may want to try a newer one or even try to get them to include it in cygwin proper. I checked a cygwin installation. The /bin/sh had no builtin printf, but its builtin echo does _not_ interpret backslashes, so it's actually usable for libtool. CONFIG_SHELL=/bin/sh /bin/sh configure should thus work. Sorry for not checking earlier. Meaning 40% speedup for config.status on cygwin. :) I get about 30% on linux/x86. OK to apply to HEAD? Regards, Ralf * m4/libtool.m4 (_LT_CONFIG_COMMANDS): Only call sed if necessary while quoting all libtool variables. Reported by Robert Ögren [EMAIL PROTECTED]. Index: m4/libtool.m4 === RCS file: /cvsroot/libtool/libtool/m4/libtool.m4,v retrieving revision 1.177 diff -u -r1.177 libtool.m4 --- m4/libtool.m4 28 Mar 2005 09:32:59 - 1.177 +++ m4/libtool.m4 28 Mar 2005 16:00:34 - @@ -462,12 +462,26 @@ # Quote evaled strings. for var in lt_decl_all_varnames([[ ]], lt_decl_quote_varnames); do -eval lt_\$var=\`\\\$ECHO \\X\\\$\$var\\ | \\\$Xsed -e \$sed_quote_subst\`\\ +case \`eval \$ECHO X\$\$var\` in +*[[\\\`\$]]*) + eval lt_\$var=\`\\\$ECHO \\X\\\$\$var\\ | \\\$Xsed -e \$sed_quote_subst\`\\ + ;; +*) + eval lt_\$var=\$\$var\\ + ;; +esac done # Double-quote double-evaled strings. for var in lt_decl_all_varnames([[ ]], lt_decl_dquote_varnames); do -eval lt_\$var=\`\\\$ECHO \\X\\\$\$var\\ | \\\$Xsed -e \$double_quote_subst\\ -e \$sed_quote_subst\\ -e \$delay_variable_subst\`\\ +case \`eval \$ECHO X\$\$var\` in +*[[\\\`\$]]*) + eval lt_\$var=\`\\\$ECHO \\X\\\$\$var\\ | \\\$Xsed -e \$double_quote_subst\\ -e \$sed_quote_subst\\ -e \$delay_variable_subst\`\\ + ;; +*) + eval lt_\$var=\$\$var\\ + ;; +esac done # Fix-up fallback echo if it was mangled by the above quoting rules.
Re: Speeding up libtool
Hi Ralf, Ralf Wildenhues wrote: Hi Robert, First I'd like to say many thanks for all your testing and valuable feedback. Thank you for the responses. Will take a little while to work through all of it. Hehe, sorry, I had too much time on my hands last weekend and largely spent it on fiddling with Libtool. ... What it basically does is to build GLib 2.6.3 using different versions of libtool and with and without caching. As I said before, I'm usually using Cygwin but targetting MinGW (using gcc -mno-cygwin) to get native win32 binaries, and pretending the host is also mingw to avoid having to set up cross-compilation stuff. This works very well with one minor change in ltmain.sh: Replace $LTCC $LTCFLAGS with /usr/bin/gcc at the spot where it builds the wrapper exes from c source. (I don't expect you to change this in Libtool, I'm just explaining how I use it. Maybe there is even a better way...) It'd be interesting to know /why/ you have to change it, and what the value of $LTCC and $LTCFLAGS are. LTCC=gcc -mno-cygwin LTCFLAGS=-O1 -g -march=pentium -pipe -mms-bitfields -Wall The main reason I had to change it is that execv(/bin/sh,...) does not make sense in a native Windows program, which is what you get if the wrapper programs are compiled with gcc -mno-cygwin and a Cygwin environment is used. The Windows path to Cygwin's sh is something like C:\cygwin\bin\sh.exe, but this depends on where Cygwin is installed. Using plain cygwin gcc for compiling the wrappers solves this problem. There may be better solutions, I stopped searching when I found a simple workaround that worked for me. I don't know how this works if MinGW and MSYS is used - I assume it must work somehow, otherwise people surely would have complained? I personally don't use MSYS, I find Cygwin much more complete and consistent, but that's just my personal opinion. Don't spend time on it if noone else complains. ... The increase in time for the first compilation with Libtool HEAD is caused by a large amount of sed calls, mainly by the Quote evaled strings stuff from _LT_CONFIG_COMMANDS that is placed in config.status, since config.status is invoked a few extra times on win32 to create glib/glib.rc, gobject/gobject.rc, gmodule/gmodule.rc and gthread/gthread.rc. Each config.status invocation cost about 10 seconds. This is a minor thing, and could probably be fixed by adding them to AC_CONFIG_FILES in configure.in with some win32 conditional stuff. Hmm. It's possible to do a similar trick here as in general.m4sh, it's just not clear how much it will gain. And whether some old shell starts screaming again. Something along the lines of (this will be in config.status) var_xvalue=`eval $ECHO X$var` case $var_xvalue in *[\\\`\\$]*) eval lt_\$var=..# sed invocation ;; *) eval lt_$var=\$$var ;; esac Attached is a lightly tested patch for this. The speedup of config.status execution would be interesting. For small projects this might be more pronounced. With ash and lt_ECHO=printf...: Without the patch: sh ./config.status glib/glib.rc takes approx 9.8 seconds With the patch: sh ./config.status glib/glib.rc takes approx 8.8 seconds I think the disappointing speedup is caused by a lot of spawning of printf (printf isn't a builtin in ash, right?) Setting ECHO=echo in config.status makes it take 5.7 seconds after the patch (8.2 before). .. 1. glib's configure.in defines a few variables like LT_RELEASE, and these names are forbidden with Libtool HEAD so I had to patch in a m4_pattern_allow for these Yes. I think it would be better if glib changed their names, at least in the long run. The autotools have more-or-less gone towards reserving their respective prefixes AC_, AT_, AS_, AM_, LT_ for the respective tools. It makes sense to forbid others, e.g. because that way typos can be diagnosed. For the time being, m4_pattern_allow is a decent workaround. Ok 2. configure.in did not have AC_LIBTOOL_WIN32_DLL which now is required for OBJDUMP to be set. However, even though I added AC_LIBTOOL_WIN32_DLL before AM_PROG_LIBTOOL, it did not work, but if I replaced the call to AM_PROG_LIBTOOL with LT_INIT([win32-dll]) it worked. I think this is a bug in Libtool = 2.0. I'm not sure, though. I sent a testcase in another mail. Looks like a bug to me. 3. (I had to modify libtool-cache so it removes the libtool: compile: stuff that libtool HEAD prefixes output lines with) Well, yeah, Libtool considered its output to be read by humans, not parsed by more than our own testsuite. It would, as already noted, require very cautious audit for things like quoting level. Agreed, I wasn't complaining :) OTOH, if we can decide on putting a caching mechanism into libtool proper, there might just be a simpler way than that. Sure, but doing it in portable shell could be a challenge. For example, some recent tests show that all the stuff that is done by AS_SHELL_SANITIZE seems to be responsible for around 0.75 s
Re: Speeding up libtool
Ralf Wildenhues wrote: libtool HEAD should tell you upon invocation of `autoconf -Wall' to remove AC_LIBTOOL_WIN32_DLL in favor of adding the win32-dll option to LT_INIT. After doing that, things should work. (But leaving things like they are should also work.) Unfortunately AC_LIBTOOL_WIN32_DLL didn't work for me with Libtool HEAD. I created the following ugly test case: $ cat configure.ac AC_INIT(hello.c) AM_INIT_AUTOMAKE(hello,0.1) AC_PROG_CC AC_LIBTOOL_WIN32_DLL AC_PROG_LIBTOOL AC_OUTPUT(Makefile) $ cat Makefile.am bin_PROGRAMS = hello hello_SOURCES = hello.c $ aclocal -I /home/robert/devel/libtool-HEAD/share/libtool-2.1a/m4/ autoconf automake $ grep enable_win32_dll configure $ grep OBJDUMP configure lt_cv_file_magic_cmd='$OBJDUMP -f' $ The win32-dll stuff is missing. If I replace AC_LIBTOOL_WIN32_DLL and AC_PROG_LIBTOOL with LT_INIT([win32-dll]), it works as expected: $ cat configure.ac AC_INIT(hello.c) AM_INIT_AUTOMAKE(hello,0.1) AC_PROG_CC LT_INIT([win32-dll]) AC_OUTPUT(Makefile) $ aclocal -I /home/robert/devel/libtool-HEAD/share/libtool-2.1a/m4/ autoconf automake $ grep enable_win32_dll configure enable_win32_dll=yes $ My M4-fu is weak, but to me it seems suspicious that _LT_SET_OPTIONS only dispatches to the handler macros for the options that are passed in as a parameter to _LT_SET_OPTIONS since AC_LIBTOOL_WIN32_DLL just sets the option without dispatching. With the following change to ltoptions.m4 my testcase works (produces the desired stuff in configure): AU_DEFUN([AC_LIBTOOL_WIN32_DLL], [_LT_SET_OPTION([win32-dll]) +m4_ifdef(_LT_MANGLE_DEFUN([win32-dll]), + _LT_MANGLE_DEFUN([win32-dll]), + [m4_fatal([Unknown option win32-dll])]) AC_DIAGNOSE([obsolete], [$0: Remove this warning and the call to _LT_SET_OPTION when you put the `win32-dll' option into LT_INIT's first parameter.]) If I just put in _LT_MANGLE_DEFUN([win32-dll]) instead of the m4_ifdef it does not work. I don't know enough M4 to explain why. This is with latest Autoconf, Automake and Libtool (1.1902) from CVS on a GNU/Linux x86_64 system. Robert ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Speeding up libtool
[ upon reading Robert's mail again ] * Tor Lillqvist wrote on Sun, Mar 27, 2005 at 03:00:14AM CEST: Robert Ögren writes: 2. configure.in did not have AC_LIBTOOL_WIN32_DLL which now is required for OBJDUMP to be set. Hmm. Wasn't AC_LIBTOOL_WIN32_DLL supposed to be deprecated and unnecessary? Or was it just that at some point it was de facto unnecessary but never officially deprecated, and now if using libtool HEAD it is again necessary ? libtool HEAD should tell you upon invocation of `autoconf -Wall' to remove AC_LIBTOOL_WIN32_DLL in favor of adding the win32-dll option to LT_INIT. After doing that, things should work. (But leaving things like they are should also work.) Regards, Ralf ___ http://lists.gnu.org/mailman/listinfo/libtool
Testsuite failures on Cygwin with ECHO=printf (was: Re: Speeding up libtool)
Ralf Wildenhues wrote:
[...]
Yes. We recently identified that an audit of libtool.m4 for wrong use
of $ECHO is necessary.
[details left out]
Another question: How did you find those bugs? I.e., were they exposed
by the Libtool testsuite? If not, how can we trigger them?
The testsuite triggers the EXPORTn bug and some other bugs if $ECHO is
set to 'printf %s\n' on Cygwin. When running the testsuite normally,
All 106 tests passed. (The new testsuite thingy fails 2 of 5 tests,
but I'll report that in a separate mail). When running the testsuite
like this:
make check VERBOSE=yes lt_ECHO='printf %s\n'
The result is
9 of 95 tests failed
(11 tests were not run)
I've uploaded the verbose log to
http://www.roboros.com/tmp/libtool-echo-errs.txt
Some highlights:
PASS: mdemo-static.test
PASS: mdemo-make.test
SKIP: mdemo-exec.test
SKIP: mdemo-inst.test
SKIP: mdemo-unst.test
This is interesting. Linking actually fails in mdemo-make due to syntax
error in the def file, but the test still passes because libtool returns
success. As far as I can see this is because the code tests $? after
func_show_eval $link_command, but func_show_eval evals ':' last if an
error occurs and only one argument is supplied. A change that appears to
fix this is to replace
my_fail_exp=${2-:}
by
my_fail_exp=${2-false}
in func_show_eval.
PASS: demo-conf.test
PASS: demo-make.test
PASS: demo-exec.test
PASS: demo-inst.test
FAIL: demo-unst.test
Uninstall fails:
libtool: uninstall: dldll=`/bin/bash 21 -c '.
/Robert/libtool-HEAD/src/libtool
/tests/_inst/lib/libhello.la; printf %s\n ../bin/cyghello-2.dll'`
libtool: uninstall:
dlpath=/Robert/libtool-HEAD/src/libtool/tests/_inst/lib/../
bin/cyghello-2.dlln
libtool: uninstall: rm -f
/Robert/libtool-HEAD/src/libtool/tests/_inst/lib/../b
in/cyghello-2.dlln
(ECHO in postinstall_cmds and postuninstall_cmds)
PASS: mdemo-conf.test
FAIL: mdemo-make.test
SKIP: mdemo-exec.test
SKIP: mdemo-inst.test
SKIP: mdemo-unst.test
FAIL: mdemo-dryrun.test
Syntax error in def files (EXPORTn)
PASS: mdemo-shared.test
FAIL: mdemo-make.test
SKIP: mdemo-exec.test
SKIP: mdemo-inst.test
SKIP: mdemo-unst.test
Same as previous
Robert
___
http://lists.gnu.org/mailman/listinfo/libtool
Re: Speeding up libtool
Robert Ögren writes: 2. configure.in did not have AC_LIBTOOL_WIN32_DLL which now is required for OBJDUMP to be set. Hmm. Wasn't AC_LIBTOOL_WIN32_DLL supposed to be deprecated and unnecessary? Or was it just that at some point it was de facto unnecessary but never officially deprecated, and now if using libtool HEAD it is again necessary ? --tml ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Speeding up libtool
Hi Robert,
* Robert Ögren wrote on Fri, Mar 25, 2005 at 01:24:52AM CET:
Ralf Wildenhues wrote:
Oh dear. This should be
CONFIG_SHELL=/bin/ash lt_ECHO='printf %s\n' /bin/ash configure [...]
instead.
This seems to mess up the result of func_win32_libid with recent libtool
from CVS (1.1888 2005/03/18 15:57:13). Instead of returning (echoing)
x86 archive import
it echoes
x86
archive
import
and that does not satisfy the egrep test.
Yes. We recently identified that an audit of libtool.m4 for wrong use
of $ECHO is necessary.
If you have time to produce a patch to this extent for the system(s) you
can test, that would be great! If not -- could I ask the favor to test
a patch on your systems? I will try to produce one by next week unless
anyone beats me to it.
*snip*
eval ${ECHO} EXPORTS ' $output_objdir/$outputname.def'
Ouch. This rule I had not thought of:
$ECHO may not be eval'ed.
It produces EXPORTSn instead of EXPORTS(newline) in the .def file. I
just replaced $ECHO with echo in archive_expsym_cmds to get the stuff to
work, but that has to be fixed properly somehow, and probably in several
places.
Yes. In this case, replacing it with `echo' is the right thing to do
(or another level of quoting, if possible: eval '$ECHO EXPORTS'...)
See [1] for the other rules which replacement is necessary.
(Note: Here I target normal Cygwin. Usually I use Cygwin with gc
c -mno-cygwin and --{build,host}=i686-pc-mingw32 to get binaries that
are native Win32 (don't need cygwin1.dll). For the libtool-cache
benchmark I wanted to test both alternatives.)
That would be great.
Another question: How did you find those bugs? I.e., were they exposed
by the Libtool testsuite? If not, how can we trigger them?
Thanks you very much for testing,
Ralf
[1] http://lists.gnu.org/archive/html/libtool-patches/2005-03/msg00213.html
___
http://lists.gnu.org/mailman/listinfo/libtool
Re: Speeding up libtool
Hi Ralf, Ralf Wildenhues wrote: ... Yes. We recently identified that an audit of libtool.m4 for wrong use of $ECHO is necessary. Oops, I glanced over the thread index for the patches list but missed that conversation. If you have time to produce a patch to this extent for the system(s) you can test, that would be great! If not -- could I ask the favor to test a patch on your systems? I will try to produce one by next week unless anyone beats me to it. I can provide a patch for the specific cases that gave me trouble if that's good enough. And I can test a patch that you create. But first I want to get the benchmark thingie finished. Hopefully later today if nothing unexpected happens. Another question: How did you find those bugs? I.e., were they exposed by the Libtool testsuite? If not, how can we trigger them? I just configured and built glib-2.6.3 (from ftp://ftp.gtk.org/pub/gtk/v2.6/) on Cygwin with export SHELL=/bin/sh export CONFIG_SHELL=/bin/sh export lt_ECHO='printf %s\n' I don't know if it happens with the testsuite. I did start it after building Libtool but must admit I got impatient and didn't let it finish. I didn't run it with sh (ash) either. But I will try to do that soon. Anyway, the problems I experienced are in Win32-specific code paths, so you need to run under Cygwin or similar to reproduce them. Robert ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Speeding up libtool
I wrote: I just configured and built glib-2.6.3 (from ftp://ftp.gtk.org/pub/gtk/v2.6/) on Cygwin with export SHELL=/bin/sh export CONFIG_SHELL=/bin/sh export lt_ECHO='printf %s\n' I'm sorry for replying to myself, but there are some compatibility issues with glib and Libtool HEAD that I forgot to mention. 1. glib's configure.in defines a few variables like LT_RELEASE, and these names are forbidden with libtool HEAD so I had to patch in a m4_pattern_allow for these 2. configure.in did not have AC_LIBTOOL_WIN32_DLL which now is required for OBJDUMP to be set. However, even though I added AC_LIBTOOL_WIN32_DLL before AM_PROG_LIBTOOL, it did not work, but if I replaced the call to AM_PROG_LIBTOOL with LT_INIT([win32-dll]) it worked. This needs to be investigated further. Maybe a testcase already exists that I should run... I will post the script I used for the benchmark, which patches the glib sources to work around these issues. Robert ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Speeding up libtool
Hi again Ralf, I finally have some numbers for you. The script I used is available at http://libtool-cache.sourceforge.net/libtool-cache-bench.sh What it basically does is to build GLib 2.6.3 using different versions of libtool and with and without caching. As I said before, I'm usually using Cygwin but targetting MinGW (using gcc -mno-cygwin) to get native win32 binaries, and pretending the host is also mingw to avoid having to set up cross-compilation stuff. This works very well with one minor change in ltmain.sh: Replace $LTCC $LTCFLAGS with /usr/bin/gcc at the spot where it builds the wrapper exes from c source. (I don't expect you to change this in Libtool, I'm just explaining how I use it. Maybe there is even a better way...) The test script does the following things for each configuration: 1. Unpack sources into a clean directory 2. Run aclocal, libtoolize etc etc 3. Run configure 4. Run make. At this stage, the cache databases are empty if caching is used. 5. Run make clean and make. Only the time for make is measured, not for make clean. Now the cache is fully populated, and additionally a few things like message catalogs are not rebuilt. (remake) 6. Remove libglib-2.0.la and relink it using make (about 50 objects are linked along with some libraries) (relink) 7. Remove gmain.lo and recompile it using make (recomp) After step 4 and 5 it also runs make check in a subdirectory to test that everything seemed to be compiled OK. Note: I only run a few tests since some don't seem to work when targetting win32. Times with MinGW (win32) target (but running in a Cygwin shell): (Running configure takes about 170 seconds) I) With bash and libtool 1.5.10: Without libtool-cache: make324 seconds remake 302 relink 11.9 recomp 3.4 With libtool-cache: make146 (speedup=2.2) remake 61 (speedup=5.0) relink 1.2 recomp 1.2 II) With sh and libtool 1.5.10: Without libtool-cache: make293 remake 272 relink 11.0 recomp 3.2 With libtool-cache: make134 (speedup=2.2) remake 58 (speedup=4.7) relink 1.2 recomp 1.2 III) With sh and libtool from CVS (1.1888 2005/03/18 15:57:13) 2.1a: Without libtool-cache: make302 remake 251 relink 5.2 recomp 2.9 With libtool-cache: make160 (speedup=1.9) remake 58 (speedup=4.3) relink 1.2 recomp 1.2 The test script was run with a command line like this: ./bench.sh testout 2 testlog on an otherwise unloaded machine with antivirus software completely disabled. Note that the time for relinking and recompiling includes the time taken by make and by gcc. The increase in time for the first compilation with Libtool HEAD is caused by a large amount of sed calls, mainly by the Quote evaled strings stuff from _LT_CONFIG_COMMANDS that is placed in config.status, since config.status is invoked a few extra times on win32 to create glib/glib.rc, gobject/gobject.rc, gmodule/gmodule.rc and gthread/gthread.rc. Each config.status invocation cost about 10 seconds. This is a minor thing, and could probably be fixed by adding them to AC_CONFIG_FILES in configure.in with some win32 conditional stuff. Times with Cygwin target: I) With bash and libtool 1.5.10: Without libtool-cache: make295 seconds remake 289 relink 13.4 recomp 3.1 With libtool-cache: make122 (speedup=2.4) remake 55 (speedup=5.3) relink 1.3 recomp 0.9 II) With sh and libtool 1.5.10: Without libtool-cache: make267 remake 261 relink 12.5 recomp 2.8 With libtool-cache: make114 (speedup=2.3) remake 53 (speedup=4.9) relink 1.3 recomp 0.9 III) With sh and libtool from CVS (1.1888 2005/03/18 15:57:13) 2.1a: Without libtool-cache: make254 remake 245 relink 5.9 recomp 2.7 With libtool-cache: make112 (speedup=2.3) remake 53 (speedup=4.6) relink 1.3 recomp 0.9 The test script was run with a command line like this: TARGET_CYGWIN=y ./bench.sh testout2 2 testlog2 As I said in another mail there were some compatibility issues: 1. glib's configure.in defines a few variables like LT_RELEASE, and these names are forbidden with Libtool HEAD so I had to patch in a m4_pattern_allow for these 2. configure.in did not have AC_LIBTOOL_WIN32_DLL which now is required for OBJDUMP to be set. However, even though I added AC_LIBTOOL_WIN32_DLL before AM_PROG_LIBTOOL, it did not work, but if I replaced the call to AM_PROG_LIBTOOL with LT_INIT([win32-dll]) it worked. 3. (I had to modify libtool-cache so it removes the libtool: compile: stuff that libtool HEAD prefixes output lines with) For Libtool HEAD I also rebuilt automake-2.59 and autoconf-1.9.2 so that Cygwin's Libtool files would not be picked up. Selected Cygwin package versions (I'm just using what Cygwin considers stable/current): autoconf-devel 2.59-1 automake-devel 1.9.2-1 bash 2.05b-16 binutils 20041229-1 coreutils 5.3.0-3 cygwin 1.5.13-1 gcc 3.3.3-3 gcc-mingw 20040810-1 libtool-devel 1.5.10-1 m4 1.4.2-1 make 3.80-1 mingw-runtime 3.7-1 sed 4.1.3-1 w32api 3.2-1 Conclusions: - Using (a)sh instead of bash makes Libtool faster -
Re: Speeding up libtool
Ralf Wildenhues wrote:
Oh dear. This should be
CONFIG_SHELL=/bin/ash lt_ECHO='printf %s\n' /bin/ash configure [...]
instead.
This seems to mess up the result of func_win32_libid with recent libtool
from CVS (1.1888 2005/03/18 15:57:13). Instead of returning (echoing)
x86 archive import
it echoes
x86
archive
import
and that does not satisfy the egrep test.
Replacing $ECHO $win32_libid_type with $ECHO $win32_libid_type solves
that problem. There is another problem with the code that generates .def
files using archive_expsym_cmds and perhaps also the lines that look
like this:
eval ${ECHO} EXPORTS ' $output_objdir/$outputname.def'
It produces EXPORTSn instead of EXPORTS(newline) in the .def file. I
just replaced $ECHO with echo in archive_expsym_cmds to get the stuff to
work, but that has to be fixed properly somehow, and probably in several
places. Here's some libtool output and corresponding .def file before my
ugly fix for this:
libtool: link: mv -f .libs/libglib-2.0.expT .libs/libglib-2.0.exp
libtool: link: if test x`/usr/bin/sed 1q .libs/libglib-2.0.exp` =
xEXPORTS; then cp .libs/libglib-2.0.exp .libs/cygglib-2.0-0.dll.def;
else printf %s\n EXPORTS .libs/cygglib-2.0-0.dll.def; cat
.libs/libglib-2.0.exp .libs/cygglib-2.0-0.dll.def; fi
EXPORTSng_allocator_free
g_allocator_new
g_array_append_vals
g_array_free
...
(Note: Here I target normal Cygwin. Usually I use Cygwin with gc
c -mno-cygwin and --{build,host}=i686-pc-mingw32 to get binaries that
are native Win32 (don't need cygwin1.dll). For the libtool-cache
benchmark I wanted to test both alternatives.)
Robert
___
http://lists.gnu.org/mailman/listinfo/libtool
Re: Speeding up libtool
Hi Max, * Max Bowsher wrote on Tue, Mar 15, 2005 at 11:14:42PM CET: Ralf Wildenhues wrote: for your projects. For best results on Cygwin, you should probably configure (with libtool HEAD) like CONFIG_SHELL=/bin/ash lt_ECHO='printf %s' /bin/ash configure [...] Oh dear. This should be CONFIG_SHELL=/bin/ash lt_ECHO='printf %s\n' /bin/ash configure [...] instead. (While we're at it, I found another mostly harmless bug in libtool exposed by ash which makes a couple of testsuite tests fail. Patch at eleven on the libtool-patches list.) (replace /bin/ash with the path to Cygwin's Almquist shell clone). That path would be /bin/sh, so it's possible libtool is using it already. I'm pretty sure to remember that the shell selection algorithm chooses bash over /bin/sh. It does so on other systems. Regards, Ralf ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Speeding up libtool
Hi Robert, * Robert Ögren wrote on Mon, Mar 14, 2005 at 11:11:00PM CET: I looked through the TODO thread from 2004-11 and saw some talk about speeding up libtool. I'm particularly interested in improved compilation speed on Cygwin (Windows). Is there any progress on this? For the HEAD branch of Libtool, I have made some changes that cause it to fork less often. For the particular bad case I was looking at (linking against ~500 objects), I got it from about 11 sec to about 5 sec, both of which include 2.5 sec of time spent actually linking (all data cached) on a fast linux/x86_64 machine. The improvements also benefit the general case, but are most likely less pronounced, even if Cygwin should be the candidate to benefit most. If you have resources/time to test, it would be nice to see a comparison for your projects. For best results on Cygwin, you should probably configure (with libtool HEAD) like CONFIG_SHELL=/bin/ash lt_ECHO='printf %s' /bin/ash configure [...] (replace /bin/ash with the path to Cygwin's Almquist shell clone). I have wanted to get this to work right automatically, but have not had time to look into it (the shell/ECHO selection is mixed Libtool/Autoconf code and focuses on portability more than speed -- there have been old and quite buggy ash variants on some systems). The speedup between using bash and using ash should be quite noticeable on Cygwin! A while ago I created a temporary solution for myself which caches the commands executed by libtool 1.5 to speed up the compilation process. It is a small Perl script that is run in place of libtool and that maintains a simple database of libtool command line = resulting commands by running the real libtool any time a command is not already in the database and parsing its standard out and storing the commands and .la files. It understands very little of what libtool actually does, the speedup is achieved by invoking libtool less often. This is a very nice idea! The hairy stuff is in the details, however. By replacing the name of the source file and the corresponding output file with placeholders if they match it usually only requires a few libtool invocations for compiling many source files. The link commands are usually unique but at least the second time a source tree is rebuilt they are already cached. Ah, ok. For use within the same $buildtree and compilation only this should be mostly fine (save locking, as mentioned in your README). Linking is a problem, though: shell wrappers break, as do generating symbol lists. Can't think of more right away, but I'm sure there are many more corner cases which break. This is aimed at software developers like myself who build projects from CVS and frequently modify some source files, compile and relink. The code was not written with portability in mind and certainly has bugs and quirks, but works well for me and should be possible to port to other platforms than Cygwin. The speedup might not be very large on those platforms however. But if anyone tries, please tell me how it worked. How much is the actual speedup in numbers? How much with the HEAD branch of Libtool? Iff this gives a decent speedup, I think the idea could belong in Libtool proper. That is, if you and the other Libtool developers are so inclined. Thinking out loud more: if this were created as output of `configure' (possibly acting upon some configure and/or libtoolize switch), it would also be possible to make libtool-cache (more) portable and much more foolproof. For example, if configure decides locking is necessary, the cache could be bypassed. $objdir, PATH_SEPARATOR, etc. would be trivially available, the `libtoolcacheize' beast could be killed.. We could add a marker whenever libtool does non-cacheable things so that the cache would be bypassed then. Or just let libtool output everything it does so that even writing wrappers would work. Maybe into a separate file descriptor. Most likely the cache would have to be invalidated whenever config.status changes. And it would have to be $top_builddir-local, so you'd have several caches for several subpackages. But for all of this to work reasonably, I think libtool needs to be audited so it outputs the executed commands very closely. I'd also think it would be possible to rewrite libtool-cache in portable shell, but then remaining speedup would have to be evaluated (one `perl' exec is, after all, not cheap). I'm pretty sure this would be more useful as a separate script than being directly integrated into the `libtool' script, since many shells need a long time for the initial parsing of `libtool'. Minor nit for your current script: Why do you remove comments from the created .la files? I don't see a benefit here. What do you think? What do the other Libtool developers think? Note that it's probably not worth investing work unless there is a demonstrable speedup involved. Regards, Ralf
Re: Speeding up libtool
Ralf Wildenhues writes: Linking is a problem, though: shell wrappers break, I have never liked (or understood...) libtool's shell wrappers or its relinking on Win32. I always use --disable-static, I always run a make install, and make sure the bin directory of the installation location is in my PATH, before any make check. And I always comment out the need_relink=yes in any ltmain.sh that I come across ;-) This means shell wrappers won't be produced. Or am I completely confused? How much is the actual speedup in numbers? Very significant. I don't have my old machine (450 MHz PIII running Win2k) around any longer, but libtool-cache made it feel like a totally new machine. A rough guess would five times faster builds of stuff like GLib or GTK. --tml ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Speeding up libtool
Hi Tor, * Tor Lillqvist wrote on Tue, Mar 15, 2005 at 10:32:41AM CET: Ralf Wildenhues writes: Linking is a problem, though: shell wrappers break, I have never liked (or understood...) libtool's shell wrappers or its relinking on Win32. I always use --disable-static, I always run a make install, and make sure the bin directory of the installation location is in my PATH, before any make check. And I always comment out the need_relink=yes in any ltmain.sh that I come across ;-) This means shell wrappers won't be produced. Or am I completely confused? Well, the feeling towards shell wrappers and their need in general are different matters. :) It might be the case that libtool actually creates wrappers in situations where they are not strictly necessary. This has been reported before, and I would be happy to take patches which improve on the situation while not compromising functionality and portability. But in general, if you link against uninstalled libraries, you will have to relink upon `make install'. In general, you then also have to have add some linker hackery upon execution of the uninstalled binary, so that it will actually pick up the uninstalled libraries versus older installed versions. If that is not the case on some architectures, we could optimize here. Honestly, I don't know off the head whether Cygwin falls into that category (I think win uses PATH for searching libraries). How much is the actual speedup in numbers? Very significant. I don't have my old machine (450 MHz PIII running Win2k) around any longer, but libtool-cache made it feel like a totally new machine. A rough guess would five times faster builds of stuff like GLib or GTK. Now it would _really_ be interesting to see how much speedup is left with Libtool HEAD using ash, vs. libtool-cache with Libtool HEAD using ash. And if there are bottlenecks left that can /easily/ be improved. Regards, Ralf ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Speeding up libtool
Ralf Wildenhues wrote:
Hi Robert,
Hi Ralf,
For the HEAD branch of Libtool, I have made some changes that cause it
to fork less often. For the particular bad case I was looking at
(linking against ~500 objects), I got it from about 11 sec to about 5
sec, both of which include 2.5 sec of time spent actually linking (all
data cached) on a fast linux/x86_64 machine.
The improvements also benefit the general case, but are most likely less
pronounced, even if Cygwin should be the candidate to benefit most.
If you have resources/time to test, it would be nice to see a comparison
for your projects. For best results on Cygwin, you should probably
configure (with libtool HEAD) like
CONFIG_SHELL=/bin/ash lt_ECHO='printf %s' /bin/ash configure [...]
Interesting, I'll have to find some time to try that soon.
By replacing the name of the source file and the corresponding output
file with placeholders if they match it usually only requires a few
libtool invocations for compiling many source files. The link commands
are usually unique but at least the second time a source tree is rebuilt
they are already cached.
Ah, ok. For use within the same $buildtree and compilation only this
should be mostly fine (save locking, as mentioned in your README).
Linking is a problem, though: shell wrappers break, as do generating
symbol lists. Can't think of more right away, but I'm sure there are
many more corner cases which break.
Shell wrappers? If you mean the shell scripts that are created to set up
PATH etc. before running an uninstalled EXE from .libs, they are also
saved by libtool-cache by using the same mechanism that is used to save
the .la{,i} files (serialize_ltfile) and it seems to work.
I just open the generated files, transform them into a set of echo
statements that recreate the files, and splice them into the command
sequence that Libtool outputs. The corresponding wrapper .exe file that
is generated by Libtool when running on Cygwin is just copied from a
precompiled version.
I don't handle the generation of symbol lists apart from ignoring the
line that Libtool prints when it's generating it, so that is probably
broken. I haven't needed it yet...
How much is the actual speedup in numbers?
I will have to get back to you after doing some proper benchmarks, but
you can find at least some numbers here:
http://mail.gnome.org/archives/gtk-devel-list/2005-March/msg00062.html
This is on a quite fast machine with lots of RAM for disk cache, however
I did leave the antivirus software running (although set to ignore the
source tree). I should turn that off and see if it makes any difference.
How much with the HEAD branch of Libtool?
Sorry, I must admit I haven't actually tried it yet, although I did
check out the source from anoncvs recently. I'll try it soon.
Iff this gives a decent speedup, I think the idea
could belong in Libtool proper. That is, if you and the other Libtool
developers are so inclined.
I certainly wouldn't mind this functionality being included in Libtool
(or perhaps just becoming unnecessary if Libtool is rewritten in C or
bytecode). If anyone wants to they should feel free to improve my crude
code or just take the useful ideas and reimplement them properly. I
can't promise that I'll have time myself to do that implementation.
Thinking out loud more: if this were created as output of `configure'
(possibly acting upon some configure and/or libtoolize switch), it would
also be possible to make libtool-cache (more) portable and much more
foolproof. For example, if configure decides locking is necessary, the
cache could be bypassed. $objdir, PATH_SEPARATOR, etc. would be
trivially available, the `libtoolcacheize' beast could be killed..
That would be great.
We could add a marker whenever libtool does non-cacheable things so that
the cache would be bypassed then. Or just let libtool output
everything it does so that even writing wrappers would work.
Maybe into a separate file descriptor.
Yes, that would be nice. Right now I remove some messages that Libtool
writes, and try to detect warning messages and skip caching in such
cases. But it is not perfect.
Most likely the cache would have to be invalidated whenever
config.status changes. And it would have to be $top_builddir-local, so
you'd have several caches for several subpackages.
But for all of this to work reasonably, I think libtool needs to be
audited so it outputs the executed commands very closely.
Agreed.
I'd also think it would be possible to rewrite libtool-cache in portable
shell, but then remaining speedup would have to be evaluated (one `perl'
exec is, after all, not cheap). I'm pretty sure this would be more
useful as a separate script than being directly integrated into the
`libtool' script, since many shells need a long time for the initial
parsing of `libtool'.
Also agree.
Minor nit for your current script: Why do you remove comments from the
created .la files? I don't see a benefit here.
That is just
Re: Speeding up libtool
Ralf Wildenhues wrote: for your projects. For best results on Cygwin, you should probably configure (with libtool HEAD) like CONFIG_SHELL=/bin/ash lt_ECHO='printf %s' /bin/ash configure [...] (replace /bin/ash with the path to Cygwin's Almquist shell clone). That path would be /bin/sh, so it's possible libtool is using it already. Max. ___ http://lists.gnu.org/mailman/listinfo/libtool
