Re: ports/94826: [patch] Very slow startup for libtool ltdl clients (eg gnucash)
* Peter Jeremy wrote on Wed, Mar 22, 2006 at 07:42:33PM CET: I should acknowledge that the problem and fix were pointed out by [EMAIL PROTECTED] Thanks, I'll mention Joerg. On Wed, 2006-Mar-22 17:28:11 +0100, Ralf Wildenhues wrote: I'll put adding such a test on the TODO list. I'll also put reworking the manual loading algorithm on the TODO list, it shouldn't scale nonlinearly in the number of loaded deplibs, however many modules we open. I've still got a trace of all calls to lt_dlopenext() during gnucash startup but I've cleaned up the more detailed debugging output that I collected (though I could probably re-create it). I suspect it's a combination of: - both dlopen() and lt_dlopen() are recursively attempting to load dependencies Right. - The .la file can contain duplicate dependencies (which don't appear to be stripped out) This is normal: there are systems where you actually need to link each library against all its deplibs. So we store all of them. The linker and runtime linker on FreeBSD both follow DT_NEEDED entries and look in DT_RPATH entries inside libraries for finding and loading indirect library dependencies, though (I think). So it should be possible to improve quite a bit upon your 10 seconds startup time, by changing Libtool to not link in indirect deplibs on FreeBSD (and DragonFly). This would be similar to the changes that Debian currently has in their Libtool package. We plan to fix need_all_deplibs=no (it will have a different name then, and it has a couple of known issues at the moment), and then make it available for all systems that support this. (You'd have to relink all libraries once for this, though.) - It looks like duplicate entries in the library search path (inherited from LD_LIBRARY_PATH and added via .la) aren't deleted Ah, ok. This is another limitation in libltdl. - Tracking lt_dlopen/lt_dlclose shows that libraries are being opened and closed multiple times during startup. Hmm. Surely for FreeBSD this should be a non-issue now; but for systems where we _do_ need to load all deplibs, I'm not sure whether we can get around unloading them in the case that the module itself could not be opened. After all, those deplibs could have been the wrong ones to use in the first place, but we're not able to know until we've tried loading the module itself. Luckily this issue is important on very few systems only, today. Before joerg pointed me in the right direction, I was looking at adding caching to lt_dlopenext() or lt_dlopen() but hadn't worked out how to correctly track reference counts across open/close calls. I think that should be possible, but then again the details may depend on the system in question. I'll try to test on Tru64 when I have some more time at hand. Cheers, Ralf ___ Bug-libtool mailing list Bug-libtool@gnu.org http://lists.gnu.org/mailman/listinfo/bug-libtool
Re: ports/94826: [patch] Very slow startup for libtool ltdl clients (eg gnucash)
Hi Noah, * Noah Misch wrote on Fri, Mar 24, 2006 at 01:45:33AM CET: On Wed, Mar 22, 2006 at 05:28:11PM +0100, Ralf Wildenhues wrote: * Peter Jeremy wrote on Wed, Mar 22, 2006 at 09:27:31AM CET: Ideally, libtool should implement a test case to determine whether an OS tracks shared library dependencies or now. This should definitely be fixed. However, I really don't know how to write such a test so that it also works well in the cross compiling case. So for that we would still need the table. But at least in the non-cross-compiling case it would also fix the cases where we are not sure whether the system supports it right, or where the system needs some patch/upgrade to fix this issue. A native build could use the test but compare the result with the tabulated value used for cross-compiling, perhaps asking the user to report discrepancies. Yes. But then I think it's not so much better than a good testsuite test that reports this, but carries the overhead that every package has to ship with this test. Let's be realistic here: Libtool is a package that pretty much needs to be adapted for each new system it should support (except that static libraries should work out-of-the-box); this was recognized a long time ago, and consequently, it has very few runtime tests (compared to the number of decisions) in its configuration, which at least takes away some of the configure-time-overhead burden away from the end user. So, when we accept the fact that Libtool needs active porting, the remaining point is IMVHO that we should improve the test suite insofar that it has a chance to expose as many missing or wrong settings. Then a passing testsuite is a good indication of a good port. Actually, I think we're moving in the right direction with the test suite of CVS HEAD Libtool. Also please note that a portable way of testing above feature would incur quite a large overhead: basically the libltdl loaders would need to be provided from within `configure', the right one compiled, and then the test itself. I don't think that is such a good idea. Cheers, Ralf ___ Bug-libtool mailing list Bug-libtool@gnu.org http://lists.gnu.org/mailman/listinfo/bug-libtool
Re: ports/94826: [patch] Very slow startup for libtool ltdl clients (eg gnucash)
* Peter Jeremy wrote on Wed, Mar 22, 2006 at 07:42:33PM CET: On Wed, 2006-Mar-22 17:28:11 +0100, Ralf Wildenhues wrote: Thanks for the patch. A question before I apply it: this should work on DragonFly as well, right? Could somebody confirm this, to avoid another bug report to this end? joerg found the problem using KDE on DragonFly. Also, has the FreeBSD rtld always supported loading dependent libraries upon dlopen() (including honoring DT_RPATH entries in the module) or was this not working at some point in the past? AFAIK, it's worked since FreeBSD transitioned to ELF (FreeBSD3.0). I'm not sure about the a.out rtld but FreeBSD 2.x is ancient history. OK, thanks. FWIW, for the historically interested, http://www.freebsd.org/cgi/cvsweb.cgi/src/libexec/rtld-aout/Attic/rtld.c?hideattic=0 indicates this to be working in aout from rev. 1.25, backported to FreeBSD 2.1. I've applied the following patches into Libtool branch-1-5 and HEAD, respectively. Thanks again. Cheers, Ralf branch-1-5: * ltdl.m4 (AC_LTDL_SYS_DLOPEN_DEPLIBS) [ freebsd, dragonfly ]: Set libltdl_cv_sys_dlopen_deplibs to yes. Fixes excessive lt_dlopen times on these systems. * NEWS, THANKS: Update. Bug reported by Peter Jeremy [EMAIL PROTECTED], patch by Joerg Sonnenberger [EMAIL PROTECTED]. Index: NEWS === RCS file: /cvsroot/libtool/libtool/NEWS,v retrieving revision 1.109.2.44 diff -u -r1.109.2.44 NEWS --- NEWS16 Mar 2006 20:48:52 - 1.109.2.44 +++ NEWS23 Mar 2006 19:27:05 - @@ -6,6 +6,9 @@ * Fix regression on OpenBSD, NetBSD, DragonFly, and other systems with hardcode_direct=yes that wrongly removed paths to uninstalled libraries during link mode. +* Let libltdl know that FreeBSD and DragonFly dlopen causes dependent + modules to be loaded. This fixes excessive load times for modules + with large library dependency graphs. * Bug Fixes. New in 1.5.22: 2005-12-18; CVS version 1.5.21a, Libtool team: Index: THANKS === RCS file: /cvsroot/libtool/libtool/THANKS,v retrieving revision 1.34.2.17 diff -u -r1.34.2.17 THANKS --- THANKS 16 Mar 2006 20:48:52 - 1.34.2.17 +++ THANKS 23 Mar 2006 19:27:05 - @@ -102,6 +102,7 @@ Patrick Welche [EMAIL PROTECTED] Paul Eggert [EMAIL PROTECTED] Peter Eisentraut [EMAIL PROTECTED] + Peter Jeremy [EMAIL PROTECTED] Rainer Orth [EMAIL PROTECTED] Ralf Menzel [EMAIL PROTECTED] Robert Ögren [EMAIL PROTECTED] Index: ltdl.m4 === RCS file: /cvsroot/libtool/libtool/Attic/ltdl.m4,v retrieving revision 1.47.2.11 diff -u -r1.47.2.11 ltdl.m4 --- ltdl.m4 18 Dec 2005 22:14:06 - 1.47.2.11 +++ ltdl.m4 23 Mar 2006 19:27:05 - @@ -127,6 +127,9 @@ # If you are looking for one http://www.opendarwin.org/projects/dlcompat libltdl_cv_sys_dlopen_deplibs=yes ;; + freebsd* | dragonfly*) +libltdl_cv_sys_dlopen_deplibs=yes +;; gnu* | linux* | kfreebsd*-gnu | knetbsd*-gnu) # GNU and its variants, using gnu ld.so (Glibc) libltdl_cv_sys_dlopen_deplibs=yes HEAD: * libltdl/m4/ltdl.m4 (LT_SYS_DLOPEN_DEPLIBS) [ freebsd ] [ dragonfly ]: Set libltdl_cv_sys_dlopen_deplibs to yes. Fixes excessive lt_dlopen times on these systems. * NEWS, THANKS: Update. Bug reported by Peter Jeremy [EMAIL PROTECTED], patch by Joerg Sonnenberger [EMAIL PROTECTED]. Index: NEWS === RCS file: /cvsroot/libtool/libtool/NEWS,v retrieving revision 1.191 diff -u -r1.191 NEWS --- NEWS12 Jan 2006 22:02:55 - 1.191 +++ NEWS23 Mar 2006 09:47:43 - @@ -20,7 +20,7 @@ * Support for linux-dietlibc (`diet' as well as `diet-dyn', separately). * Shell optimizations which break use of the stdin file descriptor in libtool. * `libtoolize --install' now also installs `install-sh'. -* Support (mostly) for DragonFly BSD. +* Support for DragonFly BSD, improved support for FreeBSD. * Allow shell special characters like `$' in source file names, but not in object names, to enhance GCJ support. * Detection of compiler wrappers like distcc/ccache and $host_alias prefix. Index: THANKS === RCS file: /cvsroot/libtool/libtool/THANKS,v retrieving revision 1.54 diff -u -r1.54 THANKS --- THANKS 16 Mar 2006 20:47:54 - 1.54 +++ THANKS 23 Mar 2006 09:47:43 - @@ -102,6 +102,7 @@ Patrick Welche [EMAIL PROTECTED] Paul Eggert [EMAIL PROTECTED] Peter Eisentraut [EMAIL PROTECTED] + Peter Jeremy [EMAIL
