Hi Robert, * Robert Ögren wrote on Mon, Mar 14, 2005 at 11:11:00PM CET: > > I looked through the TODO thread from 2004-11 and saw some talk about > speeding up libtool. I'm particularly interested in improved compilation > speed on Cygwin (Windows). Is there any progress on this?
For the HEAD branch of Libtool, I have made some changes that cause it to fork less often. For the particular bad case I was looking at (linking against ~500 objects), I got it from about 11 sec to about 5 sec, both of which include 2.5 sec of time spent actually linking (all data cached) on a fast linux/x86_64 machine. The improvements also benefit the general case, but are most likely less pronounced, even if Cygwin should be the candidate to benefit most. If you have resources/time to test, it would be nice to see a comparison for your projects. For best results on Cygwin, you should probably configure (with libtool HEAD) like CONFIG_SHELL=/bin/ash lt_ECHO='printf %s' /bin/ash configure [...] (replace /bin/ash with the path to Cygwin's Almquist shell clone). I have wanted to get this to work right automatically, but have not had time to look into it (the shell/ECHO selection is mixed Libtool/Autoconf code and focuses on portability more than speed -- there have been old and quite buggy ash variants on some systems). The speedup between using bash and using ash should be quite noticeable on Cygwin! > A while ago I created a temporary solution for myself which caches the > commands executed by libtool 1.5 to speed up the compilation process. It > is a small Perl script that is run in place of libtool and that > maintains a simple database of libtool command line => resulting > commands by running the real libtool any time a command is not already > in the database and parsing its standard out and storing the commands > and .la files. It understands very little of what libtool actually does, > the speedup is achieved by invoking libtool less often. This is a very nice idea! The hairy stuff is in the details, however. > By replacing the name of the source file and the corresponding output > file with placeholders if they match it usually only requires a few > libtool invocations for compiling many source files. The link commands > are usually unique but at least the second time a source tree is rebuilt > they are already cached. Ah, ok. For use within the same $buildtree and compilation only this should be mostly fine (save locking, as mentioned in your README). Linking is a problem, though: shell wrappers break, as do generating symbol lists. Can't think of more right away, but I'm sure there are many more corner cases which break. > This is aimed at software developers like myself who build projects from > CVS and frequently modify some source files, compile and relink. > > The code was not written with portability in mind and certainly has bugs > and quirks, but works well for me and should be possible to port to > other platforms than Cygwin. The speedup might not be very large on > those platforms however. But if anyone tries, please tell me how it worked. How much is the actual speedup in numbers? How much with the HEAD branch of Libtool? Iff this gives a decent speedup, I think the idea could belong in Libtool proper. That is, if you and the other Libtool developers are so inclined. Thinking out loud more: if this were created as output of `configure' (possibly acting upon some configure and/or libtoolize switch), it would also be possible to make libtool-cache (more) portable and much more foolproof. For example, if configure decides locking is necessary, the cache could be bypassed. $objdir, PATH_SEPARATOR, etc. would be trivially available, the `libtoolcacheize' beast could be killed.. We could add a marker whenever libtool does non-cacheable things so that the cache would be bypassed then. Or just let libtool output "everything" it does so that even writing wrappers would work. Maybe into a separate file descriptor. Most likely the cache would have to be invalidated whenever config.status changes. And it would have to be $top_builddir-local, so you'd have several caches for several subpackages. But for all of this to work reasonably, I think libtool needs to be audited so it outputs the executed commands very closely. I'd also think it would be possible to rewrite libtool-cache in portable shell, but then remaining speedup would have to be evaluated (one `perl' exec is, after all, not cheap). I'm pretty sure this would be more useful as a separate script than being directly integrated into the `libtool' script, since many shells need a long time for the initial parsing of `libtool'. Minor nit for your current script: Why do you remove comments from the created .la files? I don't see a benefit here. What do you think? What do the other Libtool developers think? Note that it's probably not worth investing work unless there is a demonstrable speedup involved. Regards, Ralf _______________________________________________ http://lists.gnu.org/mailman/listinfo/libtool
