Jeremy Chadwick wrote:
On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote:
I have been thinking a lot about looking for speed increases for "make
index" and pkg_version and things like that. So for example, in
pkg_version, it calls "make -V PKGNAME" for every installed package. Now
"make -V PKGNAME" should be a speedy operation, but the make has to load in
and analyze bsd.port.mk, a quite complicated file with about 200,000
characters in it, when all it is needing to do is to figure out the value of
the variable PKGNAME.
I have a related question, pertaining to "make all-depends-list" and the
utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't
know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in
bsd.ports.mk.
I call it an atrocity because it's a mix of make variable expansion
combined with sh scripting, and it's nearly impossible to read. It's
not commented either, so folks like myself are left thinking "What IS
this mess?!". It's expanded via $$(${ALL-DEPENDS-LIST}) in for loops,
throughout several places in bsd.port.mk.
I do not entirely understand what ALL-DEPENDS-LIST is about (that should
be apparent), but upon performing some of my benchmarks, I found this to
be a very slow piece of bsd.port.mk. make -V _DEPEND_DIRS is incredibly
fast, but ALL-DEPENDS-LIST is not.
Does it need to be done this way? Can we just iterate through all of
the ports, call make -V _DEPEND_DIRS, then sort | uniq the results? I
suppose that depends on the operation (make vs. make clean vs.
others)...
The port I used for testing some of the benchmarks was net/gacxtool. It
seems to be a good example of a "hefty" port.
I looked very hard at this particular piece of code. Once you
understand how it works, you realize that it is rather well written. It
basically does what you suggest, except it keeps a list of ports it has
already checked, so that it doesn't do them over again. This piece of
code is as efficient as it can possibly be, given that the program has
to recursively call make on every dependency port at least once (and as
it happens only once).
I suggest rewriting "make" so that variables are only evaluated on a "need
to know" basis. So, for example, if all we need to know is PKGNAME, there
is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of
that particular port has done something like having PORTNAME depend on the
value of _RUN_LIB_DEPENDS. So "make" should analyze all the code it is
given, and only figure it out if it is needed to do so. This would include,
for example, figuring out .for and .if directives on a need to know basis as
well.
This sounds like a good solution. In fact, I'm lead to believe that
heavy reliance on /bin/sh is part of why the ports collection is slow.
No, it's not the sole reason, but it's one of many. I'm of the belief
that anything we can do to migrate portions into native make would be
benefitial.
I have done profiling tests on make, and in its current form,
bsd.ports.mk actually spends rather little time inside of bin/sh. Thw
writers of bsd.ports.mk have done a very good job of minimizing the
bin/sh calls.
That said, I'll ask this out in the open: am I the only one who sees the
benefit of GNU make in regards to this? There's a lot of built-in
functions in GNU make which could help in regards to ports. I have no
qualms with PMake per se, but if another tool gives us what we need,
then maybe we should consider the pros and cons of adapting that.
There's also CMake, which is incredibly fast.
Maybe I should look at the inner workings of cmake and gmake. Maybe
they have some good ideas. However having looked through the source
code of make, and also looking at the cvs logs, it does seem to be well
written. The only possibility I see of making it go a lot faster is a
complete redesign, e.g. my just in time idea for processing variables.
Stephen
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
