Jeroen T. Vermeulen has proposed merging lp:~jtv/launchpad/parallel-make into
lp:launchpad/devel.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers): code
= Parallel make =
This fixes the Makefile so that the default target (or "build") can be built in
parallel. So instead of "make" you can do "make -j2" and get a nice speedup:
on my system right after a "make clean" it takes the build time from 2:43 to
1:32. Should be a welcome change.
What changes? A single buildout run in the makefile produces bin/py and a
bunch of other files. But "make" isn't aware that there's no point to running
two buildout instances at once, and indeed it's likely to break.
So I channeled all the files produced by buildout through a single virtual
target. Now "make" realizes when it's about to run two buildout instances in
parallel that they are actually producing the same target—and wait for the
existing run to complete instead of starting a redundant one.
(Actually I had to start out with completing dependencies on bin/py, because
not all targets that used it depended on it. These things become more
noticeable when you run parallel builds.)
The good news is that the build parallelizes almost perfectly to two processes;
the bad news is that that's where the ride ends. This is because one
single-process job, the generation of WADL files, takes up just about half the
build time. With two processors, the two halvse will complete almost
simultaneously at just over half the single-process build time. Adding a third
CPU won't help the WADL target, and so can't do much for overall build time.
At best it gets you to the point where everything waits for that one big job a
little faster. On my dual-core SMT laptop, a third process even slows the
build down a little.
Jeroen
--
https://code.launchpad.net/~jtv/launchpad/parallel-make/+merge/41108
Your team Launchpad code reviewers is requested to review the proposed merge of
lp:~jtv/launchpad/parallel-make into lp:launchpad/devel.
=== modified file 'Makefile'
--- Makefile 2010-11-02 01:34:05 +0000
+++ Makefile 2010-11-17 22:07:46 +0000
@@ -55,10 +55,10 @@
newsampledata:
$(MAKE) -C database/schema newsampledata
-hosted_branches: $(PY)
+hosted_branches: buildout_bin
$(PY) ./utilities/make-dummy-hosted-branches
-$(API_INDEX): $(BZR_VERSION_INFO)
+$(API_INDEX): $(BZR_VERSION_INFO) buildout_bin
mkdir -p $(APIDOC_DIR).tmp
LPCONFIG=$(LPCONFIG) $(PY) ./utilities/create-lp-wadl-and-apidoc.py --force "$(WADL_TEMPLATE)"
mv $(APIDOC_DIR).tmp $(APIDOC_DIR)
@@ -66,12 +66,12 @@
apidoc: compile $(API_INDEX)
# Run by PQM.
-check_merge: $(PY)
+check_merge: buildout_bin
[ `PYTHONPATH= bzr status -S database/schema/ | \
grep -v "\(^P\|pending\|security.cfg\|Makefile\|unautovacuumable\|_pythonpath.py\)" | wc -l` -eq 0 ]
${PY} lib/canonical/tests/test_no_conflict_marker.py
-check_db_merge: $(PY)
+check_db_merge: buildout_bin
${PY} lib/canonical/tests/test_no_conflict_marker.py
check_config: build
@@ -109,16 +109,16 @@
${PY} -t ./test_on_merge.py $(VERBOSITY) $(TESTOPTS) \
--layer=MailmanLayer
-lint: ${PY}
+lint: buildout_bin
@bash ./bin/lint.sh
-lint-verbose: ${PY}
+lint-verbose: buildout_bin
@bash ./bin/lint.sh -v
-xxxreport: $(PY)
+xxxreport: buildout_bin
${PY} -t ./utilities/xxxreport.py -f csv -o xxx-report.csv ./
-check-configs: $(PY)
+check-configs: buildout_bin
${PY} utilities/check-configs.py
pagetests: build
@@ -145,7 +145,7 @@
# roll-up files. They fiddle with built-in functions! See Bug 482340.
${SHHH} bin/jsbuild $(JSFLAGS) -b $(LAZR_BUILT_JS_ROOT) -x testing/ -c $(LAZR_BUILT_JS_ROOT)/yui
-jsbuild: jsbuild_lazr bin/jsbuild bin/jssize
+jsbuild: jsbuild_lazr bin/jsbuild bin/jssize buildout_bin
${SHHH} bin/jsbuild \
$(JSFLAGS) \
-n launchpad \
@@ -173,7 +173,7 @@
@exit 1
endif
-buildonce_eggs: $(PY)
+buildonce_eggs: buildout_bin
find eggs -name '*.pyc' -exec rm {} \;
# The download-cache dependency comes *before* eggs so that developers get the
@@ -192,19 +192,21 @@
# and the other bits might run into problems like bug 575037. This
# target runs buildout, and then removes everything created except for
# the eggs.
-build_eggs: $(BUILDOUT_BIN) clean_buildout
+build_eggs: buildout_bin clean_buildout
+
+$(BUILDOUT_BIN): buildout_bin
# This builds bin/py and all the other bin files except bin/buildout.
# Remove the target before calling buildout to ensure that buildout
# updates the timestamp.
-$(BUILDOUT_BIN): bin/buildout versions.cfg $(BUILDOUT_CFG) setup.py
+buildout_bin: bin/buildout versions.cfg $(BUILDOUT_CFG) setup.py
$(RM) $@
$(SHHH) PYTHONPATH= ./bin/buildout \
configuration:instance_name=${LPCONFIG} -c $(BUILDOUT_CFG)
# bin/compile_templates is responsible for building all chameleon templates,
# of which there is currently one, but of which many more are coming.
-compile: $(PY) $(BZR_VERSION_INFO)
+compile: buildout_bin $(BZR_VERSION_INFO)
mkdir -p /var/tmp/vostok-archive
${SHHH} $(MAKE) -C sourcecode build PYTHON=${PYTHON} \
LPCONFIG=${LPCONFIG}
@@ -463,7 +465,7 @@
--docformat restructuredtext --verbose-about epytext-summary \
$(PYDOCTOR_OPTIONS)
-.PHONY: apidoc check tags TAGS zcmldocs realclean clean debug stop\
+.PHONY: apidoc buildout_bin check tags TAGS zcmldocs realclean clean debug stop\
start run ftest_build ftest_inplace test_build test_inplace pagetests\
check check_merge \
schema default launchpad.pot check_merge_ui pull scan sync_branches\
_______________________________________________
Mailing list: https://launchpad.net/~launchpad-reviewers
Post to : [email protected]
Unsubscribe : https://launchpad.net/~launchpad-reviewers
More help : https://help.launchpad.net/ListHelp
