On Thu, Feb 4, 2010 at 6:51 AM, Craig Citro <[email protected]> wrote:
> FWIW, this isn't so hard to do. I implemented this for Sage, and I 2am > actually planning to try and push something upstream soon. The biggest > issue (well, other than "the state of the distutils community" :P) is > that there's currently a single method in build_ext which both (1) > decides whether or not an extension needs rebuilt, and (2) actually > rebuilds it. If you just naively call this in parallel, especially on > a rebuild, you end up spawning a whole bunch of threads which decide > not to rebuild, and end up losing the speedup from doing the work in > parallel. The obvious fix is to just split this logic; this could > definitely be done in such a way as to be backward-compatible, but it > means that any project that implements its own builder (i.e. > subclasses build_ext) can't take advantage of this without reworking > their code. (We ran into exactly this problem with numpy when I first > tried to merge this into Sage.) One issue with threading in general build tools is to synchronize things if you parallelize at a fine grained level. I guess you would not encounter this is you only parallelize at the build_ext/build_clib level, but one constant source of issues with make -j8 is that ar/ranlib are run before every .o is done (when you update archives, in particular). Another issue is controlling memory pressure on some very expensive operations (typically c++ compilation and linking). If you try building scipy on a 4 cores machine, you almost have to use a 64 bits OS to get enough memory. Both scons and waf has some relatively elaborate job scheduler (waf's book has a nice chapter on it: http://freehackers.org/~tnagy/wafbook/single.html#job_control). The last issue with // builds you have to care about is tasks which are not thread-safe and written in python. For scipy, the canonical example is f2py; you need to execute f2py in its own process. cheers, David _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
