[Scons-dev] Mercurial and remote branches
Martin Geisler just told me about: http://mercurial.selenic.com/wiki/RemoteBranchesExtension It is remote branches for Mercurial. I may well be able to like Mercurial more with this :-) -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] Subprocess issue on Linux?
Hi guys, I just got a patch to Parts internal at Intel to fix some issues with subprocess. I find myself sort of surprised by this, and honestly felt that this seems to be an issue that should be looked at in Scons as well. The problem is this. We have been building a given huge product here at Intel on RHEL 4.8. IT is old and finally time has come to start moving to a newer system. What was found when we moved to a newer RHEL 5.10 ( I know bleeding edge :) ) we found that the build was twice as slow. Looking into it more it was found that it was the use of fork() in subprocess. It was found that posix_spawn() fixed this. Attached is a monkey patch that is up for internal review to modify subprocess module a little to fix this for Unix based systems. Now I am not an expert on Linux internal details. However I thought fork() was the way process got spawned on Linux. Ie fork() was the win32 CreateProcess(). I am clearly wrong, but this seems to be a problem I would not expect to have seen on Linux. The root of the problem seems to be that the system takes a long time to setup the fork when you have a large build ( and we have a very large build ) because of the memory usage (2-6GB) needed by SCons alone. Using a different method to create a process that does not try to clone memory seems to be the fix that gets the build times to be about ~2x+ faster. Does anyone else have input on this? IF this is a good patch, it seem that we will want to apply it to SCons, for a speed boost. Thanks Jason Also I attached a test file that to show fork() is the problem. You can see the difference importing the stubprocess.py file and running the test again. subprocess_test.py Description: subprocess_test.py stubprocess.py Description: stubprocess.py ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi there, On 01.04.2014 18:13, Gary Oberbrunner wrote: I've found posix spawn can be much faster than fork/exec with large memory processes, so I'd be in favor of this. Not every system has it though so there would have to be a fallback to fork/exec. -- Gary Oberbrunner (sent from my Android) On Apr 1, 2014 11:52 AM, Kenny, Jason L jason.l.ke...@intel.com mailto:jason.l.ke...@intel.com wrote: Hi guys, I just got a patch to Parts internal at Intel to fix some issues with subprocess. I find myself sort of surprised by this, and honestly felt that this seems to be an issue that should be looked at in Scons as well. [...] Does anyone else have input on this? IF this is a good patch, it seem that we will want to apply it to SCons, for a speed boost. a few minutes ago I updated the page http://www.scons.org/wiki/WhySconsIsNotSlow with my latest results. Check out the bottom of the page, and the repository http://www.bitbucket.org/dirkbaechle/scons_testresults for all the results. I'm also talking to Tzvetan Mikov off-list, he volunteered to write a CPython extension for posix_spawn() to help SCons out. I'll try to get him subscribed to this list, so the three of you can talk some more about this, okay? Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
