I think the test round about line 144 needs to be a little different. For
instance, aix and solaris (at least the versions we use at work) claim to
support posix_spawn and the man pages are identical, but they don't identify
themselves as linux
One other question that comes to mind from time to time: do we /really/ need to
spawn a shell just to execute a command. Obviously if it include '>' and other
shell specials, you should. I'd really like to make it do something like perl:
If you pass an array [ 'prog', '$TARGET', '$SOURCE' ] it doesn't execute a
shell. Otherwise (passing a single string), it will examine for special
characters, and if none are found it will split on white space and execute
that, or it will pass that line to the shell.
NB Yes, I realise this will potentially break things, but why execute a shell
if you don't have to?
----- Original Message -----
From: scons-dev@scons.org
To: scons-dev@scons.org
At: Apr 1 2014 17:13:48
I've found posix spawn can be much faster than fork/exec with large memory
processes, so I'd be in favor of this. Not every system has it though so there
would have to be a fallback to fork/exec.
--
Gary Oberbrunner
(sent from my Android)
On Apr 1, 2014 11:52 AM, "Kenny, Jason L" <jason.l.ke...@intel.com> wrote:
Hi guys,
I just got a patch to Parts internal at Intel to fix some issues with
subprocess. I find myself sort of surprised by this, and honestly felt that
this seems to be an issue that should be looked at in Scons as well.
The problem is this. We have been building a given huge product here at Intel
on RHEL 4.8. IT is old and finally time has come to start moving to a newer
system. What was found when we moved to a newer RHEL 5.10 ( I know bleeding
edge J ) we found that the build was twice as slow. Looking into it more it was
found that it was the use of fork() in subprocess. It was found that
posix_spawn() fixed this. Attached is a monkey patch that is up for internal
review to modify subprocess module a little to fix this for Unix based systems.
Now I am not an expert on Linux internal details. However I thought fork() was
the way process got spawned on Linux. Ie fork() was the win32 CreateProcess().
I am clearly wrong, but this seems to be a problem I would not expect to have
seen on Linux. The root of the problem seems to be that the system takes a long
time to setup the fork when you have a large build ( and we have a very large
build ) because of the memory usage (2-6GB) needed by SCons alone. Using a
different method to create a process that does not try to “clone” memory seems
to be the fix that gets the build times to be about ~2x+ faster.
Does anyone else have input on this? IF this is a good patch, it seem that we
will want to apply it to SCons, for a speed boost.
Thanks
Jason
Also I attached a test file that to show fork() is the problem. You can see the
difference importing the stubprocess.py file and running the test again.
_______________________________________________
Scons-dev mailing list
Scons-dev@scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev
_______________________________________________ Scons-dev mailing list
Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
_______________________________________________
Scons-dev mailing list
Scons-dev@scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev
