Re: [Scons-dev] Subprocess issue on Linux?
I have no problems with stubprocess.py on my Mac. I ran the test suite and my day-job project with 32 and 64-bit python 2.7, and all seems well. -- Gary ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On 23.04.2014 01:42, Bill Deegan wrote: Dirk, So if I understand correctly, the stubprocess patch passes all the regression tests and is signficantly faster than the current implementation? That's correct, but I'm not sure whether things like the redirection of stdout/stderr works in all cases, or might create quirky problems in certain situations. I just can't judge the technical stability level of the stubprocess.py source. Is there any downside to using it? (Does it work on py3?) I haven't tried the wrapper under Python3 yet. But as far as I understood Eugene and Jason, it's not implemented. Another downside is that this doesn't work under Windows obviously, I don't have any infos about OSx, and Alexandre Leblot already asked me in a PM about the support for Solaris. I can't test these things, so I've no idea. ;) If not is there any reason not to send a pull request? Creating a pull request is probably not difficult, on the technical level. But should the wrapper get activated automatically under Linux, transparently for the user? Or would it be an experimental option at first, that the user can activate via a command-line option...since we aren't that sure about all the possible implications? Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Gary, Please find the stubprocess.py in the attachment. To make it work you will need to import it. Just put it next to SCons/Platform/posix.py and modify the latter by adding ‘import strubprocess’ just after ‘import subprocess’. -- Regards, Eugene From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Gary Oberbrunner Sent: Wednesday, April 23, 2014 3:16 PM To: Dirk Baechle; SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? I can try it on OSX, and review the stubprocess.py source (which I haven't yet). I don't think we need to worry about Windows; it has its own process creation methods with their own characteristics and bugs. On Wed, Apr 23, 2014 at 3:07 AM, Dirk Bächle tshor...@gmx.demailto:tshor...@gmx.de wrote: On 23.04.2014 01:42, Bill Deegan wrote: Dirk, So if I understand correctly, the stubprocess patch passes all the regression tests and is signficantly faster than the current implementation? That's correct, but I'm not sure whether things like the redirection of stdout/stderr works in all cases, or might create quirky problems in certain situations. I just can't judge the technical stability level of the stubprocess.py source. Is there any downside to using it? (Does it work on py3?) I haven't tried the wrapper under Python3 yet. But as far as I understood Eugene and Jason, it's not implemented. Another downside is that this doesn't work under Windows obviously, I don't have any infos about OSx, and Alexandre Leblot already asked me in a PM about the support for Solaris. I can't test these things, so I've no idea. ;) If not is there any reason not to send a pull request? Creating a pull request is probably not difficult, on the technical level. But should the wrapper get activated automatically under Linux, transparently for the user? Or would it be an experimental option at first, that the user can activate via a command-line option...since we aren't that sure about all the possible implications? Dirk ___ Scons-dev mailing list Scons-dev@scons.orgmailto:Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev -- Gary Closed Joint Stock Company Intel A/O Registered legal address: Krylatsky Hills Business Park, 17 Krylatskaya Str., Bldg 4, Moscow 121614, Russian Federation This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. stubprocess.py Description: stubprocess.py ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi guys, I just found this talk by Christine Spang, given at the PyCon 2014 in Montréal: http://pyvideo.org/video/2640/subprocess-to-ffi-memory-performance-and-why-y It's really worthwhile to watch, I think. However, I don't think CFFI can solve our basic problem...it looks as if there is no support for the redirection of stdout/stderr and only actual library calls can be wrapped. Does anybody know more about this? Given our current results with the stubprocess.py wrapper, I'd like to raise the question again about how to proceed from here in general. I basically have enough material to show at the EuroPython 2014 in Berlin (if my proposal gets accepted), but what's our plan for the further development with this? Your comments are welcome... Best regards, Dirk On 11.04.2014 13:55, Leskinen, Eugene wrote: Here comes the latest version of stubprocess.py. It supports Python up to version 2.7.6 from now. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Here comes the latest version of stubprocess.py. It supports Python up to version 2.7.6 from now. -Original Message- From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Kenny, Jason L Sent: Thursday, April 10, 2014 8:31 PM To: 'dl9...@darc.de'; scons-dev@scons.org Subject: Re: [Scons-dev] Subprocess issue on Linux? I think there will be one more fix coming as well. Just found that the API being wrapped as an extra argument in python 2.7.6 ( I think we tested with python 2.7.3) Jason -Original Message- From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bächle Sent: Thursday, April 10, 2014 10:44 AM To: scons-dev@scons.org Subject: Re: [Scons-dev] Subprocess issue on Linux? On 10.04.2014 08:47, Leskinen, Eugene wrote: Hi Dirk, Thank you for testing the stuff. I have re-produced and fixed the issue. Please try the latest version of stubprocess.py Thanks for the fix Eugene! All tests (runtest.py -a) are passing now on my side, while having 73 no results. Dirk -Original Message- From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bachle Sent: Wednesday, April 09, 2014 11:28 PM To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On 09.04.2014 19:24, Bill Deegan wrote: Dirk, That's pretty impressive! Does it pass the full regression suite? No, it doesn't work: ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev Closed Joint Stock Company Intel A/O Registered legal address: Krylatsky Hills Business Park, 17 Krylatskaya Str., Bldg 4, Moscow 121614, Russian Federation This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. stubprocess.py Description: stubprocess.py ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
I think there will be one more fix coming as well. Just found that the API being wrapped as an extra argument in python 2.7.6 ( I think we tested with python 2.7.3) Jason -Original Message- From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bächle Sent: Thursday, April 10, 2014 10:44 AM To: scons-dev@scons.org Subject: Re: [Scons-dev] Subprocess issue on Linux? On 10.04.2014 08:47, Leskinen, Eugene wrote: Hi Dirk, Thank you for testing the stuff. I have re-produced and fixed the issue. Please try the latest version of stubprocess.py Thanks for the fix Eugene! All tests (runtest.py -a) are passing now on my side, while having 73 no results. Dirk -Original Message- From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bachle Sent: Wednesday, April 09, 2014 11:28 PM To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On 09.04.2014 19:24, Bill Deegan wrote: Dirk, That's pretty impressive! Does it pass the full regression suite? No, it doesn't work: ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Dirk, That's pretty impressive! Does it pass the full regression suite? -Bill On Wed, Apr 9, 2014 at 10:14 AM, Dirk Bächle tshor...@gmx.de wrote: Hi all, I ran another benchmark for clean builds with the stubprocess.py imported. Please, find the results/graphs attached... Best regards, Dirk On 07.04.2014 20:39, Leskinen, Eugene wrote: Hi Dirk, Please find latest version of stubprocess.py attached. This version has fixes which make it work under Python 2.6 and correctly handle situation when executable to run does not exist or has wrong permissions. Hope you'll find it useful. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On 09.04.2014 19:24, Bill Deegan wrote: Dirk, That's pretty impressive! Does it pass the full regression suite? No, it doesn't work: 501/1110 (45.14%) /usr/bin/python -tt test/LEX/live.py /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py returned 2 STDOUT = scons: Reading SConscript files ... STDERR = KeyError: 'PATH': File /tmp/testcmd.4854.Zu6ZsJ/SConstruct, line 2: foo = Environment() File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py, line 1003: apply_tools(self, tools, toolpath) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py, line 107: env.Tool(tool) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py, line 1787: tool(self) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py, line 183: self.generate(env, *args, **kw) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/default.py, line 40: for t in SCons.Tool.tool_list(env['PLATFORM'], env): File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py, line 819: ], env) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py, line 690: return list(filter (ToolExists, tools)) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py, line 689: return Tool(tool).exists(env) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/wix.py, line 71: for path in os.environ['PATH'].split(os.pathsep): File /usr/lib/python2.7/UserDict.py, line 23: raise KeyError(key) FAILED test of /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py Looks like the wrapping of Subprocess.Popen in stubprocess.py prevents the os.environ settings to get through, so all tests that setup a simple Environment and auto-detect Tools are bound to fail. :( Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Dirk, Is this available in your bitbucket repo? (URL?) -Bill On April 9, 2014 at 12:35:52 PM, Kenny, Jason L (jason.l.ke...@intel.com) wrote: There has to be a little more to this as Part only does auto detection of tools, and this is working fine in Parts from our tests. ( and the fact the builds did not utterly break because it could not find a tool) Jason -Original Message- From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bächle Sent: Wednesday, April 09, 2014 2:28 PM To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On 09.04.2014 19:24, Bill Deegan wrote: Dirk, That's pretty impressive! Does it pass the full regression suite? No, it doesn't work: 501/1110 (45.14%) /usr/bin/python -tt test/LEX/live.py /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py returned 2 STDOUT = scons: Reading SConscript files ... STDERR = KeyError: 'PATH': File /tmp/testcmd.4854.Zu6ZsJ/SConstruct, line 2: foo = Environment() File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py, line 1003: apply_tools(self, tools, toolpath) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py, line 107: env.Tool(tool) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py, line 1787: tool(self) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py, line 183: self.generate(env, *args, **kw) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/default.py, line 40: for t in SCons.Tool.tool_list(env['PLATFORM'], env): File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py, line 819: ], env) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py, line 690: return list(filter (ToolExists, tools)) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py, line 689: return Tool(tool).exists(env) File /home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/wix.py, line 71: for path in os.environ['PATH'].split(os.pathsep): File /usr/lib/python2.7/UserDict.py, line 23: raise KeyError(key) FAILED test of /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py Looks like the wrapping of Subprocess.Popen in stubprocess.py prevents the os.environ settings to get through, so all tests that setup a simple Environment and auto-detect Tools are bound to fail. :( Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On 09.04.2014 23:56, William Deegan wrote: Dirk, Is this available in your bitbucket repo? (URL?) No, I just used the subprocess.py from Eugene's mail and put it in my local copy of the SCons source tree (plus the import in posix.py). I didn't bother to add it to any repo yet... Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi Dirk, Please find latest version of stubprocess.py attached. This version has fixes which make it work under Python 2.6 and correctly handle situation when executable to run does not exist or has wrong permissions. Hope you'll find it useful. From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bachle Sent: Sunday, April 06, 2014 6:02 PM To: scons-dev@scons.org Subject: Re: [Scons-dev] Subprocess issue on Linux? Hi Eugene, thanks a lot for your quick answer and very helpful advice. On 06.04.2014 15:21, Leskinen, Eugene wrote: I have just place the stubprocess.py to /opt/python27/lib/scons-2.1.0/SCons/Platform/ directory and added 'import stubprocess' statement to SCons.Platfrom.posix module just after import subprocess: --- /opt/python27/lib/scons-2.1.0/SCons/Platform/posix.py 2014-04-06 17:17:30.0 +0400 +++ /opt/python27/lib/scons-2.1.0/SCons/Platform/posix.py.new 2014-04-06 17:17:07.0 +0400 @@ -36,6 +36,7 @@ import os import os.path import subprocess +import stubprocess import sys import select This what worked for me. This approach works on my side too. I had tried to copy-paste part of the source code directly to the top of SCons/Platform/posix.py, which gave me the reported errors. Best regards, Dirk Closed Joint Stock Company Intel A/O Registered legal address: Krylatsky Hills Business Park, 17 Krylatskaya Str., Bldg 4, Moscow 121614, Russian Federation This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. stubprocess.py Description: stubprocess.py ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi Jason, On 05.04.2014 00:17, Kenny, Jason L wrote: I think yes, in that it does what should be done by the system under posix_spawn.. ie call vfork and execve. Here is the last version of the monkey patch I have from the people working on it. It has a fallback to the classic fork exec if the API's don't exists. It seems to solve the main speed problem for us at the moment. I believe it still being tested to find possible issues. I tried to let SCons run with this patch, but I'm not able to integrate the patching of subprocess.Popen() successfully. I always get an import error for SCons.Util, and fail to see where this is actually coming from: scons: Reading SConscript files ... scons: done reading SConscript files. scons: Building targets ... gcc -o d1_0/f0_sconsbld_d1_0.o -c -Id1_0/lup000_sconsbld_d1_0 -Id1_0/lup001_sconsbld_d1_0 d1_0/f0_sconsbld_d1_0.c Traceback (most recent call last): File /home/dirk/workspace/scons_dirkbaechle/src/engine/SCons/Platform/posix.py, line 42, in module import SCons.Util ImportError: No module named SCons.Util scons: *** [d1_0/f0_sconsbld_d1_0.o] Error 1 scons: building terminated because of errors. Does anybody have a clue and can give me a pointer? I'd like to compare the times for a clean build against the default sources. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi Dirk, I am Eugene Leskinen and I am the stubprocess.py module author. How do you import it? From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bachle Sent: Sunday, April 06, 2014 4:59 PM To: scons-dev@scons.org Subject: Re: [Scons-dev] Subprocess issue on Linux? Hi Jason, On 05.04.2014 00:17, Kenny, Jason L wrote: I think yes, in that it does what should be done by the system under posix_spawn.. ie call vfork and execve. Here is the last version of the monkey patch I have from the people working on it. It has a fallback to the classic fork exec if the API's don't exists. It seems to solve the main speed problem for us at the moment. I believe it still being tested to find possible issues. I tried to let SCons run with this patch, but I'm not able to integrate the patching of subprocess.Popen() successfully. I always get an import error for SCons.Util, and fail to see where this is actually coming from: scons: Reading SConscript files ... scons: done reading SConscript files. scons: Building targets ... gcc -o d1_0/f0_sconsbld_d1_0.o -c -Id1_0/lup000_sconsbld_d1_0 -Id1_0/lup001_sconsbld_d1_0 d1_0/f0_sconsbld_d1_0.c Traceback (most recent call last): File /home/dirk/workspace/scons_dirkbaechle/src/engine/SCons/Platform/posix.py, line 42, in module import SCons.Util ImportError: No module named SCons.Util scons: *** [d1_0/f0_sconsbld_d1_0.o] Error 1 scons: building terminated because of errors. Does anybody have a clue and can give me a pointer? I'd like to compare the times for a clean build against the default sources. Regards, Dirk Closed Joint Stock Company Intel A/O Registered legal address: Krylatsky Hills Business Park, 17 Krylatskaya Str., Bldg 4, Moscow 121614, Russian Federation This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
I have just place the stubprocess.py to /opt/python27/lib/scons-2.1.0/SCons/Platform/ directory and added 'import stubprocess' statement to SCons.Platfrom.posix module just after import subprocess: --- /opt/python27/lib/scons-2.1.0/SCons/Platform/posix.py 2014-04-06 17:17:30.0 +0400 +++ /opt/python27/lib/scons-2.1.0/SCons/Platform/posix.py.new 2014-04-06 17:17:07.0 +0400 @@ -36,6 +36,7 @@ import os import os.path import subprocess +import stubprocess import sys import select This what worked for me. From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Leskinen, Eugene Sent: Sunday, April 06, 2014 5:05 PM To: dl9...@darc.de; SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? Hi Dirk, I am Eugene Leskinen and I am the stubprocess.py module author. How do you import it? From: scons-dev-boun...@scons.orgmailto:scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bachle Sent: Sunday, April 06, 2014 4:59 PM To: scons-dev@scons.orgmailto:scons-dev@scons.org Subject: Re: [Scons-dev] Subprocess issue on Linux? Hi Jason, On 05.04.2014 00:17, Kenny, Jason L wrote: I think yes, in that it does what should be done by the system under posix_spawn.. ie call vfork and execve. Here is the last version of the monkey patch I have from the people working on it. It has a fallback to the classic fork exec if the API's don't exists. It seems to solve the main speed problem for us at the moment. I believe it still being tested to find possible issues. I tried to let SCons run with this patch, but I'm not able to integrate the patching of subprocess.Popen() successfully. I always get an import error for SCons.Util, and fail to see where this is actually coming from: scons: Reading SConscript files ... scons: done reading SConscript files. scons: Building targets ... gcc -o d1_0/f0_sconsbld_d1_0.o -c -Id1_0/lup000_sconsbld_d1_0 -Id1_0/lup001_sconsbld_d1_0 d1_0/f0_sconsbld_d1_0.c Traceback (most recent call last): File /home/dirk/workspace/scons_dirkbaechle/src/engine/SCons/Platform/posix.py, line 42, in module import SCons.Util ImportError: No module named SCons.Util scons: *** [d1_0/f0_sconsbld_d1_0.o] Error 1 scons: building terminated because of errors. Does anybody have a clue and can give me a pointer? I'd like to compare the times for a clean build against the default sources. Regards, Dirk Closed Joint Stock Company Intel A/O Registered legal address: Krylatsky Hills Business Park, 17 Krylatskaya Str., Bldg 4, Moscow 121614, Russian Federation This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. Closed Joint Stock Company Intel A/O Registered legal address: Krylatsky Hills Business Park, 17 Krylatskaya Str., Bldg 4, Moscow 121614, Russian Federation This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi Eugene, thanks a lot for your quick answer and very helpful advice. On 06.04.2014 15:21, Leskinen, Eugene wrote: I have just place the stubprocess.py to /opt/python27/lib/scons-2.1.0/SCons/Platform/ directory and added 'import stubprocess' statement to SCons.Platfrom.posix module just after import subprocess: --- /opt/python27/lib/scons-2.1.0/SCons/Platform/posix.py 2014-04-06 17:17:30.0 +0400 +++ /opt/python27/lib/scons-2.1.0/SCons/Platform/posix.py.new 2014-04-06 17:17:07.0 +0400 @@ -36,6 +36,7 @@ import os import os.path import subprocess +import stubprocess import sys import select This what worked for me. This approach works on my side too. I had tried to copy-paste part of the source code directly to the top of SCons/Platform/posix.py, which gave me the reported errors. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On 02.04.2014 23:38, Gary Oberbrunner wrote: On Wed, Apr 2, 2014 at 4:51 PM, Dirk Bächle tshor...@gmx.de mailto:tshor...@gmx.de wrote: This idea may be feasible, but I'd rather try to get the actual shell spawning to be as fast as possible. We have some valid approaches for this, so let's try them out...maybe one of them is fast enough, such that we don't have to care about the extra work mentioned above anymore. Speeding up the spawn/fork stuff would be more transparent to the user than trying to detect which commands need a full shell and which don't. They're orthogonal. Both useful, but either can be pursued independently of the other. Avoiding shells will be most valuable in builds with lots of tiny commands (could halve the build time). Avoiding fork is most valuable when SCons is using lots of memory (which it often is). My sense, based largely on Dirk's research, is fixing spawn has a bigger payoff right now. Since we now know that the fork problem is something to care about, I'd really like to use the current momentum. I don't want to push anybody, I want us to have a clear vision about how to take steps in the right direction. Is someone working on this right now? If not, who volunteers? Let's just communicate...and be on the same page about knowing *who* does *what*, and *when*. What's a little behind this is, that I think this point has some strategic importance. It's a single issue that keeps a lot of users from switching to SCons. By getting it out of the way, we can present (and kind of sell) our project much better in any media out there. I have submitted a proposal for the poster session at the EuroPython 2014 in Berlin. It's title is How spawning many processes sequentially can kill performance, and if I could not only talk about our problem, but also present a solution that has some benefit for the Python community, that definitely carries some potential in my opinion. So let's really get this crackin'! ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi dirk, While I don't have a patch to SCons at the moment we have people in my team fixing this with Parts as a monkey path to sub process module. I asked the two people working on this to speak up and update their patch work online for SCons to consider. I think from my point of view there is a need to make sure we can continue to pipe output correctly. This is critical to the Parts addon to SCons as it used to do coloring, logging and to sync the output on -J based builds. I am not sure but seems to me that we need a better Spawn function. From the view of Parts and internal need at Intel we are doing a monkey patch to get a fix quickly, but I think we need a better, more stable fix. I think we want to consider a few different items, vs a quick rush fix, as there are issues Anatoly techtonik is looking at with input from me with streaming and handing of the spawned tasks output from a Scons point of view that currently is a bit of a mess. This is important to SCons as it would be good to have a clear way to spawn stuff in SCons from the user point of view. I whack the SPAWN variable in SCons to do this myself and fill in details to make it work with Parts. It would be better if we had a clear function to call to do a subprocess in SCons that I could call instead, that allowed for control over how we spawned, output of the task, and control over killing the task. Given what had to be done in Parts to get this stuff to work, I think Scons needs to take some ownership of this for the user, and not say implement your own SPAWN callback in Scons is the good enough answer. Jason From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bächle Sent: Friday, April 04, 2014 3:54 PM To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On 02.04.2014 23:38, Gary Oberbrunner wrote: On Wed, Apr 2, 2014 at 4:51 PM, Dirk Bächle tshor...@gmx.demailto:tshor...@gmx.de wrote: This idea may be feasible, but I'd rather try to get the actual shell spawning to be as fast as possible. We have some valid approaches for this, so let's try them out...maybe one of them is fast enough, such that we don't have to care about the extra work mentioned above anymore. Speeding up the spawn/fork stuff would be more transparent to the user than trying to detect which commands need a full shell and which don't. They're orthogonal. Both useful, but either can be pursued independently of the other. Avoiding shells will be most valuable in builds with lots of tiny commands (could halve the build time). Avoiding fork is most valuable when SCons is using lots of memory (which it often is). My sense, based largely on Dirk's research, is fixing spawn has a bigger payoff right now. Since we now know that the fork problem is something to care about, I'd really like to use the current momentum. I don't want to push anybody, I want us to have a clear vision about how to take steps in the right direction. Is someone working on this right now? If not, who volunteers? Let's just communicate...and be on the same page about knowing *who* does *what*, and *when*. What's a little behind this is, that I think this point has some strategic importance. It's a single issue that keeps a lot of users from switching to SCons. By getting it out of the way, we can present (and kind of sell) our project much better in any media out there. I have submitted a proposal for the poster session at the EuroPython 2014 in Berlin. It's title is How spawning many processes sequentially can kill performance, and if I could not only talk about our problem, but also present a solution that has some benefit for the Python community, that definitely carries some potential in my opinion. So let's really get this crackin'! ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On Fri, Apr 4, 2014 at 5:32 PM, Kenny, Jason L jason.l.ke...@intel.comwrote: Hi dirk, While I don't have a patch to SCons at the moment we have people in my team fixing this with Parts as a monkey path to sub process module. I asked the two people working on this to speak up and update their patch work online for SCons to consider. Jason, do you have a way to do posix_spawn from python (without a C extension)? -- Gary ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
I think yes, in that it does what should be done by the system under posix_spawn.. ie call vfork and execve. Here is the last version of the monkey patch I have from the people working on it. It has a fallback to the classic fork exec if the API's don't exists. It seems to solve the main speed problem for us at the moment. I believe it still being tested to find possible issues. Jason From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Gary Oberbrunner Sent: Friday, April 04, 2014 4:46 PM To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On Fri, Apr 4, 2014 at 5:32 PM, Kenny, Jason L jason.l.ke...@intel.commailto:jason.l.ke...@intel.com wrote: Hi dirk, While I don't have a patch to SCons at the moment we have people in my team fixing this with Parts as a monkey path to sub process module. I asked the two people working on this to speak up and update their patch work online for SCons to consider. Jason, do you have a way to do posix_spawn from python (without a C extension)? -- Gary stubprocess.py Description: stubprocess.py ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
I think the test round about line 144 needs to be a little different. For instance, aix and solaris (at least the versions we use at work) claim to support posix_spawn and the man pages are identical, but they don't identify themselves as linux One other question that comes to mind from time to time: do we /really/ need to spawn a shell just to execute a command. Obviously if it include '' and other shell specials, you should. I'd really like to make it do something like perl: If you pass an array [ 'prog', '$TARGET', '$SOURCE' ] it doesn't execute a shell. Otherwise (passing a single string), it will examine for special characters, and if none are found it will split on white space and execute that, or it will pass that line to the shell. NB Yes, I realise this will potentially break things, but why execute a shell if you don't have to? - Original Message - From: scons-dev@scons.org To: scons-dev@scons.org At: Apr 1 2014 17:13:48 I've found posix spawn can be much faster than fork/exec with large memory processes, so I'd be in favor of this. Not every system has it though so there would have to be a fallback to fork/exec. -- Gary Oberbrunner (sent from my Android) On Apr 1, 2014 11:52 AM, Kenny, Jason L jason.l.ke...@intel.com wrote: Hi guys, I just got a patch to Parts internal at Intel to fix some issues with subprocess. I find myself sort of surprised by this, and honestly felt that this seems to be an issue that should be looked at in Scons as well. The problem is this. We have been building a given huge product here at Intel on RHEL 4.8. IT is old and finally time has come to start moving to a newer system. What was found when we moved to a newer RHEL 5.10 ( I know bleeding edge J ) we found that the build was twice as slow. Looking into it more it was found that it was the use of fork() in subprocess. It was found that posix_spawn() fixed this. Attached is a monkey patch that is up for internal review to modify subprocess module a little to fix this for Unix based systems. Now I am not an expert on Linux internal details. However I thought fork() was the way process got spawned on Linux. Ie fork() was the win32 CreateProcess(). I am clearly wrong, but this seems to be a problem I would not expect to have seen on Linux. The root of the problem seems to be that the system takes a long time to setup the fork when you have a large build ( and we have a very large build ) because of the memory usage (2-6GB) needed by SCons alone. Using a different method to create a process that does not try to “clone” memory seems to be the fix that gets the build times to be about ~2x+ faster. Does anyone else have input on this? IF this is a good patch, it seem that we will want to apply it to SCons, for a speed boost. Thanks Jason Also I attached a test file that to show fork() is the problem. You can see the difference importing the stubprocess.py file and running the test again. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On Wed, Apr 2, 2014 at 11:16 AM, Tom Tanner (BLOOMBERG/ LONDON) ttann...@bloomberg.net wrote: NB Yes, I realise this will potentially break things, but why execute a shell if you don't have to? Fair point. I was once a Windoze user, who didn't realize what shell is and just want things to just work (like PATH etc). I'd really like to make it do something like perl: If you pass an array [ 'prog', '$TARGET', '$SOURCE' ] it doesn't execute a shell. Otherwise (passing a single string), it will examine for special characters, and if none are found it will split on white space and execute that, or it will pass that line to the shell. Magic is good. Mystery should be banned. Rule of thumb for a good code is calculating chances. What is the chance that somebody no familiar with semantics will get what exactly code does without reading the docs? If the value is below 80% - drop it. If you can't drop it - write a comment. If the comment is too huge - find an issue on the internet and link to it. -- anatoly t. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On Wed, Apr 2, 2014 at 4:16 AM, Tom Tanner (BLOOMBERG/ LONDON) ttann...@bloomberg.net wrote: If you pass an array [ 'prog', '$TARGET', '$SOURCE' ] it doesn't execute a shell. Otherwise (passing a single string), it will examine for special characters, and if none are found it will split on white space and execute that, or it will pass that line to the shell. NB Yes, I realise this will potentially break things, but why execute a shell if you don't have to? I don't think this would break anything. If the command has no shell metachars (and we can be quite conservative about that), then executing it directly vs. via a shell will be exactly the same, except for not needing intervening shell process. In fact I thought SCons did this, many years ago. I must be thinking of Perl though. :-) This technique can also be used if a list is passed in; just check each word for shell metachars. -- Gary ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
I think we have had a number patches that tried to remove the shell form the spawned command on linux and windows. I am personally for it, given we have some way to do it when we do need the shell. Anatoly? Do you think this might be a good place to tweak up your subprocess code? I would be happy to try it out in Parts as a replacement to the normal subprocess. If it can pass the needs we have for our build at Intel, I think it would help validate what you are trying to do with it. I know that if there are issues we could work on fixing it easily enough. Jason From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Gary Oberbrunner Sent: Wednesday, April 02, 2014 9:32 AM To: Tom Tanner; SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On Wed, Apr 2, 2014 at 4:16 AM, Tom Tanner (BLOOMBERG/ LONDON) ttann...@bloomberg.netmailto:ttann...@bloomberg.net wrote: If you pass an array [ 'prog', '$TARGET', '$SOURCE' ] it doesn't execute a shell. Otherwise (passing a single string), it will examine for special characters, and if none are found it will split on white space and execute that, or it will pass that line to the shell. NB Yes, I realise this will potentially break things, but why execute a shell if you don't have to? I don't think this would break anything. If the command has no shell metachars (and we can be quite conservative about that), then executing it directly vs. via a shell will be exactly the same, except for not needing intervening shell process. In fact I thought SCons did this, many years ago. I must be thinking of Perl though. :-) This technique can also be used if a list is passed in; just check each word for shell metachars. -- Gary ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On Wed, Apr 2, 2014 at 4:51 PM, Dirk Bächle tshor...@gmx.de wrote: This idea may be feasible, but I'd rather try to get the actual shell spawning to be as fast as possible. We have some valid approaches for this, so let's try them out...maybe one of them is fast enough, such that we don't have to care about the extra work mentioned above anymore. Speeding up the spawn/fork stuff would be more transparent to the user than trying to detect which commands need a full shell and which don't. They're orthogonal. Both useful, but either can be pursued independently of the other. Avoiding shells will be most valuable in builds with lots of tiny commands (could halve the build time). Avoiding fork is most valuable when SCons is using lots of memory (which it often is). My sense, based largely on Dirk's research, is fixing spawn has a bigger payoff right now. -- Gary ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi there, On 01.04.2014 18:13, Gary Oberbrunner wrote: I've found posix spawn can be much faster than fork/exec with large memory processes, so I'd be in favor of this. Not every system has it though so there would have to be a fallback to fork/exec. -- Gary Oberbrunner (sent from my Android) On Apr 1, 2014 11:52 AM, Kenny, Jason L jason.l.ke...@intel.com mailto:jason.l.ke...@intel.com wrote: Hi guys, I just got a patch to Parts internal at Intel to fix some issues with subprocess. I find myself sort of surprised by this, and honestly felt that this seems to be an issue that should be looked at in Scons as well. [...] Does anyone else have input on this? IF this is a good patch, it seem that we will want to apply it to SCons, for a speed boost. a few minutes ago I updated the page http://www.scons.org/wiki/WhySconsIsNotSlow with my latest results. Check out the bottom of the page, and the repository http://www.bitbucket.org/dirkbaechle/scons_testresults for all the results. I'm also talking to Tzvetan Mikov off-list, he volunteered to write a CPython extension for posix_spawn() to help SCons out. I'll try to get him subscribed to this list, so the three of you can talk some more about this, okay? Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
