[Scons-dev] Subprocess issue on Linux?

Hi guys, I just got a patch to Parts internal at Intel to fix some issues with subprocess. I find myself sort of surprised by this, and honestly felt that this seems to be an issue that should be looked at in Scons as well. The problem is this. We have been building a given huge product here a

Re: [Scons-dev] Subprocess issue on Linux?

I've found posix spawn can be much faster than fork/exec with large memory processes, so I'd be in favor of this. Not every system has it though so there would have to be a fallback to fork/exec. -- Gary Oberbrunner (sent from my Android) On Apr 1, 2014 11:52 AM, "Kenny, Jason L" wrote: > > > H

Re: [Scons-dev] Subprocess issue on Linux?

Hi there, On 01.04.2014 18:13, Gary Oberbrunner wrote: I've found posix spawn can be much faster than fork/exec with large memory processes, so I'd be in favor of this. Not every system has it though so there would have to be a fallback to fork/exec. -- Gary Oberbrunner (sent from my Androi

Re: [Scons-dev] Subprocess issue on Linux?

I think the test round about line 144 needs to be a little different. For instance, aix and solaris (at least the versions we use at work) claim to support posix_spawn and the man pages are identical, but they don't identify themselves as linux One other question that comes to mind from time to

Re: [Scons-dev] Subprocess issue on Linux?

On Wed, Apr 2, 2014 at 11:16 AM, Tom Tanner (BLOOMBERG/ LONDON) wrote: > NB Yes, I realise this will potentially break things, but why execute a > shell if you don't have to? Fair point. I was once a Windoze user, who didn't realize what shell is and just want things to just work (like PATH etc).

Re: [Scons-dev] Subprocess issue on Linux?

On Wed, Apr 2, 2014 at 4:16 AM, Tom Tanner (BLOOMBERG/ LONDON) < ttann...@bloomberg.net> wrote: > If you pass an array [ 'prog', '$TARGET', '$SOURCE' ] it doesn't execute a > shell. Otherwise (passing a single string), it will examine for special > characters, and if none are found it will split o

Re: [Scons-dev] Subprocess issue on Linux?

...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Gary Oberbrunner Sent: Wednesday, April 02, 2014 9:32 AM To: Tom Tanner; SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On Wed, Apr 2, 2014 at 4:16 AM, Tom Tanner (BLOOMBERG/ LONDON) mailto:ttann

Re: [Scons-dev] Subprocess issue on Linux?

On 02.04.2014 10:16, Tom Tanner (BLOOMBERG/ LONDON) wrote: I think the test round about line 144 needs to be a little different. For instance, aix and solaris (at least the versions we use at work) claim to support posix_spawn and the man pages are identical, but they don't identify themselves

Re: [Scons-dev] Subprocess issue on Linux?

On Wed, Apr 2, 2014 at 4:51 PM, Dirk Bächle wrote: > This idea may be feasible, but I'd rather try to get the actual shell > spawning to be as fast as possible. We have some valid approaches for this, > so let's try them out...maybe one of them is fast enough, such that we > don't have to care ab

Re: [Scons-dev] Subprocess issue on Linux?

On 02.04.2014 23:38, Gary Oberbrunner wrote: On Wed, Apr 2, 2014 at 4:51 PM, Dirk Bächle > wrote: This idea may be feasible, but I'd rather try to get the actual shell spawning to be as fast as possible. We have some valid approaches for this, so let's try t

Re: [Scons-dev] Subprocess issue on Linux?

t: Re: [Scons-dev] Subprocess issue on Linux? On 02.04.2014 23:38, Gary Oberbrunner wrote: On Wed, Apr 2, 2014 at 4:51 PM, Dirk Bächle mailto:tshor...@gmx.de>> wrote: This idea may be feasible, but I'd rather try to get the actual shell spawning to be as fast as possible. We have some

Re: [Scons-dev] Subprocess issue on Linux?

On Fri, Apr 4, 2014 at 5:32 PM, Kenny, Jason L wrote: > Hi dirk, > > > > While I don't have a patch to SCons at the moment we have people in my > team fixing this with Parts as a monkey path to sub process module. I asked > the two people working on this to speak up and update their patch work >

Re: [Scons-dev] Subprocess issue on Linux?

main speed problem for us at the moment. I believe it still being tested to find possible issues. Jason From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Gary Oberbrunner Sent: Friday, April 04, 2014 4:46 PM To: SCons developer list Subject: Re: [Scons-dev]

Re: [Scons-dev] Subprocess issue on Linux?

On 05.04.2014 00:17, Kenny, Jason L wrote: I think yes, in that it does what should be done by the system under posix_spawn.. ie call vfork and execve. Here is the last version of the monkey patch I have from the people working on it. It has a fallback to the classic fork exec if the API's

Re: [Scons-dev] Subprocess issue on Linux?

Hi Jason, On 05.04.2014 00:17, Kenny, Jason L wrote: I think yes, in that it does what should be done by the system under posix_spawn.. ie call vfork and execve. Here is the last version of the monkey patch I have from the people working on it. It has a fallback to the classic fork exec if

Re: [Scons-dev] Subprocess issue on Linux?

Hi Dirk, I am Eugene Leskinen and I am the stubprocess.py module author. How do you import it? From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bachle Sent: Sunday, April 06, 2014 4:59 PM To: scons-dev@scons.org Subject: Re: [Scons-dev] Subprocess issue

Re: [Scons-dev] Subprocess issue on Linux?

s-dev-boun...@scons.org] On Behalf Of Leskinen, Eugene Sent: Sunday, April 06, 2014 5:05 PM To: dl9...@darc.de; SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? Hi Dirk, I am Eugene Leskinen and I am the stubprocess.py module author. How do you import it? From: scon

Re: [Scons-dev] Subprocess issue on Linux?

Hi Eugene, thanks a lot for your quick answer and very helpful advice. On 06.04.2014 15:21, Leskinen, Eugene wrote: I have just place the stubprocess.py to /opt/python27/lib/scons-2.1.0/SCons/Platform/ directory and added 'import stubprocess' statement to SCons.Platfrom.posix module just af

Re: [Scons-dev] Subprocess issue on Linux?

scons-dev-boun...@scons.org] On Behalf Of Dirk Bachle Sent: Sunday, April 06, 2014 6:02 PM To: scons-dev@scons.org Subject: Re: [Scons-dev] Subprocess issue on Linux? Hi Eugene, thanks a lot for your quick answer and very helpful advice. On 06.04.2014 15:21, Leskinen, Eugene wrote: I have just

Re: [Scons-dev] Subprocess issue on Linux?

Dirk, That's pretty impressive! Does it pass the full regression suite? -Bill On Wed, Apr 9, 2014 at 10:14 AM, Dirk Bächle wrote: > Hi all, > > I ran another benchmark for clean builds with the stubprocess.py imported. > Please, find the results/graphs attached... > > Best regards, > > Dirk

Re: [Scons-dev] Subprocess issue on Linux?

Wow! -- Gary Oberbrunner (sent from my Android) On Apr 9, 2014 10:14 AM, "Dirk Bächle" wrote: > Hi all, > > I ran another benchmark for clean builds with the stubprocess.py imported. > Please, find the results/graphs attached... > > Best regards, > > Dirk > > On 07.04.2014 20:39, Leskinen, Eug

Re: [Scons-dev] Subprocess issue on Linux?

On 09.04.2014 19:24, Bill Deegan wrote: Dirk, That's pretty impressive! Does it pass the full regression suite? Haven't tried yethang on a few seconds. ;) Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listi

Re: [Scons-dev] Subprocess issue on Linux?

On 09.04.2014 19:24, Bill Deegan wrote: Dirk, That's pretty impressive! Does it pass the full regression suite? No, it doesn't work: 501/1110 (45.14%) /usr/bin/python -tt test/LEX/live.py /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py returned 2 STDOUT ==

Re: [Scons-dev] Subprocess issue on Linux?

-dev-boun...@scons.org] On Behalf Of Dirk Bächle Sent: Wednesday, April 09, 2014 2:28 PM To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On 09.04.2014 19:24, Bill Deegan wrote: > Dirk, > > That's pretty impressive! > Does it pass the full regression

Re: [Scons-dev] Subprocess issue on Linux?

2014 2:28 PM > To: SCons developer list > Subject: Re: [Scons-dev] Subprocess issue on Linux? > > On 09.04.2014 19:24, Bill Deegan wrote: > > Dirk, > > > > That's pretty impressive! > > Does it pass the full regression suite? > > > > No, it doesn&

Re: [Scons-dev] Subprocess issue on Linux?

On 09.04.2014 23:56, William Deegan wrote: Dirk, Is this available in your bitbucket repo? (URL?) No, I just used the subprocess.py from Eugene's mail and put it in my local copy of the SCons source tree (plus the import in posix.py). I didn't bother to add it to any repo yet... Dirk _

Re: [Scons-dev] Subprocess issue on Linux?

To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On 09.04.2014 19:24, Bill Deegan wrote: > Dirk, > > That's pretty impressive! > Does it pass the full regression suite? > No, it doesn't work: 501/1110 (45.14%) /usr/bin/python -tt test/LEX/live

Re: [Scons-dev] Subprocess issue on Linux?

quot;no results". Dirk -Original Message- From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bachle Sent: Wednesday, April 09, 2014 11:28 PM To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On 09.04.2014 19:24, Bill

Re: [Scons-dev] Subprocess issue on Linux?

Sent: Thursday, April 10, 2014 10:44 AM To: scons-dev@scons.org Subject: Re: [Scons-dev] Subprocess issue on Linux? On 10.04.2014 08:47, Leskinen, Eugene wrote: > Hi Dirk, > > Thank you for testing the stuff. I have re-produced and fixed the > issue. Please try the lates

Re: [Scons-dev] Subprocess issue on Linux?

scons-dev@scons.org Subject: Re: [Scons-dev] Subprocess issue on Linux? I think there will be one more fix coming as well. Just found that the API being wrapped as an extra argument in python 2.7.6 ( I think we tested with python 2.7.3) Jason -Original Message- From: scons-dev-boun...

Re: [Scons-dev] Subprocess issue on Linux?

Hi guys, I just found this talk by Christine Spang, given at the PyCon 2014 in Montréal: http://pyvideo.org/video/2640/subprocess-to-ffi-memory-performance-and-why-y It's really worthwhile to watch, I think. However, I don't think CFFI can solve our basic problem...it looks as if there is no

Re: [Scons-dev] Subprocess issue on Linux?

Dirk, So if I understand correctly, the stubprocess patch passes all the regression tests and is signficantly faster than the current implementation? Is there any downside to using it? (Does it work on py3?) If not is there any reason not to send a pull request? -Bill On Tue, Apr 22, 2014 at 3

Re: [Scons-dev] Subprocess issue on Linux?

On 23.04.2014 01:42, Bill Deegan wrote: Dirk, So if I understand correctly, the stubprocess patch passes all the regression tests and is signficantly faster than the current implementation? That's correct, but I'm not sure whether things like the redirection of stdout/stderr works in all ca

Re: [Scons-dev] Subprocess issue on Linux?

I can try it on OSX, and review the stubprocess.py source (which I haven't yet). I don't think we need to worry about Windows; it has its own process creation methods with their own characteristics and bugs. On Wed, Apr 23, 2014 at 3:07 AM, Dirk Bächle wrote: > On 23.04.2014 01:42, Bill Deegan

Re: [Scons-dev] Subprocess issue on Linux?

[mailto:scons-dev-boun...@scons.org] On Behalf Of Gary Oberbrunner Sent: Wednesday, April 23, 2014 3:16 PM To: Dirk Baechle; SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? I can try it on OSX, and review the stubprocess.py source (which I haven't yet). I don't think

Re: [Scons-dev] Subprocess issue on Linux?

probably anyone will have any problem at all. Jason -Original Message- From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bächle Sent: Wednesday, April 23, 2014 2:08 AM To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On 23

Re: [Scons-dev] Subprocess issue on Linux?

I have no problems with stubprocess.py on my Mac. I ran the test suite and my day-job project with 32 and 64-bit python 2.7, and all seems well. -- Gary ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev