Re: [Scons-dev] Subprocess issue on Linux?
On 02.04.2014 23:38, Gary Oberbrunner wrote: On Wed, Apr 2, 2014 at 4:51 PM, Dirk Bächle tshor...@gmx.de mailto:tshor...@gmx.de wrote: This idea may be feasible, but I'd rather try to get the actual shell spawning to be as fast as possible. We have some valid approaches for this, so let's try them out...maybe one of them is fast enough, such that we don't have to care about the extra work mentioned above anymore. Speeding up the spawn/fork stuff would be more transparent to the user than trying to detect which commands need a full shell and which don't. They're orthogonal. Both useful, but either can be pursued independently of the other. Avoiding shells will be most valuable in builds with lots of tiny commands (could halve the build time). Avoiding fork is most valuable when SCons is using lots of memory (which it often is). My sense, based largely on Dirk's research, is fixing spawn has a bigger payoff right now. Since we now know that the fork problem is something to care about, I'd really like to use the current momentum. I don't want to push anybody, I want us to have a clear vision about how to take steps in the right direction. Is someone working on this right now? If not, who volunteers? Let's just communicate...and be on the same page about knowing *who* does *what*, and *when*. What's a little behind this is, that I think this point has some strategic importance. It's a single issue that keeps a lot of users from switching to SCons. By getting it out of the way, we can present (and kind of sell) our project much better in any media out there. I have submitted a proposal for the poster session at the EuroPython 2014 in Berlin. It's title is How spawning many processes sequentially can kill performance, and if I could not only talk about our problem, but also present a solution that has some benefit for the Python community, that definitely carries some potential in my opinion. So let's really get this crackin'! ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Hi dirk, While I don't have a patch to SCons at the moment we have people in my team fixing this with Parts as a monkey path to sub process module. I asked the two people working on this to speak up and update their patch work online for SCons to consider. I think from my point of view there is a need to make sure we can continue to pipe output correctly. This is critical to the Parts addon to SCons as it used to do coloring, logging and to sync the output on -J based builds. I am not sure but seems to me that we need a better Spawn function. From the view of Parts and internal need at Intel we are doing a monkey patch to get a fix quickly, but I think we need a better, more stable fix. I think we want to consider a few different items, vs a quick rush fix, as there are issues Anatoly techtonik is looking at with input from me with streaming and handing of the spawned tasks output from a Scons point of view that currently is a bit of a mess. This is important to SCons as it would be good to have a clear way to spawn stuff in SCons from the user point of view. I whack the SPAWN variable in SCons to do this myself and fill in details to make it work with Parts. It would be better if we had a clear function to call to do a subprocess in SCons that I could call instead, that allowed for control over how we spawned, output of the task, and control over killing the task. Given what had to be done in Parts to get this stuff to work, I think Scons needs to take some ownership of this for the user, and not say implement your own SPAWN callback in Scons is the good enough answer. Jason From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bächle Sent: Friday, April 04, 2014 3:54 PM To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On 02.04.2014 23:38, Gary Oberbrunner wrote: On Wed, Apr 2, 2014 at 4:51 PM, Dirk Bächle tshor...@gmx.demailto:tshor...@gmx.de wrote: This idea may be feasible, but I'd rather try to get the actual shell spawning to be as fast as possible. We have some valid approaches for this, so let's try them out...maybe one of them is fast enough, such that we don't have to care about the extra work mentioned above anymore. Speeding up the spawn/fork stuff would be more transparent to the user than trying to detect which commands need a full shell and which don't. They're orthogonal. Both useful, but either can be pursued independently of the other. Avoiding shells will be most valuable in builds with lots of tiny commands (could halve the build time). Avoiding fork is most valuable when SCons is using lots of memory (which it often is). My sense, based largely on Dirk's research, is fixing spawn has a bigger payoff right now. Since we now know that the fork problem is something to care about, I'd really like to use the current momentum. I don't want to push anybody, I want us to have a clear vision about how to take steps in the right direction. Is someone working on this right now? If not, who volunteers? Let's just communicate...and be on the same page about knowing *who* does *what*, and *when*. What's a little behind this is, that I think this point has some strategic importance. It's a single issue that keeps a lot of users from switching to SCons. By getting it out of the way, we can present (and kind of sell) our project much better in any media out there. I have submitted a proposal for the poster session at the EuroPython 2014 in Berlin. It's title is How spawning many processes sequentially can kill performance, and if I could not only talk about our problem, but also present a solution that has some benefit for the Python community, that definitely carries some potential in my opinion. So let's really get this crackin'! ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On Fri, Apr 4, 2014 at 5:32 PM, Kenny, Jason L jason.l.ke...@intel.comwrote: Hi dirk, While I don't have a patch to SCons at the moment we have people in my team fixing this with Parts as a monkey path to sub process module. I asked the two people working on this to speak up and update their patch work online for SCons to consider. Jason, do you have a way to do posix_spawn from python (without a C extension)? -- Gary ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
I think yes, in that it does what should be done by the system under posix_spawn.. ie call vfork and execve. Here is the last version of the monkey patch I have from the people working on it. It has a fallback to the classic fork exec if the API's don't exists. It seems to solve the main speed problem for us at the moment. I believe it still being tested to find possible issues. Jason From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Gary Oberbrunner Sent: Friday, April 04, 2014 4:46 PM To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On Fri, Apr 4, 2014 at 5:32 PM, Kenny, Jason L jason.l.ke...@intel.commailto:jason.l.ke...@intel.com wrote: Hi dirk, While I don't have a patch to SCons at the moment we have people in my team fixing this with Parts as a monkey path to sub process module. I asked the two people working on this to speak up and update their patch work online for SCons to consider. Jason, do you have a way to do posix_spawn from python (without a C extension)? -- Gary stubprocess.py Description: stubprocess.py ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
