Re: [Scons-dev] Subprocess issue on Linux?
Hi Dirk, Thank you for testing the stuff. I have re-produced and fixed the issue. Please try the latest version of stubprocess.py -Original Message- From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bachle Sent: Wednesday, April 09, 2014 11:28 PM To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On 09.04.2014 19:24, Bill Deegan wrote: > Dirk, > > That's pretty impressive! > Does it pass the full regression suite? > No, it doesn't work: 501/1110 (45.14%) /usr/bin/python -tt test/LEX/live.py /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py returned 2 STDOUT = scons: Reading SConscript files ... STDERR = KeyError: 'PATH': File "/tmp/testcmd.4854.Zu6ZsJ/SConstruct", line 2: foo = Environment() File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py", line 1003: apply_tools(self, tools, toolpath) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py", line 107: env.Tool(tool) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py", line 1787: tool(self) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", line 183: self.generate(env, *args, **kw) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/default.py", line 40: for t in SCons.Tool.tool_list(env['PLATFORM'], env): File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", line 819: ], env) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", line 690: return list(filter (ToolExists, tools)) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", line 689: return Tool(tool).exists(env) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/wix.py", line 71: for path in os.environ['PATH'].split(os.pathsep): File "/usr/lib/python2.7/UserDict.py", line 23: raise KeyError(key) FAILED test of /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py Looks like the wrapping of Subprocess.Popen in stubprocess.py prevents the os.environ settings to get through, so all tests that setup a simple Environment and auto-detect Tools are bound to fail. :( Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev Closed Joint Stock Company Intel A/O Registered legal address: Krylatsky Hills Business Park, 17 Krylatskaya Str., Bldg 4, Moscow 121614, Russian Federation This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. stubprocess.py Description: stubprocess.py ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On 09.04.2014 23:56, William Deegan wrote: Dirk, Is this available in your bitbucket repo? (URL?) No, I just used the subprocess.py from Eugene's mail and put it in my local copy of the SCons source tree (plus the import in posix.py). I didn't bother to add it to any repo yet... Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Dirk, Is this available in your bitbucket repo? (URL?) -Bill On April 9, 2014 at 12:35:52 PM, Kenny, Jason L (jason.l.ke...@intel.com) wrote: > There has to be a little more to this as Part only does auto detection of > tools, and this > is working fine in Parts from our tests. ( and the fact the builds did not > utterly break > because it could not find a tool) > > Jason > > -Original Message- > From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On > Behalf > Of Dirk Bächle > Sent: Wednesday, April 09, 2014 2:28 PM > To: SCons developer list > Subject: Re: [Scons-dev] Subprocess issue on Linux? > > On 09.04.2014 19:24, Bill Deegan wrote: > > Dirk, > > > > That's pretty impressive! > > Does it pass the full regression suite? > > > > No, it doesn't work: > > 501/1110 (45.14%) /usr/bin/python -tt test/LEX/live.py > /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py > returned 2 STDOUT > = > scons: Reading SConscript files ... > > STDERR > = > KeyError: 'PATH': > File "/tmp/testcmd.4854.Zu6ZsJ/SConstruct", line 2: > foo = Environment() > File > "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py", > > line 1003: > apply_tools(self, tools, toolpath) > File > "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py", > > line 107: > env.Tool(tool) > File > "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py", > > line 1787: > tool(self) > File > "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", > > line 183: > self.generate(env, *args, **kw) > File > "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/default.py", > > line 40: > for t in SCons.Tool.tool_list(env['PLATFORM'], env): > File > "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", > > line 819: > ], env) > File > "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", > > line 690: > return list(filter (ToolExists, tools)) > File > "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", > > line 689: > return Tool(tool).exists(env) > File > "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/wix.py", > > line 71: > for path in os.environ['PATH'].split(os.pathsep): > File "/usr/lib/python2.7/UserDict.py", line 23: > raise KeyError(key) > > FAILED test of /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py > > > Looks like the wrapping of Subprocess.Popen in stubprocess.py prevents the > os.environ > settings to get through, so all tests that setup a simple Environment and > auto-detect > Tools are bound to fail. :( > > Dirk > > ___ > Scons-dev mailing list > Scons-dev@scons.org > http://two.pairlist.net/mailman/listinfo/scons-dev > ___ > Scons-dev mailing list > Scons-dev@scons.org > http://two.pairlist.net/mailman/listinfo/scons-dev > ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
There has to be a little more to this as Part only does auto detection of tools, and this is working fine in Parts from our tests. ( and the fact the builds did not utterly break because it could not find a tool) Jason -Original Message- From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bächle Sent: Wednesday, April 09, 2014 2:28 PM To: SCons developer list Subject: Re: [Scons-dev] Subprocess issue on Linux? On 09.04.2014 19:24, Bill Deegan wrote: > Dirk, > > That's pretty impressive! > Does it pass the full regression suite? > No, it doesn't work: 501/1110 (45.14%) /usr/bin/python -tt test/LEX/live.py /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py returned 2 STDOUT = scons: Reading SConscript files ... STDERR = KeyError: 'PATH': File "/tmp/testcmd.4854.Zu6ZsJ/SConstruct", line 2: foo = Environment() File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py", line 1003: apply_tools(self, tools, toolpath) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py", line 107: env.Tool(tool) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py", line 1787: tool(self) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", line 183: self.generate(env, *args, **kw) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/default.py", line 40: for t in SCons.Tool.tool_list(env['PLATFORM'], env): File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", line 819: ], env) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", line 690: return list(filter (ToolExists, tools)) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", line 689: return Tool(tool).exists(env) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/wix.py", line 71: for path in os.environ['PATH'].split(os.pathsep): File "/usr/lib/python2.7/UserDict.py", line 23: raise KeyError(key) FAILED test of /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py Looks like the wrapping of Subprocess.Popen in stubprocess.py prevents the os.environ settings to get through, so all tests that setup a simple Environment and auto-detect Tools are bound to fail. :( Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
On 09.04.2014 19:24, Bill Deegan wrote: Dirk, That's pretty impressive! Does it pass the full regression suite? No, it doesn't work: 501/1110 (45.14%) /usr/bin/python -tt test/LEX/live.py /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py returned 2 STDOUT = scons: Reading SConscript files ... STDERR = KeyError: 'PATH': File "/tmp/testcmd.4854.Zu6ZsJ/SConstruct", line 2: foo = Environment() File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py", line 1003: apply_tools(self, tools, toolpath) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py", line 107: env.Tool(tool) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Environment.py", line 1787: tool(self) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", line 183: self.generate(env, *args, **kw) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/default.py", line 40: for t in SCons.Tool.tool_list(env['PLATFORM'], env): File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", line 819: ], env) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", line 690: return list(filter (ToolExists, tools)) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/__init__.py", line 689: return Tool(tool).exists(env) File "/home/dirk/workspace/scons_dirkbaechle/src/script/../engine/SCons/Tool/wix.py", line 71: for path in os.environ['PATH'].split(os.pathsep): File "/usr/lib/python2.7/UserDict.py", line 23: raise KeyError(key) FAILED test of /home/dirk/workspace/scons_dirkbaechle/src/script/scons.py Looks like the wrapping of Subprocess.Popen in stubprocess.py prevents the os.environ settings to get through, so all tests that setup a simple Environment and auto-detect Tools are bound to fail. :( Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Fix async subprocess execution with proper handling of std* streams
I finial took some time today to update Parts online. The latest stuff is in the Part 0.10.8 drop including the subprocess fix we propose for SCons. Let me know if you have any questions Jason -Original Message- From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On Behalf Of anatoly techtonik Sent: Tuesday, February 25, 2014 1:53 AM To: SCons developer list Subject: Re: [Scons-dev] Fix async subprocess execution with proper handling of std* streams That's cool. The document is a very good starting point. To me this streams framework is a very useful component that should be merged into provisional packages of Python standard library. I still need to find time to read it thoroughly to adapt to what I know. On Mon, Feb 24, 2014 at 9:57 PM, Kenny, Jason L wrote: > Hi Anatoly, > > Attached is a word document with the overview of the deign I have with > thoughts on some good and bad of it. Hopefully I proofed the document enough, > document writing is not my strong suit. Anyways let me know if you have any > other questions, etc.. on the design. > > Eugene who has been helping with Parts work inside thinks he fixed a > Scons/Parts rebuild issues we have been dealing with. I testing it out, if it > looks good I will push this drop in a few days. > > Thanks for looking at this! > Jason > > -Original Message- > From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] > On Behalf Of anatoly techtonik > Sent: Tuesday, February 18, 2014 10:35 AM > To: SCons developer list > Subject: Re: [Scons-dev] Fix async subprocess execution with proper > handling of std* streams > > Thanks. That would be great. ;) > > On Tue, Feb 18, 2014 at 5:21 PM, Kenny, Jason L > wrote: >> I will get it pushed… life getting in the way. >> >> >> >> From: scons-dev-boun...@scons.org >> [mailto:scons-dev-boun...@scons.org] >> On Behalf Of Eugene Leskinen >> Sent: Tuesday, February 18, 2014 7:03 AM >> To: SCons developer list >> Subject: Re: [Scons-dev] Fix async subprocess execution with proper >> handling of std* streams >> >> >> >> Hi Anatoly, >> >> >> >> The code supporting I/O redirection in Parts is located in >> parts/part_logger.py file. Look at >> http://parts.tigris.org/source/browse/parts/trunk/parts/parts/part_lo >> g >> ger.py?revision=486&view=markup >> >> >> >> The code uses Python threading module to create threads for each >> stream and then runs a loop in every thready like your code does: >> >> >> >> line = ' ' >> >> try: >> >> while line: >> >> line = self.pipein.readline() >> >> if line: >> >> self.output.WriteStream(self.taskId, >> self.streamId, >> line) >> >> except: >> >> # There was an error... that shouldn't happen, but still it did. >> So we report it >> >> # to the caller and close our pipe end so that spawned >> program won't block >> >> self.error = traceback.format_exc() >> >> self.pipein.close() >> >> self.pipein = None >> >> >> >> Please note that public repo is a bit out-of-date. I hope Jason will >> find time to pull up-to-date version from Intel internal repository. >> >> >> >> 2014-02-18 16:30 GMT+04:00 anatoly techtonik : >> >> On Wed, Feb 12, 2014 at 6:25 PM, Kenny, Jason L >> >> wrote: Fix async subprocess execution with proper handling of std* streams. >>> >>> So I fixed the issue with output in Parts. I not saying it could not >>> be clean up some more. However it solves the output issues, and adds >>> coloring ( which seems to need some fixing in certain cases.. ie mac >>> mostly) and logging support. It might not be too hard for someone to >>> move the code over to SCons and integrated its usage into SCons. >> >> Do you have, by any chance a description of output issues? >> It seems that I've lost my list during the last Windows partition crash. >> >> Also, can you pinpoint the parts in Parts that are relevant to the issues? >> I may have time to port them over to SCons. >> >> My approach >> https://bitbucket.org/techtonik/absub >> to make subprocess asynchronous by moving processing of input/output >> streams into separate threads. I don't remember why, but the work >> stalled. I remember there was some kind of error that I was not able >> to debug due to distributed nature of the process. I needed to write >> more tests for that, but it is either I run out of time, >> multithreaded tests were complicated or I just had an annual Windows >> filesystem crash. >> >> Now I am thinking about replacing concept of input/output streams >> with the concept of channel, which is a queue of strings with limited size. >> The queue can be rotated to become unlimited. Can be synchronous to >> wait (block) until messages are read if the queue is full. This may >> help to clear confusion around all that input output streams >> handling, pipes and descriptor inheritances. >> >
Re: [Scons-dev] Subprocess issue on Linux?
On 09.04.2014 19:24, Bill Deegan wrote: Dirk, That's pretty impressive! Does it pass the full regression suite? Haven't tried yethang on a few seconds. ;) Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Wow! -- Gary Oberbrunner (sent from my Android) On Apr 9, 2014 10:14 AM, "Dirk Bächle" wrote: > Hi all, > > I ran another benchmark for clean builds with the stubprocess.py imported. > Please, find the results/graphs attached... > > Best regards, > > Dirk > > On 07.04.2014 20:39, Leskinen, Eugene wrote: > > Hi Dirk, > > > > Please find latest version of stubprocess.py attached. This version has > fixes which make it work under Python 2.6 and correctly handle situation > when executable to run does not exist or has wrong permissions. > > > > Hope you'll find it useful. > > > > > > ___ > Scons-dev mailing list > Scons-dev@scons.org > http://two.pairlist.net/mailman/listinfo/scons-dev > > ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Subprocess issue on Linux?
Dirk, That's pretty impressive! Does it pass the full regression suite? -Bill On Wed, Apr 9, 2014 at 10:14 AM, Dirk Bächle wrote: > Hi all, > > I ran another benchmark for clean builds with the stubprocess.py imported. > Please, find the results/graphs attached... > > Best regards, > > Dirk > > > On 07.04.2014 20:39, Leskinen, Eugene wrote: > > Hi Dirk, > > > > Please find latest version of stubprocess.py attached. This version has > fixes which make it work under Python 2.6 and correctly handle situation > when executable to run does not exist or has wrong permissions. > > > > Hope you'll find it useful. > > > > > > ___ > Scons-dev mailing list > Scons-dev@scons.org > http://two.pairlist.net/mailman/listinfo/scons-dev > > ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
