Re: [Python-Dev] Better module shutdown procedure

2009-10-16 Thread Amaury Forgeot d'Arc
2009/10/16 Neil Schemenauer n...@arctrix.com: After some experimentation I realize this idea is not ready yet. The main problem comes from references to Python objects that modules keep but don't expose to the garbage collector. For example, gcmodule.c has a static pointer tmod that is a

Re: [Python-Dev] Better module shutdown procedure

2009-10-16 Thread Benjamin Peterson
2009/10/16 Amaury Forgeot d'Arc amaur...@gmail.com: 2009/10/16 Neil Schemenauer n...@arctrix.com: After some experimentation I realize this idea is not ready yet. The main problem comes from references to Python objects that modules keep but don't expose to the garbage collector. For example,

Re: [Python-Dev] Better module shutdown procedure

2009-10-15 Thread Antoine Pitrou
Le Wed, 14 Oct 2009 18:27:37 -0500, Daniel Stutzbach a écrit : Here is my understanding of the proposed procedure: 1. Replace modules in sys.modules with weakrefs 2. Run the garbage collector 3. Replace globals in any remaining modules with None 4. Run the garbage collector Is it

Re: [Python-Dev] Better module shutdown procedure

2009-10-15 Thread Neil Schemenauer
On Wed, Oct 14, 2009 at 02:16:35PM -0600, Neil Schemenauer wrote: The procedure implemented by http://bugs.python.org/issue812369 seems to be a better idea. After some experimentation I realize this idea is not ready yet. The main problem comes from references to Python objects that modules

[Python-Dev] Better module shutdown procedure

2009-10-14 Thread Neil Schemenauer
The current shutdown code in pythonrun.c zaps module globals by setting them to None (an attempt to break reference cycles). That causes problems since __del__ methods can try to use the globals after they have been set to None. The procedure implemented by http://bugs.python.org/issue812369

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread exarkun
On 08:16 pm, n...@arctrix.com wrote: The current shutdown code in pythonrun.c zaps module globals by setting them to None (an attempt to break reference cycles). That causes problems since __del__ methods can try to use the globals after they have been set to None. The procedure implemented by

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Antoine Pitrou
Neil Schemenauer nas at arctrix.com writes: I would like to commit this change but I'm not sure if it is a good time in the development cycle or if anyone has objections to the idea. Please speak up if you have input. Have you tried to remove the various hacks/workarounds in the stdlib that

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Daniel Stutzbach
On Wed, Oct 14, 2009 at 3:16 PM, Neil Schemenauer n...@arctrix.com wrote: The current shutdown code in pythonrun.c zaps module globals by setting them to None (an attempt to break reference cycles). That causes problems since __del__ methods can try to use the globals after they have been set

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Neil Schemenauer
On Wed, Oct 14, 2009 at 04:13:12PM -0500, Daniel Stutzbach wrote: Based on the description, it still resorts to zapping module globals by setting them to None. It zaps them to weakrefs first, which means that globals are more likely to be valid during __del__, but it still cannot make any

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Neil Schemenauer
On Wed, Oct 14, 2009 at 08:35:28PM -, exar...@twistedmatrix.com wrote: I notice that the patch doesn't include any unit tests for the feature being provided That's hard to do although I suppose not impossible. We would be trying to test the interpreter shutdown procedure. I suppose on

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Glyph Lefkowitz
On Wed, Oct 14, 2009 at 2:45 PM, Neil Schemenauer n...@arctrix.com wrote: On Wed, Oct 14, 2009 at 08:35:28PM -, exar...@twistedmatrix.com wrote: I notice that the patch doesn't include any unit tests for the feature being provided That's hard to do although I suppose not impossible. We

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Daniel Stutzbach
On Wed, Oct 14, 2009 at 4:42 PM, Neil Schemenauer n...@arctrix.com wrote: Yes, it does still resort to setting globals to None. However, the weakref step makes it much more likely that __del__ methods run before that happens. After this change, referencing global variables from __del__

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Neil Schemenauer
On Wed, Oct 14, 2009 at 05:35:38PM -0500, Daniel Stutzbach wrote: The first and last sentences seem like a contradiction to me. If I cannot guarantee that globals will be valid when __del__ is executed, then I must not reference globals from __del__. I think there is confusion about the word

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Daniel Stutzbach
On Wed, Oct 14, 2009 at 6:05 PM, Neil Schemenauer n...@arctrix.com wrote: def __del__(): print sys.version the global variable reference to 'sys' is not a reference on the GC referencing counting sense. IOW, it does not result in a a Py_INCREF while the function is not being

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Neil Schemenauer
Antoine Pitrou solip...@pitrou.net wrote: Have you tried to remove the various hacks/workarounds in the stdlib that currently avoid module finalization problems? Good idea. These hacks are the best argument for applying the change, IMHO. I made a quick review of Lib modules using __del__: