Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Neil Schemenauer 
Antoine Pitrou  wrote:
> Have you tried to remove the various hacks/workarounds in the
> stdlib that currently avoid module finalization problems?

Good idea. These hacks are the best argument for applying the
change, IMHO. I made a quick review of Lib modules using __del__:

tarfile: looks not safe
tempfile: goes through elaborate motions to avoid globals
threading: yikes, I didn't attempt to decypher this one
shelve: probably not safe
dumbdb: avoids globals, easy to remove hacks
_pyio: try/except can be removed (hopefully)
urllib: avoids globals, easy to remove hacks

Will try to cook up a patch in my copious free time.

  Neil

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Daniel Stutzbach 
On Wed, Oct 14, 2009 at 6:05 PM, Neil Schemenauer  wrote:

>def __del__():
>print sys.version
>
> the global variable reference to 'sys' is not a reference on the GC
> referencing counting sense. IOW, it does not result in a a Py_INCREF
> while the function is not being executed and therefore should be
> safe after the proposed change. Currently, it could result in 'None'
> being printed.
>

Currently it throws an exception since "sys" is None. :-)

Here is my understanding of the proposed procedure:

1. Replace modules in sys.modules with weakrefs
2. Run the garbage collector
3. Replace globals in any remaining modules with None
4. Run the garbage collector

Is it possible for a __del__ method to be called in step 4 or not?  I am
still unclear on this point. :-)

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Neil Schemenauer 
On Wed, Oct 14, 2009 at 05:35:38PM -0500, Daniel Stutzbach wrote:
> The first and last sentences seem like a contradiction to me.  If I cannot
> guarantee that globals will be valid when __del__ is executed, then I must
> not reference globals from __del__.

I think there is confusion about the word "reference". In the
method:

def __del__():
print sys.version

the global variable reference to 'sys' is not a reference on the GC
referencing counting sense. IOW, it does not result in a a Py_INCREF
while the function is not being executed and therefore should be
safe after the proposed change. Currently, it could result in 'None'
being printed.

HTH,

  Neil
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Daniel Stutzbach 
On Wed, Oct 14, 2009 at 4:42 PM, Neil Schemenauer  wrote:

> Yes, it does still resort to setting globals to None. However, the
> weakref step makes it much more likely that __del__ methods run
> before that happens. After this change, referencing global variables
> from __del__ methods is okay.
>

The first and last sentences seem like a contradiction to me.  If I cannot
guarantee that globals will be valid when __del__ is executed, then I must
not reference globals from __del__.

I think I'm missing something here.  Is there some way the programmer can
determine that referencing a global variable in __del__ will be 100% safe?
(not just "likely")

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Glyph Lefkowitz 
On Wed, Oct 14, 2009 at 2:45 PM, Neil Schemenauer  wrote:

> On Wed, Oct 14, 2009 at 08:35:28PM -, exar...@twistedmatrix.com wrote:
> > I notice that the patch doesn't include any unit tests for the feature
> > being provided
>
> That's hard to do although I suppose not impossible. We would be
> trying to test the interpreter shutdown procedure. I suppose on
> plaforms that support it we could fork a subprocess and then inspect
> the output from __del__ methods.


Why not just expose the module-teardown procedure and call it on a module,
then inspect the state there as it's being torn down?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Neil Schemenauer 
On Wed, Oct 14, 2009 at 08:35:28PM -, exar...@twistedmatrix.com wrote:
> I notice that the patch doesn't include any unit tests for the feature  
> being provided

That's hard to do although I suppose not impossible. We would be
trying to test the interpreter shutdown procedure. I suppose on
plaforms that support it we could fork a subprocess and then inspect
the output from __del__ methods.

> (it does change test_sys.py - although I don't understand  how
> that change is related to this feature).

That's a spurious change and will be left out of the real fix.

  Neil
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Neil Schemenauer 
On Wed, Oct 14, 2009 at 04:13:12PM -0500, Daniel Stutzbach wrote:
> Based on the description, it still resorts to zapping module globals by
> setting them to None.  It zaps them to weakrefs first, which means that
> globals are more likely to be valid during __del__, but it still cannot make
> any guarantees and referencing globals from __del__ is still a bad idea.  Is
> that a correct synopsis?

Yes, it does still resort to setting globals to None. However, the
weakref step makes it much more likely that __del__ methods run
before that happens. After this change, referencing global variables
from __del__ methods is okay. What is not a good idea is creating
__del__ methods that are part of a reference cycle (i.e. an island
of references) and also refer to that cycle somehow.  That has never
been a good idea and those __del__ methods will never get run,
before or after the proposed change.

HTH,

  Neil
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Daniel Stutzbach 
On Wed, Oct 14, 2009 at 3:16 PM, Neil Schemenauer  wrote:

> The current shutdown code in pythonrun.c zaps module globals by
> setting them to None (an attempt to break reference cycles). That
> causes problems since __del__ methods can try to use the globals
> after they have been set to None.
>
> The procedure implemented by http://bugs.python.org/issue812369
> seems to be a better idea. References to modules are replaced by
> weak references and the GC is allowed to cleanup reference cycles.
>

Based on the description, it still resorts to zapping module globals by
setting them to None.  It zaps them to weakrefs first, which means that
globals are more likely to be valid during __del__, but it still cannot make
any guarantees and referencing globals from __del__ is still a bad idea.  Is
that a correct synopsis?

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread Antoine Pitrou 
Neil Schemenauer  arctrix.com> writes:
> 
> I would like to commit this change but I'm not sure if it is a good
> time in the development cycle or if anyone has objections to the
> idea. Please speak up if you have input.

Have you tried to remove the various hacks/workarounds in the stdlib that
currently avoid module finalization problems?
(take a look at e.g. threading.py)
It would be a good validation that the patch improves on the current behaviour.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better module shutdown procedure

2009-10-14 Thread exarkun 

On 08:16 pm, n...@arctrix.com wrote:

The current shutdown code in pythonrun.c zaps module globals by
setting them to None (an attempt to break reference cycles). That
causes problems since __del__ methods can try to use the globals
after they have been set to None.

The procedure implemented by http://bugs.python.org/issue812369
seems to be a better idea. References to modules are replaced by
weak references and the GC is allowed to cleanup reference cycles.

I would like to commit this change but I'm not sure if it is a good
time in the development cycle or if anyone has objections to the
idea. Please speak up if you have input.


I notice that the patch doesn't include any unit tests for the feature 
being provided (it does change test_sys.py - although I don't understand 
how that change is related to this feature).  I hope that regardless of 
whatever else is decided, if the change is to be made, it will be 
accompanied by new unit tests verify its proper operation.


Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Better module shutdown procedure

2009-10-14 Thread Neil Schemenauer 
The current shutdown code in pythonrun.c zaps module globals by
setting them to None (an attempt to break reference cycles). That
causes problems since __del__ methods can try to use the globals
after they have been set to None.

The procedure implemented by http://bugs.python.org/issue812369
seems to be a better idea. References to modules are replaced by
weak references and the GC is allowed to cleanup reference cycles.

I would like to commit this change but I'm not sure if it is a good
time in the development cycle or if anyone has objections to the
idea. Please speak up if you have input.

  Neil
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday?

2009-10-14 Thread Barry Warsaw 

On Oct 14, 2009, at 8:06 AM, Martin v. Löwis wrote:


That seems to argue for doing rc2 on Sunday the 18th.  If I tag the
release some time Saturday, you could have the binaries by Sunday
right?


Correct.


Cool.  I've updated the Python Release Schedule calendar to reflect  
the new dates.


-Barry



PGP.sig
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] On track for Python 2.6.4 final this Sunday?

2009-10-14 Thread Martin v. Löwis 

> That seems to argue for doing rc2 on Sunday the 18th.  If I tag the
> release some time Saturday, you could have the binaries by Sunday
> right?

Correct.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com