Re: [Python-Dev] [Python-checkins] r45321 - in python/trunk: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS

[Tim Peters]

[John J Lee]
>> Assuming this is fixed in 2.5 final, is there some way to write doctests that
>> work on both 2.4 and 2.5?  If not, should something like
>> doctest.IGNORE_EXCEPTION_DETAIL be added -- say
>> IGNORE_EXCEPTION_MODULE?

[also John]
> Sorry, please ignore the post of mine I'm replying to here.
>
> I missed part of the thread, and Tim has already answered my question...

That's news to Tim ;-)  It's not possible to add a new doctest option
in 2.5 that would allow a doctest to work under both 2.4 and 2.5:  the
test would blow up under 2.4, because 2.4's doctest wouldn't recognize
the (new in 2.5) option, and would raise a ValueError of its own
griping about the unknown option name.

It would be possible to add a new doctest option in 2.5 that would
(just) allow a doctest running _under_ 2.5 to accept a bare "name" in
an exception despite that the traceback module changed in 2.5 to
produce "some.dotted.name" instead.  That doesn't seem very useful,
though (as above, it would force the test to fail when run under 2.4).

Of course a doctest can contain any Python code, so there are many
ways to write a doctest that passes under all versions of Python,
without active help from doctest.  For example, using the running
example in this thread, this way works under all versions of Python
with a decimal module:

"""
>>> import decimal
>>> try:
>>> 1 / decimal.Decimal(0)
>>> except decimal.DivisionByZero:
>>> print "good"
good
"""

Oddly enough, ELLIPSIS doesn't actually work for this purpose:

"""
>>> import decimal
>>> 1 / decimal.Decimal(0) #doctest: +ELLIPSIS
Traceback (most recent call last):
   [etc]
...DivisionByZero: x / 0
"""

That test fails under 2.4, and would also fail under the presumably
changed 2.5.  The problem is that, as the docs say, when trying to
guess where an exception message starts, doctest skips down to the
first line after the "Traceback" line indented the same as the
"Traceback" line and starting with an _alphanumeric_ character.  There
is no such line in this example (the exception line starts with a
period), so doctest believes the example is showing expected stdout,
not that it's showing an exception.  The delightfully baffling result
is that doctest complains that the example raises an exception when an
exception wasn't expected.  Stinking magic ;-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Debugging opportunity :-)

[Tim Peters]

[Phillip J. Eby]
> Even with the close() gimmicks still in place?  If so, it sounds like the
> safest thing for a2 might be to claim that generators always need
> finalizing.  :(

LOL!  Been there many times before:  Python's gc really does work, but
it's as unforgivingly delicate as any dance you're likely to learn
while still mortal :-)  This one was actually typical, ending with an
unlikely comedy of errors involving microscopic details of how
multiple objects' cleanup routines could barely manage to fool each
other into tearing down an object more than once.  Until you get to
that point of clarity, it's also typical to fear that it can never be
fixed ;-)

You and Thomas did a wonderful of job of tracking this one down
quickly.  Good show!  Take Easter off -- but only if you have to :-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Debugging opportunity :-)

[Phillip J. Eby]

At 01:11 AM 4/15/2006 +0200, Thomas Wouters wrote:
>On 4/15/06, Thomas Wouters <[EMAIL PROTECTED]> 
>wrote:
>>(I first thought the problem was caused by gen_dealloc doing 
>>'Py_DECREF(gen->gen_frame)' instead of 'frame = gen->gen_frame; 
>>gen->gen_frame = NULL; Py_DECREF(frame)', but that isn't the case. It 
>>should do it that way, I believe, but it's not the cause of this crash.)
>
>fixes the crash.Ah, found the problem. After I hit 'send', I realized I 
>hadn't checked frameobject's tp_clear, and sure enough, it calls 
>Py_XDECREF in-place. That explains why the first generator object gets 
>dealloced twice, although it doesn't explain why it doesn't blow up when 
>it reaches a negative refcount.

You're missing another piece of the puzzle: the problem is that since 
generators don't have a tp_clear, they don't know they're pointing to an 
invalid frame; it appears to still be running.  So when the frame object 
releases items off the stack, it releases its reference to the generator, 
which is released normally, dropping out the local variable reference back 
to the generator whose frame is being cleared.  That generator doesn't know 
it's holding a garbage frame, and thus proceeds to finalize it by resuming 
it...  which then pops the blockstack, and f_stacktop is still valid, so 
popping the blockstack decrefs the frame a second time.


>  Fixing frameobject and genobject to both use Py_CLEAR() makes both the 
> 'minimal' testcase and test_generators work.

It seems to me that frame_clear() should also set f_stacktop to NULL before 
doing any clearing; otherwise it's possible for a generator to think that 
the frame is still executable, and the double-decref could thus be replaced 
by a null pointer dereference.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Debugging opportunity :-)

[Thomas Wouters]

On 4/15/06, Thomas Wouters <[EMAIL PROTECTED]> wrote:
(I first thought the problem was caused by gen_dealloc doing 'Py_DECREF(gen->gen_frame)' instead of 'frame = gen->gen_frame; gen->gen_frame = NULL; Py_DECREF(frame)', but that isn't the case. It should do it that way, I believe, but it's not the cause of this crash.)
 fixes the crash.Ah, found the problem. After I hit 'send', I realized I hadn't checked frameobject's tp_clear, and sure enough, it calls Py_XDECREF in-place. That explains why the first generator object gets dealloced twice, although it doesn't explain why it doesn't blow up when it reaches a negative refcount. Fixing frameobject and genobject to both use Py_CLEAR() makes both the 'minimal' testcase and test_generators work. Your testcase also stops leaking, but alas, test_generators still leaks 255 references. WTF-time for me, meaning I hit the sack and not think about it until next week :)
I'll upload a patch to Tim's bugreport after I remove my debugging cruft. All tp_clear/tp_traverse methods should really always use Py_CLEAR/Py_VISIT. I guess revisiting all tp_clear and tp_traverse methods is well worth putting on the TODO list,
Bad-pun'ly y'rs,-- Thomas Wouters <[EMAIL PROTECTED]>Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Debugging opportunity :-)

[Phillip J. Eby]

At 06:26 PM 4/14/2006 -0400, Phillip J. Eby wrote:
>If you replace the fun_tests in test_generators with this, and remove all
>the other tests (e.g. pep_tests, tut_tests, etc.) you can still get it to
>crash.  The code above is much shorter, but has the disadvantage that I
>don't really know what it's supposed to do, other than cause a crash.  :)
>
>Interestingly, this code does *not* crash the interpreter on its own, only
>when run as a doctest (with or without regrtest).  I still haven't figured
>out what the actual problem is, but at least this cuts out all the merge()
>and times() crud in the case this was derived from, reducing it to a pure
>zen essence of non-meaning.  :)

I believe this is now the irreducible minimum code necessary to produce the 
crash:

fun_tests = """
 >>> def cycle(g):
... yield 1
... yield 1

 >>> def root():
... for i in c:
... yield i

 >>> r = root()
 >>> c = cycle(r)
 >>> c.next()
1
 >>> r.next()
1
"""

I have not been able to think of any way to further simplify this, that 
doesn't make the crash go away.  For example, if I replace the "for" loop 
inside of "root()" with  iter() and next() operations (either with or 
without a "while" loop), the crash does NOT occur.  I don't know if that's 
a red herring, or if it means the crash is inherently about there being a 
"for" loop on the stack.

I also know that the object 'r' has to be created before 'c' -- in previous 
iterations I found that just creating a cycle between two generators isn't 
enough; the one that has the paused "for" loop has to be created before the 
thing it's looping over.  I'm guessing that this means the order they're 
listed in the GC tracking lists makes a difference.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Debugging opportunity :-)

[Thomas Wouters]

On 4/15/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
Interestingly, this code does *not* crash the interpreter on its own, onlywhen run as a doctest (with or without regrtest).Actually, it does, you just have to force GC to clean up stuff (rather than wait for the interpreter-exit to just toss the whole lot out without giving it a second glance.) You do that by making it unreachable and calling 
gc.collect().centurion:~/python/python/trunk > cat test_gencrash.pyimport gcclass LazyList:    def __init__(self, g):    self.v = None    self.g = g    def __iter__(self):
    yield 1    if self.v is None:    self.v = self.g.next()    yield self.vdef loop():    for i in ll:    yield ill = LazyList(loop())g=iter(ll)g.next
()g.next()del g, llprint gc.collect()centurion:~/python/python/trunk > ./python test_gencrash.pySegmentation fault (core dumped)
I still haven't figuredout what the actual problem is, but at least this cuts out all the merge()and times() crud in the case this was derived from, reducing it to a purezen essence of non-meaning.  :)
I'm sure I know where the crash is coming from: one of the generator objects is being dealloced twice. The GC module tp_clears one of the frame objects, which deallocs a generator the frame referenced. That first dealloc calls _Py_ForgetReference and then calls gen_dealloc, which calls its tp_del (as it's a running generator, albeit a simple one), which raises an exception in the frame, which causes it to be cleaned up, which causes another generator object to be cleaned up and its tp_del method to be called, which - for some reason - tries to dealloc the first generator again. The second dealloc calls _Py_ForgetReference again, and _Py_ForgetReference doesn't like that (the ob_prev/ob_next debug ptrs are already wiped when it's called the second time, so it crashes.)
The thing I don't understand is how the cleaning up of the second generator triggers a dealloc of the first generator. The refcount is 0 when the dealloc is called the first time (or _Py_ForgetReference, as well as a few other places, would have bombed out), and it is also 0 when it gets dealloced the second time. Since deallocation is triggered by DECREF'ing, that implies something is INCREF'ing the first generator somewhere -- but without having a valid reference, since the generator has been cleaned up by actually DECREF'ing the reference that a frame object held. Or, somehow, deallocation is triggered by something other than a DECREF. Hmm. No, the second dealloc is triggered by the Py_XDECREF in 
ceval.c, when clearing the stack after an exception, which is the right thing to do. Time to set some breakpoints and step through the code, I guess.(I first thought the problem was caused by gen_dealloc doing 'Py_DECREF(gen->gen_frame)' instead of 'frame = gen->gen_frame; gen->gen_frame = NULL; Py_DECREF(frame)', but that isn't the case. It should do it that way, I believe, but it's not the cause of this crash.)
-- Thomas Wouters <[EMAIL PROTECTED]>Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Debugging opportunity :-)

[Phillip J. Eby]

At 11:51 PM 4/14/2006 +0200, Thomas Wouters wrote:

>On 4/14/06, Phillip J. Eby 
><[EMAIL PROTECTED]> wrote:
>>At 03:49 PM 4/14/2006 -0400, Tim Peters wrote:
>> >   That eventually
>> >pointed to an off-by-one error in the new PyGen_NeedsFinalizing(),
>> >where reading up trash made it very likely that old generators
>> >containing an active loop would falsely claim they need finalization.
>> >Alas, fix that, and Python segfaults when running test_generators.
>>
>>Even with the close() gimmicks still in place?  If so, it sounds like the
>>safest thing for a2 might be to claim that generators always need
>>finalizing.  :(
>
>No, it sounds like there's a bug in generator cleanup :-) From Tim's 
>description, it isn't obvious to me that the bug is in the hack to clean 
>up simple generator frames. I had "finding generator-cycle leak in 
>test_generators" on my TODO list (as I'd noticed it was still leaking, 
>too), I've now replaced that with "finding the generator-cleanup crash Tim 
>encountered" :-) I don't know if I'll get to it this weekend, though, so 
>if anyone else wants to hunt for it, don't let me stop you.

FYI, the smallest code I've found so far that reproduces the crash is:

fun_tests = """
 >>> class LazyList:
... def __init__(self, g):
... self.v = None
... self.g = g
...
... def __iter__(self):
... yield 1
... if self.v is None:
... self.v = self.g.next()
... yield self.v

 >>> def loop():
... for i in ll:
... yield i

 >>> ll = LazyList(loop())
 >>> g=iter(ll)
 >>> g.next()
1
 >>> g.next()
1
"""

If you replace the fun_tests in test_generators with this, and remove all 
the other tests (e.g. pep_tests, tut_tests, etc.) you can still get it to 
crash.  The code above is much shorter, but has the disadvantage that I 
don't really know what it's supposed to do, other than cause a crash.  :)

Interestingly, this code does *not* crash the interpreter on its own, only 
when run as a doctest (with or without regrtest).  I still haven't figured 
out what the actual problem is, but at least this cuts out all the merge() 
and times() crud in the case this was derived from, reducing it to a pure 
zen essence of non-meaning.  :)

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Debugging opportunity :-)

[Thomas Wouters]

On 4/14/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
At 03:49 PM 4/14/2006 -0400, Tim Peters wrote:>   That eventually>pointed to an off-by-one error in the new PyGen_NeedsFinalizing(),>where reading up trash made it very likely that old generators>containing an active loop would falsely claim they need finalization.
>Alas, fix that, and Python segfaults when running test_generators.Even with the close() gimmicks still in place?  If so, it sounds like thesafest thing for a2 might be to claim that generators always need
finalizing.  :(No, it sounds like there's a bug in generator cleanup :-) From Tim's description, it isn't obvious to me that the bug is in the hack to clean up simple generator frames. I had "finding generator-cycle leak in test_generators" on my TODO list (as I'd noticed it was still leaking, too), I've now replaced that with "finding the generator-cleanup crash Tim encountered" :-) I don't know if I'll get to it this weekend, though, so if anyone else wants to hunt for it, don't let me stop you.
-- Thomas Wouters <[EMAIL PROTECTED]>Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 359: The "make" Statement

[Ian Bicking]

BJörn Lindqvist wrote:
> [nice way to declare properties with "make"]
> 
> 
>>Of course, properties are only one of the many possible uses of the
>>make statement.  The make statement is useful in essentially any
>>situation where a name is associated with a namespace.  So, for
> 
> 
> So far, in this thread that is the only useful use of the make
> statement that has been presented. I'd like to see more examples.

In SQLObject I would prefer:

class Foo(SQLObject):
 make IntCol bar:
 notNull = True

In FormEncode I would prefer:

make Schema registration:
 make String name:
 max_length = 100
 not_empty = True
 make PostalCode postal_code:
 not_empty = True
 make Int age:
 min = 18

In another thread on the python-3000 list I suggested (using :

class Point(object):
 make setonce x:
 "x coordinate"
 make setonce y:
 "y coordinate"

For a read-only x and y property (setonce because they have to be set to 
*something*, but just never re-set).

Interfaces are nice:

make interface IValidator:
 make attribute if_empty:
 """If this attribute is not NoDefault, then this value
 will be used in lieue of an empty value"""
 default = NoDefault
 def to_python(value, state): """..."""


Another descriptor, stricttype 
(http://svn.colorstudy.com/home/ianb/recipes/stricttype.py):

class Pixel(object):
 make stricttype x:
 type = int
 make stricttype y:
 type = int

(Both this descriptor and setonce need to know their name if they are 
going to store their value in the object in a stable location)


> It would be really cool if you could go through the standard library,
> and replace code there with code using the make statement. I think a
> patch showing how much nicer good Python code would be with the make
> statement would be a very convincing argument.

I don't know if the standard library will have a whole lot; "make" is 
really only useful when frameworks are written to use it, and there's 
just not a lot of framework in the standard library.  Maybe:

make OptionParser myparser:
 make Option verbose:
 short = '-v'
 help = """...""



-- 
Ian Bicking  /  [EMAIL PROTECTED]  /  http://blog.ianbicking.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Debugging opportunity :-)

[Phillip J. Eby]

At 03:49 PM 4/14/2006 -0400, Tim Peters wrote:
>   That eventually
>pointed to an off-by-one error in the new PyGen_NeedsFinalizing(),
>where reading up trash made it very likely that old generators
>containing an active loop would falsely claim they need finalization.
>Alas, fix that, and Python segfaults when running test_generators.

Even with the close() gimmicks still in place?  If so, it sounds like the 
safest thing for a2 might be to claim that generators always need 
finalizing.  :(

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r45321 - in python/trunk: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS

[John J Lee]

Sorry, please ignore the post of mine I'm replying to here.

I missed part of the thread, and Tim has already answered my question...

On Fri, 14 Apr 2006, John J Lee wrote:
[...]
> Assuming this is fixed in 2.5 final, is there some way to write doctests that 
> work on both 2.4 and 2.5?  If not, should something like 
> doctest.IGNORE_EXCEPTION_DETAIL be added -- say IGNORE_EXCEPTION_MODULE?
[...]

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r45321 - in python/trunk: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS

[John J Lee]

On Thu, 13 Apr 2006, "Martin v. Löwis" wrote:


Tim Peters wrote:

I'm not the one to decide, but at some time the traceback module should be
rewritten to match the interpreter behavior.


No argument from me about that.


I also think the traceback module should be corrected, and the test
cases updated, despite the objections that it may break other people's
doctest test cases.


Assuming this is fixed in 2.5 final, is there some way to write doctests 
that work on both 2.4 and 2.5?  If not, should something like 
doctest.IGNORE_EXCEPTION_DETAIL be added -- say IGNORE_EXCEPTION_MODULE?



John___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Debugging opportunity :-)

[Tim Peters]

When "possible finalizers" were added to generators, the
implementation at first said that all generators need finalizing.  As
a result, some old tests in test_generators.py started leaking cyclic
trash.  As a result of that, someone(s) added a number of explicit
"close" gimmicks to those tests, to break the cycles and stop the
leaks.

Recently smarter gc code was added, so that most old generators should
no longer say they need finalization.  But the new close-it cruft
added to test_generators wasn't removed, so I thought I'd try that. 
Alas, they still leaked without the close-it cruft.  That eventually
pointed to an off-by-one error in the new PyGen_NeedsFinalizing(),
where reading up trash made it very likely that old generators
containing an active loop would falsely claim they need finalization. 
Alas, fix that, and Python segfaults when running test_generators.

I'm out of time for this now, so wrote up what I know, and attached a
patch sufficient to reproduce it:

http://www.python.org/sf/1470508

It's nominally assigned to Phillip, but for anyone who enjoys a
significant debugging challenge, it's likely to be much more fun than
skinning the Easter Bunny :-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 359: The "make" Statement

[BJörn Lindqvist]

[nice way to declare properties with "make"]

> Of course, properties are only one of the many possible uses of the
> make statement.  The make statement is useful in essentially any
> situation where a name is associated with a namespace.  So, for

So far, in this thread that is the only useful use of the make
statement that has been presented. I'd like to see more examples.

It would be really cool if you could go through the standard library,
and replace code there with code using the make statement. I think a
patch showing how much nicer good Python code would be with the make
statement would be a very convincing argument.

--
mvh Björn
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Procedure for sandbox branches in SVN?

[Tim Peters]

[Martin v. Löwis]
> That's all true. Of course, removing the branch won't free up
> any disk space - it will just remove the branch from the view
> (IOW, it is also easy to bring it back if it was removed mistakenly).

Right!  I'm implicitly addressing a different issue, namely that two
reasons for people disliking branches in CVS can "go away by magic"
under SVN:

1. There's no way to attach a comment to a branch, explaining
   why it exists.  In SVN, that's the checkin comment on the
   `copy` command that creates the branch.

2. Branches are in your face forever after, because the CVS docs strongly
   warn against trying to use the relevant commands even for tags (let alone
   branches).  The CVS moral equivalent of "svn list repo/branches" keeps
   growing as a result, and shows up in each `cvs log`.  The ever-growing
   clutter of long-obsolete info gets depressing <0.5 wink>.  The easy
   possibility in SVN for "out of sight, out of mind" is a huge improvement.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New-style icons, .desktop file

[Fred L. Drake, Jr.]

On Friday 14 April 2006 06:35, Andrew Clover wrote:
 > Files and preview here:
 >
 >http://doxdesk.com/img/software/py/icons2.zip
 >http://doxdesk.com/img/software/py/icons2.png

Very nice!


  -Fred

-- 
Fred L. Drake, Jr.   
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 359: The "make" Statement

[skip]

Travis> I generally like the idea.  A different name would be better.

Travis> Here's a list of approximate synonyms that might work (ordered
Travis> by my preference...)

... lots of suggestions elided ...

None of the alternatives seem better to me than "make" or "create".  In
fact, many don't seem to me (as a native English speaker) to be fairly
obscure synonyms.  "construct" or "build" are maybe the best of the lot if
neither "make" nor "create" are deemed worthy.  I'm willing to put up with
the pain of some breakage to have a better keyword.  That's what __future__
imports are for.

We need to consider that Python exists outside the English-speaking world.
Making a strong connection between the keyword and its action is important.
For that reason I'd much prefer simple, common keywords.  Yes, I realize
that some keywords in Python are more symbolic than that: "def", "lambda",
"del", "elif", "exec".  But most keywords in Python are common English,
understood by almost anyone having any facility with the language.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] IMPORTANT 2.5 API changes for C Extension Modules and Embedders

[Neal Norwitz]

If you don't write or otherwise maintain Python Extension Modules
written in C (or C++) or embed Python in your application,
you can stop reading.

Python 2.5 alpha 1 was released April 5, 2006.  The second alpha
should be released in a few weeks.  There are several changes
which can cause C extension modules or embedded applications
to crash the interpreter if not fixed.  Periodically, I will send out
these reminders with updated information until 2.5 is released.

  * support for 64-bit sequences (eg, > 2GB strings)
  * memory allocation modifications

64-bit changes
--
There are important changes that are in 2.5 to support 64-bit systems.
The 64-bit changes can cause Python to crash if your module is not upgraded
to support the changes.  Python was changed internally to use 64-bit
values on 64-bit machines for indices.  If you've got a machine with
more than 16 GB of RAM, it would be great if you can test Python with
large (> 2GB) strings and other sequences.

For more details about the Python 2.5 schedule:
http://www.python.org/dev/peps/pep-0356/
For more details about the 64-bit change:
http://www.python.org/dev/peps/pep-0353/
How to fix your module:
http://www.python.org/dev/peps/pep-0353/#conversion-guidelines

The effbot wrote a program to check your code and find potential
problems with the 64-bit APIs.
http://svn.effbot.python-hosting.com/stuff/sandbox/python/ssizecheck.py

Memory Allocation Modifications
---
In previous versions of Python, it was possible to use different
families of APIs (PyMem_* vs. PyObject_*) to allocate and free
the same block of memory.  APIs in these families include:

  PyMem_*:PyMem_Malloc, PyMem_Realloc, PyMem_Free,
  PyObject_*: PyObject_Malloc, PyObject_Realloc, PyObject_Free

There are a few other APIs with similar names and also the macro variants.

In 2.5, if allocate a block of memory with one family, you must reallocate
or free with the same family.  That means:

  If you allocate with PyMem_Malloc (or MALLOC), you must reallocate
  with PyMem_Realloc (or REALLOC) and free with PyMem_Free (or FREE).
  If you allocate with PyObject_Malloc (or MALLOC), you must reallocate
  with PyObject_Realloc (or REALLOC) and free with PyObject_Free (or FREE).

Using inconsistent APIs can cause double frees or otherwise crash
the interpreter.  It is fine to mix and match functions or macros
within the same family.

Please test and upgrade your extension modules!

Cheers,
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New-style icons, .desktop file

[Nick Coghlan]

Andrew Clover wrote:
> Morning!
> 
> I've done some tweaks to the previously-posted-about icon set, taking 
> note of some of the comments here and on -list.

And I still think they're pretty :)

>- emphasised borders of 32x32 version of pycon, and changed text
>  colour, in order to distinguish more from pyc. In balance I think
>  this is enough of a change, though Nick Coghlan's idea of just using
>  the plus in this case also has merit - indeed this is what happens
>  at 16x16

A small problem with including the interactive prompt as part of that icon is 
that it leaves us without an appropriate icon for pythonw (the Windows .exe to 
start Python GUI applications without displaying a console window). The plain 
Plus-Python icon would work in both cases (the normal python executable and 
the pythonw variant).

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unicode vs buffer (array) design issue can crash interpreter

[M.-A. Lemburg]

Martin v. Löwis wrote:
> M.-A. Lemburg wrote:
>> I'd argue that such code is broken anyway: 3rd party code simply
>> cannot make any assumptions on the typedef behind Py_UNICODE.
> 
> The code would work just fine now, but will stop working with the
> change. Users of the code might not be open to arguments.

Fair enough. Let's leave things as they are for 2.4, then.

> In any case, it's up to the release manager to decide.
> 
>> Note that you'd only see this change when compiling Python in the
>> non-standard UCS4 setting on Linux.
> 
> Sure. That happens to be the default on many Linux distributions,
> though.

Unfortunately, that's true.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 14 2006)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New-style icons, .desktop file

[Andrew Clover]

Morning!

I've done some tweaks to the previously-posted-about icon set, taking 
note of some of the comments here and on -list. In particular, amongst 
more minor changes:

   - added egg icon (based on zip)
   - flipped pycon to work better with shortcut arrow
   - emphasised borders of 32x32 version of pycon, and changed text
 colour, in order to distinguish more from pyc. In balance I think
 this is enough of a change, though Nick Coghlan's idea of just using
 the plus in this case also has merit - indeed this is what happens
 at 16x16
   - built .ico files without the Windows Vista enormo-icons. If the
 icons *were* to go in the win32 distribution, it probably wouldn't
 make sense to spend the considerably larger filesize on Vista icons
 until such time as people are actually using Vista.
   - included PNG and SVG version of icons. The SVG unfortunately doesn't
 preserve all the effects of the original Xara files, partly because
 a few effects (feathering, bevels) can't be done in SVG 1.1, and
 partly because the current conversion process is bobbins (but we're
 working on that!). So the nice gradients and things are gone, but it
 should still be of use as a base for anyone wanting to hack on the
 icons.
   - excised file() in favour of open() ;-)

Files and preview here:

   http://doxdesk.com/img/software/py/icons2.zip
   http://doxdesk.com/img/software/py/icons2.png

Oh, and is the intention to deprecate the purple/green horizontal snake 
logo previously used for 'Python for Windows' (as well as PSF)? If so, 
Erik van B's installer graphic could probably do with a quick refresh.

cheers,

-- 
And Clover
mailto:[EMAIL PROTECTED]
http://www.doxdesk.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Procedure for sandbox branches in SVN?

[Fredrik Lundh]

Tim Peters wrote:

> Just do it.  Branches under SVN are cheap, go fast, and are pretty
> easy to work with.  Even better, because a branch is just another
> named directory in SVN's virtual file system, you can "svn remove" it
> when you're done with it (just like any other directory).

footnote: if you want to make huge amounts of branches (e.g. a branch
for each bug or optimization idea you're working on), I recommend using
the SVK frontend instead of SVN:

http://svk.elixus.org/





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unicode vs buffer (array) design issue can crash interpreter

[Martin v. Löwis]

M.-A. Lemburg wrote:
> I'd argue that such code is broken anyway: 3rd party code simply
> cannot make any assumptions on the typedef behind Py_UNICODE.

The code would work just fine now, but will stop working with the
change. Users of the code might not be open to arguments.

In any case, it's up to the release manager to decide.

> Note that you'd only see this change when compiling Python in the
> non-standard UCS4 setting on Linux.

Sure. That happens to be the default on many Linux distributions,
though.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unicode vs buffer (array) design issue can crash interpreter

[M.-A. Lemburg]

Martin v. Löwis wrote:
> Neal Norwitz wrote:
>> I'll leave this decision to Martin or someone else, since I'm not
>> familiar with the ramifications.  Since it was documented as unsigned,
>> I think it's reasonable to consider changing.  Though it could create
>> signed-ness warnings in other modules.  I'm not sure but it's possible
>> it could create problems for C++ compilers since they are pickier.
> 
> My concern is not so much that it becomes unsigned in 2.4.4, but that
> it stops being a typedef for wchar_t on Linux. C++ code that uses that
> assumption might stop compiling.

I'd argue that such code is broken anyway: 3rd party code simply
cannot make any assumptions on the typedef behind Py_UNICODE.

Note that you'd only see this change when compiling Python in the
non-standard UCS4 setting on Linux.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 14 2006)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Procedure for sandbox branches in SVN?

[Martin v. Löwis]

Tim Peters wrote:
> Just do it.  Branches under SVN are cheap, go fast, and are pretty
> easy to work with.  Even better, because a branch is just another
> named directory in SVN's virtual file system, you can "svn remove" it
> when you're done with it (just like any other directory).

That's all true. Of course, removing the branch won't free up
any disk space - it will just remove the branch from the view
(IOW, it is also easy to bring it back if it was removed mistakenly).

Regards,
Martn
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unicode vs buffer (array) design issue can crash interpreter

[Martin v. Löwis]

Neal Norwitz wrote:
> I'll leave this decision to Martin or someone else, since I'm not
> familiar with the ramifications.  Since it was documented as unsigned,
> I think it's reasonable to consider changing.  Though it could create
> signed-ness warnings in other modules.  I'm not sure but it's possible
> it could create problems for C++ compilers since they are pickier.

My concern is not so much that it becomes unsigned in 2.4.4, but that
it stops being a typedef for wchar_t on Linux. C++ code that uses that
assumption might stop compiling.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r45321 - in python/trunk: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS

[Guido van Rossum]

On 4/14/06, Anthony Baxter <[EMAIL PROTECTED]> wrote:
> On Friday 14 April 2006 02:31, Martin v. Löwis wrote:
> > Tim Peters wrote:
> > >> I'm not the one to decide, but at some time the traceback module
> > >> should be rewritten to match the interpreter behavior.
> > >
> > > No argument from me about that.
> >
> > I also think the traceback module should be corrected, and the test
> > cases updated, despite the objections that it may break other
> > people's doctest test cases.

Let me chime in with agreement (2.5 only of course).

> I don't mind one way or the other, but with the number of people
> working actively on the code at the moment, I think reverting broken
> patches that don't have trivial test fixes is the way to go. The
> buildbot system is useless, otherwise.

That was the right thing to do (short of fixing all the missing tests,
which requires actual thinking :-).

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] int vs ssize_t in unicode

[Neal Norwitz]

On 4/13/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Neal Norwitz wrote:
> > I just grepped for INT_MAX and there's a ton of them still (well 83 in
> > */*.c).  Some aren't an issue like posixmodule.c, those are
> > _SC_INT_MAX.  marshal is probably ok, but all uses should be verified.
> >  Really all uses of {INT,LONG}_{MIN,MAX} should be verified and
> > converted to PY_SSIZE_T_{MIN,MAX} as appropriate.
>

BTW, it would be great if someone could try to put together some tests
for bigmem machines.  I'll add it to the todo wiki.  The tests should
be broken up by those that require 2+ GB of memory, those that take
4+, etc.  Many people won't have boxes with that much memory.

The test cases should test all methods (don't forget slicing
operations) at boundary points, particularly just before and after
2GB.  Strings are probably the easiest.  There's unicode too.  lists,
dicts are good but will take more than 16 GB of RAM, so those can be
pushed out some.

I have some machines available for testing.

> I replaced all the trivial ones; the remaining ones are (IMO) more
> involved, or correct. In particular:
>
> - collectionsmodule: deque is still restricted to 2GiB elements
> - cPickle: pickling does not support huge strings (and probably
>   shouldn't); likewise marshal
> - _sre is still limited to INT_MAX completely

I've got outstanding changes somewhere to clean up _sre.

> - not sure why the mbcs codec is restricted to INT_MAX; somebody
>   should check the Win64 API whether the restriction can be
>   removed (most likely, it can)
> - PyObject_CallFunction must be duplicated for PY_SSIZE_T_CLEAN,
>   then exceptions.c can be extended.

My new favorite static analysis tool is grep:

grep '(int)' */*.c | egrep -v 'sizeof(int)' | wc -l
 418

I know a bunch of those aren't problematic, but a bunch are.  Same
with long casts.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 302 support for pydoc, runpy, etc.

[Nick Coghlan]

Phillip J. Eby wrote:
> Anybody have any thoughts, comments, or questions?

This all sounds good to me. Will it also eliminate the current quirk in the 
runpy emulation where it searches the metapath in a slightly different order 
than the actual import machinery does? (From what you wrote, I believe it 
will, but I'm not sure).

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com