Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Terry Reedy]

On 3/12/2011 10:52 PM, Glenn Linderman wrote:

On 3/12/2011 7:21 PM, Terry Reedy wrote:



The safest such character is \0,\


Works fine in Python.


unless you are coding in C,


Then \01 is next best.


I wouldn't have called you on this, except that it really is important
not to give people the idea that you can blithely use a variable length
string anywhere except at the tail of a multi-field sort string.


Sorry, my initial brief comment was for people on this list who I 
assumed understood the issue.



general, you can't. I've long since lost track of the number of times
I've helped people understand the fix to programs that tried that.


Thanks for explaining. I also get fussy about things I have explained 
too many time ;-).


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[skip]

> "Martin" == Martin v Löwis  writes:

Martin> Am 12.03.11 22:36, schrieb s...@pobox.com:
Martin> I don't think any of the regular core committers got any such
Martin> explicit veto powers on any code...
>> 
>> "Veto powers" is your term, not mine.  I suggested that Raymond's opinion
>> should be accorded extra weight.  This isn't the UN Security Council.

Martin> Yes, but you were comparing it with ElementTree, where Fredrik
Martin> *does* have veto powers. I was pointing out that the case at
Martin> hand is different.

I didn't know that, nor did I infer that Fredrik had veto power.  I wrote:

I think we would give reasonably large weight to Fredrik Lundh's opinion
on the change.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Glenn Linderman]

On 3/12/2011 7:21 PM, Terry Reedy wrote:
(Ok, I assumed that the 'word' field does not include any of 
!"#$%&'()*+. If that is not true, replace comma with space or even a 
control char such as '\a' which even precedes \t and \n.)


OK, I agree the above was your worst assumption, although you need to 
add "," to your list also, because that allows for the data puns.


You also rewrote Guido's text from "shortstring" to "word" and assumed 
it had certain content semantics, but since only integer is after the 
",", rsplit would work to separate the fields even if shortstring 
contains ",".


And the choice of delimiter really determines whether data puns can 
exist.  If and only if you know that there is a character that is lower 
in sort order than any of the characters in the sort strings, can you 
"cheat" and put a variable length string into a sort key field, by 
terminating it with such a character.  The safest such character is \0, 
unless you are coding in C, then \a as you now suggest, but only if you 
can be 100% sure it is not found in the data.  If you cannot guarantee 
the data doesn't contain them, there will be the possibility of data 
puns among variable length strings, and the algorithms will sort wrong 
in pathological cases.


I wouldn't have called you on this, except that it really is important 
not to give people the idea that you can blithely use a variable length 
string anywhere except at the tail of a multi-field sort string.  In 
general, you can't.  I've long since lost track of the number of times 
I've helped people understand the fix to programs that tried that.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[Martin v. Löwis]

Am 12.03.11 22:36, schrieb s...@pobox.com:

 Martin>  I don't think any of the regular core committers got any such
 Martin>  explicit veto powers on any code...

"Veto powers" is your term, not mine.  I suggested that Raymond's opinion
should be accorded extra weight.  This isn't the UN Security Council.


Yes, but you were comparing it with ElementTree, where Fredrik *does*
have veto powers. I was pointing out that the case at hand is different.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[skip]

Martin> I don't think any of the regular core committers got any such
Martin> explicit veto powers on any code...

"Veto powers" is your term, not mine.  I suggested that Raymond's opinion
should be accorded extra weight.  This isn't the UN Security Council.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Terry Reedy]

On 3/12/2011 8:47 PM, Glenn Linderman wrote:

On 3/12/2011 2:09 PM, Terry Reedy wrote:

I believe that if the integer field were padded with leading blanks as
needed so that all are the same length, then no key would be needed.


Did you mean that "if the integer field were" converted to string and
"padded with leading blanks..."?


Guido presented a use case of a list a strings, each of form '%s,%d', 
where the %s part is a 'word'. 'Integer field' refers to the part of 
each string after the comma.



Otherwise I'm not sure how to pad an integer with leading blanks.


The integers are already in string form. The *existing* key function his 
colleague used converted that part to an int as the second part of a 
tuple. I presume the integer field was separated by split(','), so the 
code was something like


def sikey(s):
  s,i = s.split(',')
  return s,int(i)

longlist.sort(key=sikey)

It does not matter if the splitting method is more complicated, because 
it is already part of the problem spec. I proposed instead


def sirep(s):
  s,i = s.split(',') # or whatever current key func does
  return '%s,%#s' % (s,i)
  # where appropriate value of # is known from application

longlist = map(sirep, longlist)
longlist.sort()

# or assuming that a simple split is correct

longlist = ['%s,%#s' % tuple(s.split(',')) for s in longlist]
longlist.sort()


Also, what appears to be your revised data structure, strval + ',' +
'%5.5d' % intval , assumes the strval is fixed length, also.


No it does not, and need not. ',' precedes all letters in ascii order. 
(Ok, I assumed that the 'word' field does not include any of 
!"#$%&'()*+. If that is not true, replace comma with space or even a 
control char such as '\a' which even precedes \t and \n.) Given the 
context of Google, I assumed that 'word' meant word, as in a web 
document, while the int might be a position or doc number (or both). The 
important point is that the separator cause all word-int pairs with the 
same word to string-sort before all word-int pairs with the same word + 
a suffix. My example intentionally tested that.



Consider the following strval, intval pairs, using your syntax:

['a,997, 1','a, 1000']

Nothing says the strval wouldn't contain data that look like your
structure...


The problem as presented. 'a,997' is not a word. In any case, as I said 
before, the method of correctly parsing the strings into two fields is 
already specified. I am only suggesting a change in how to proceed 
thereafter.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Raymond Hettinger]

On Mar 12, 2011, at 3:44 PM, Guido van Rossum wrote:

> I was just reminded that in Python 3, list.sort() and sorted() no
> longer support the cmp (comparator) function argument. The reason is
> that the key function argument is always better. But now I have a
> nagging doubt about this:
> 
> I recently advised a Googler who was sorting a large dataset and
> running out of memory. 

. . .

> But in Python 3 this solution is no longer available. How bad is that?
> I'm not sure. But I'd like to at least get the issue out in the open.
> 


Python3.2 should be substantially better in this regard.
It no longer wraps key objects around every input.  Instead, it
sorts two parallel arrays of pointers.   You can thank Daniel
Stutzbach (another Googler I believe) for this effort.


Raymond

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Martin v. Löwis]

[steve@sylar ~]$ python2.7 -m timeit -s "L = [(1,2), (3,4), (0,5),
(9,100), (3,7), (2,8)]" "sorted(L, lambda (p,q),(r,s): cmp(p*s, q*r))"
1 loops, best of 3: 25.1 usec per loop

[steve@sylar ~]$ python2.7 -m timeit -s "L = [(1,2), (3,4), (0,5),
(9,100), (3,7), (2,8)]" -s "from fractions import Fraction" "sorted(L,
key=lambda t: Fraction(*t))"
1000 loops, best of 3: 236 usec per loop


So for a short list, I get a factor of ten difference. For a longer
list, I'd expect the key function to win out. Much to my astonishment,
it doesn't -- I get similar results regardless of the size of L.


This shouldn't be surprising. The cost is primarily in the comparisons,
not in the creation of the Fraction objects. Now, the number of
comparisons won't change whether you use a cmp function or key objects;
the algorithm will compare and switch the same objects in the same
order. So if a Fraction.__lt__ takes seven times as long as a cmp call,
this ratio is preserved even for very long lists.

A lot can be saved if the __lt__ would assume that the other object
is of the same kind:

class Fcomp(tuple):
def __lt__(self, other):
return self[0]*other[1] < self[1]*other[0]

python -m timeit -s "L = [(1,2), (3,4), (0,5), (9,100), (3,7), (2,8)]" 
-s "from fcomp import Fcomp" "sorted(L, key=Fcomp)"


Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About raising NotPortableWarning for CPython specific code

[Terry Reedy]

On 3/12/2011 8:23 PM, Neil Schemenauer wrote:

Greg Ewing  wrote:

So am I. It seems to result from the hisorical mess
of distinguishing between numeric and sequence operations
at the C level but not the Python level. I think CPython
should be moving in the direction of eliminating that
distinction, not expecting all other Python implementations
to copy it.


As a person responsible for part of the mess, I agree.  I would hope
this could be cleaned up in Python 3 without causing too much
trouble for developers.


I opened the issue http://bugs.python.org/issue11477
but I won't be writing the patch ;-).

--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Terry Reedy]

On 3/12/2011 8:28 PM, Steven D'Aprano wrote:

Fredrik Johansson wrote:


Consider sorting a list of pairs representing fractions. This can be
done easily in Python 2.x with the comparison function lambda
(p,q),(r,s): cmp(p*s, q*r). In Python 2.6, this is about 40 times
faster than using fractions.Fraction as a key function.



[steve@sylar ~]$ python2.7 -m timeit -s "L = [(1,2), (3,4), (0,5),
(9,100), (3,7), (2,8)]" "sorted(L, lambda (p,q),(r,s): cmp(p*s, q*r))"
1 loops, best of 3: 25.1 usec per loop

[steve@sylar ~]$ python2.7 -m timeit -s "L = [(1,2), (3,4), (0,5),
(9,100), (3,7), (2,8)]" -s "from fractions import Fraction" "sorted(L,
key=lambda t: Fraction(*t))"
1000 loops, best of 3: 236 usec per loop


So for a short list, I get a factor of ten difference. For a longer
list, I'd expect the key function to win out. Much to my astonishment,
it doesn't -- I get similar results regardless of the size of L.

Size of L key/cmp
== =
6 9.4
600 13.9
6 7.0
600 6.7


Interesting. Comparing two Fractions must do the same thing as the cmp 
function, compare two products, but it does so with a *lot* more overhead:


571 def _richcmp(self, other, op):
572 """Helper for comparison operators, for internal use
574 Implement comparison between a Rational instance and
575 either another Rational instance or a float `other`. If
576 `other` is not a Rational instance or a float, return
577 NotImplemented. `op` should be one of the six standard
578 comparison operators.
580 """
581 # convert other to a Rational instance where reasonable.
582 if isinstance(other, numbers.Rational):
583 return op(self._numerator * other.denominator,
584   self._denominator * other.numerator)
585 if isinstance(other, float):
586 if math.isnan(other) or math.isinf(other):
587 return op(0.0, other)
588 else:
589 return op(self, self.from_float(other))
590 else:
591 return NotImplemented
592
593 def __lt__(a, b):
594 """a < b"""
595 return a._richcmp(b, operator.lt)

For this example, and any with suitably restricted values, key=float 
would work and should beat the cmp version. But of course, someone will 
have a use case for which that will not work. (But then, one could do a 
near linear scan to properly re-sort slices with equal float keys.)


Hmm. As I remember, one rationale for deleting cmp is that nlogn slow 
cmp() calls are slower than n slow key() calls, but that only matters if 
the nlogn '<' comparisions of stored keys are fast compared to cmp calls 
(as tends to be the case when the keys are builtin class instances).


But in this case, they are much slower. To be faster, one would need 
something like "key=lambda p,q:p*(lcm//q)", where lcm is the least 
common multiple of of all the q's (denominators). For the example above, 
lcm = 700. But for longer lists, it could be outrageously large.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] another message to release

[R. David Murray]

I believe the tracker sent an error message to python-dev as a
result of a failed hook execution.  If someone with the
power would release that message so we can see what the error
was, I'd appreciate it :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Glenn Linderman]

On 3/12/2011 2:09 PM, Terry Reedy wrote:
I believe that if the integer field were padded with leading blanks as 
needed so that all are the same length, then no key would be needed. 


Did you mean that "if the integer field were" converted to string and 
"padded with leading blanks..."?


Otherwise I'm not sure how to pad an integer with leading blanks.

Also, what appears to be your revised data structure,   strval + ',' + 
'%5.5d' % intval , assumes the strval is fixed length, also.  Consider  
the following strval, intval pairs, using your syntax:


['a,997,1','a, 1000']

Nothing says the strval wouldn't contain data that look like your 
structure... and just because they were short, didn't mean they were 
fixed length.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Steven D'Aprano]

Fredrik Johansson wrote:


Consider sorting a list of pairs representing fractions. This can be
done easily in Python 2.x with the comparison function lambda
(p,q),(r,s): cmp(p*s, q*r). In Python 2.6, this is about 40 times
faster than using fractions.Fraction as a key function.



[steve@sylar ~]$ python2.7 -m timeit -s "L = [(1,2), (3,4), (0,5), 
(9,100), (3,7), (2,8)]" "sorted(L, lambda (p,q),(r,s): cmp(p*s, q*r))"

1 loops, best of 3: 25.1 usec per loop

[steve@sylar ~]$ python2.7 -m timeit -s "L = [(1,2), (3,4), (0,5), 
(9,100), (3,7), (2,8)]" -s "from fractions import Fraction" "sorted(L, 
key=lambda t: Fraction(*t))"

1000 loops, best of 3: 236 usec per loop


So for a short list, I get a factor of ten difference. For a longer 
list, I'd expect the key function to win out. Much to my astonishment, 
it doesn't -- I get similar results regardless of the size of L.


Size of L   key/cmp
==  =
6   9.4
600 13.9
6   7.0
600 6.7




--
Steven

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Callable, non-descriptor class attributes.

[Steven D'Aprano]

Thomas Wouters wrote:


I would much rather loudly warn people to fix their code, instead of forcing
other implementations (and, more importantly to me personally, future
CPython changes :) to deal with the distinction forever. But if we declare a
wrapper to be the right way to deal with this, let's at least do it right,
not half-assed again. And if we do add the nondescriptor wrapper for that, I
wonder if there are still cases where staticmethods should be callable; it
seems to me it would have no real benefit, and it would lure people into a
false sense of security when mistakenly using staticmethod as the wrapper
(which is, after all, a lot more visible.)



The primary benefit to my mind is two avoid breaking the Principle of 
Least Astonishment (PLA). Given:


class C(object):
@staticmethod
def spam():
pass

spam looks like a callable. It is a callable when referenced via C.spam 
or C().spam. But inside the class block, it isn't callable. That comes 
as an unpleasant surprise to anyone who, like me, has tried to use a 
staticmethod as a helper function during class construction:


class C(object):
@staticmethod
def spam():
pass
result = spam()

That something that looks like a function fails to be callable violates 
PLA. There are work-arounds, but none of them are pretty or obvious:


# the underlying function doesn't seem to be available directly
result = spam.__get__(object())


or something like this:

class C(object):
def spam():
pass
result = spam()
spam = staticmethod(spam)

Making staticmethod callable would, in my opinion, be the obvious way to 
do it.



--
Steven

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About raising NotPortableWarning for CPython specific code

[Neil Schemenauer]

Greg Ewing  wrote:
> So am I. It seems to result from the hisorical mess
> of distinguishing between numeric and sequence operations
> at the C level but not the Python level. I think CPython
> should be moving in the direction of eliminating that
> distinction, not expecting all other Python implementations
> to copy it.

As a person responsible for part of the mess, I agree.  I would hope
this could be cleaned up in Python 3 without causing too much
trouble for developers.

  Neil

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Terry Reedy]

On 3/12/2011 5:09 PM, Reid Kleckner wrote:

On Sat, Mar 12, 2011 at 4:58 PM, Nick Coghlan  wrote:

On Sat, Mar 12, 2011 at 4:50 PM, Reid Kleckner  wrote:

They should be able to use a slotted cmp_to_key style class:
http://docs.python.org/howto/sorting.html

That will allocate 1 Python object with no dict per key, but that
might not be good enough.


Tuples are already slotted, so that isn't likely to help in this case.


It's three allocations vs. one.  The first is tuple + str + int, while
the adapter is just one object.  I'm not sure how it eventually shakes
out, though.

That said, it's still worse than Python 2, which is zero allocations.  :)


And revising the data so that no key and no cmp function is needed is 
zero allocations and faster. See my other post.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[Martin v. Löwis]

Maybe, but we have historically tended to give some extra weight to the
primary author of at least modules and packages.  If someone wanted to make
a significant change to xml.etree, I think we would give reasonably large
weight to Fredrik Lundh's opinion on the change.  If the peephole optimizer
is largely Raymond's work, why should that be treated any differently?  Is
it just because it can't practically be distributed outside of Python proper
the way ElementTree can?


These cases are certainly different. For Elementtree, /F has explicitly
asked that it be included into Python only under the condition that he
has the last say in all changes, and that the only exception to this
would be urgent security fixes, or systematic changes that apply to all
modules.

Whether such a privilege should have been granted in the first place is
a different question, but I personally feel obliged to honor it, until
some of the involved parties change their minds.

I don't think any of the regular core committers got any such explicit
veto powers on any code, and I don't think it should be granted to
anybody.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[skip]

Raymond> The social question:  if the person who designed, implemented,
Raymond> and maintained the optimizer recommends against a patch and
Raymond> another committer just checks it in anyway, do we care?

Guido> - you're dangerously close here to putting your ego ahead of
Guido>   things

Maybe, but we have historically tended to give some extra weight to the
primary author of at least modules and packages.  If someone wanted to make
a significant change to xml.etree, I think we would give reasonably large
weight to Fredrik Lundh's opinion on the change.  If the peephole optimizer
is largely Raymond's work, why should that be treated any differently?  Is
it just because it can't practically be distributed outside of Python proper
the way ElementTree can?

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Callable, non-descriptor class attributes.

[Terrence Cole]

On Sat, 2011-03-12 at 12:49 +1300, Greg Ewing wrote:
> Thomas Wouters wrote:
> 
> >  2. Make CFunctions turn into methods in CPython (after a period of 
> > warning about the impending change, obviously.) The actual *usecase* for 
> > this is hard to envision
> 
> While not necessary for the case being discussed here, this would
> be a big help for Pyrex and Cython, where currently some awkward
> things have to be done to get a Python class with methods implemented
> in C.

Just so.  I implemented a new C-level function type for Melano[1] just 2
days ago so that I can have the descriptor slot set (and get
__defaults__, __kw_defaults__, __annotations__, and any other missing
details from PyFunction that I want or need).

[1] https://github.com/terrence2/melano

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Fredrik Johansson]

On Sun, Mar 13, 2011 at 12:41 AM, "Martin v. Löwis"  wrote:
> Am 12.03.11 18:00, schrieb Glenn Linderman:
>>
>>  On 3/12/2011 1:55 PM, Fredrik Johansson wrote:
>>>
>>> Consider sorting a list of pairs representing fractions. This can be
>>> done easily in Python 2.x with the comparison function lambda
>>> (p,q),(r,s): cmp(p*s, q*r). In Python 2.6, this is about 40 times
>>> faster than using fractions.Fraction as a key function.
>>
>> Am I correct in concluding that various ideas to eliminate or limit the
>> size of the key= cache would not help this use case at all?
>
> That's correct. However, there is a straight-forward day of getting
> the same comparison algorithm with the cmp_to_key class in 3.x.
> Fredrik classified this as "ugly and slow"; I'm not sure where this
> classification comes from.

It's ugly because it involves creating a class wrapping a comparison
function, or importing some obscure magic from the standard library,
instead of just using a comparison function.

It's slow because it's slow (and less memory-efficient). Even with
cmp_to_key, the example with fractions is still 2.6 times slower than
with a direct cmp function.

Fredrik
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)

[Allison Randal]

On 03/12/2011 05:13 PM, Glenn Linderman wrote:

On 3/12/2011 10:42 AM, Allison Randal wrote:

I might convert it directly into a Q&A blog post.


I'd like to see that, or a summary, posted here. As a Perl-to-Python
convertee, I'm curious about the problematic semantic differences.


The short summary is that the Parrot VM is a very good semantic fit for 
Python (AFAICT, a better fit than it is for Perl 6, though I haven't 
done the feature-by-feature comparison). But, the initial implementation 
of Pynie (Python-on-Parrot) used NQP for parsing Python syntax and 
compiling it down to Parrot's AST. NQP is a library of Perl 6 regexes 
plus a stripped-down version of Perl 6 syntax, roughly the same general 
idea as PyPy's RPython. As with any Turing-complete system, it is 
certainly possible to implement a Python parser in NQP, but it required 
some ugly hacks, and it got to the point that I just hated working on it.


To be fair, the failure wasn't entirely technological, the project was 
also slowed by my PhD work, and by my new very intense and very 
interesting job.


I'd like to see Python-on-Parrot implementation work proceed in a more 
naturally Pythonic way, possibly as an alternate backend for PyPy or as 
an alternate output from the CPython grammar. I have a couple of 
volunteers who I'll be mentoring in the coming months, so we'll see what 
grows out of it.


Allison
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Martin v. Löwis]

Am 12.03.11 18:00, schrieb Glenn Linderman:

  On 3/12/2011 1:55 PM, Fredrik Johansson wrote:

Consider sorting a list of pairs representing fractions. This can be
done easily in Python 2.x with the comparison function lambda
(p,q),(r,s): cmp(p*s, q*r). In Python 2.6, this is about 40 times
faster than using fractions.Fraction as a key function.


Am I correct in concluding that various ideas to eliminate or limit the
size of the key= cache would not help this use case at all?


That's correct. However, there is a straight-forward day of getting
the same comparison algorithm with the cmp_to_key class in 3.x.
Fredrik classified this as "ugly and slow"; I'm not sure where this
classification comes from.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About raising NotPortableWarning for CPython specific code

[Greg Ewing]

Nick Coghlan wrote:


I'm actually tempted to call the current CPython semantics flatout
*wrong*.


So am I. It seems to result from the hisorical mess
of distinguishing between numeric and sequence operations
at the C level but not the Python level. I think CPython
should be moving in the direction of eliminating that
distinction, not expecting all other Python implementations
to copy it.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)

[Nick Coghlan]

On Sat, Mar 12, 2011 at 6:04 PM, Glenn Linderman  wrote:
> On 3/12/2011 3:43 AM, Nick Coghlan wrote:
>
> I posted my rough notes and additional write-ups for Wednesday's VM
> summit and Thursday's language summit:
>
> http://www.boredomandlaziness.org/2011/03/python-vm-summit-rough-notes.html
>
> 2.7 to 3.2
>   - treat PyPy Python 3 dialect like a major Python library (e.g. sponsored
> by PSF)
>
> Can someone expound on what this means?  (context in above link)

The context where this came up was the fact that the PSF has the
ability to fund specific, concrete proposals targeted at porting key
libraries to Python 3 (e.g. this already happened with PyOpenSSL). The
structure of PyPy means that Python 3 support is about implementing a
new front end without needing to change the backend (the latter
includes the JIT, etc), so it was mentioned as something the PyPy
folks might want to consider seeking funding for.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Introductions

[Ned Deily]

In article <1299940862.1632.7.camel@hobo>,
 Ross Lagerwall  wrote:
> I have been offered commit rights for Python after making a few patches
> on subprocess and the os module.
> 
> Antoine suggested that I should introduce myself on the python-dev list
> so here we go:
> 
> I am a student from South Africa and decided to do some work on Python
> in my spare time. I think I chose the Python project because it
> definitely seems to be one of the friendliest projects around and the
> easiest to get started with - the dev guide helps a lot! Also, with the
> big standard library, there is place for people with all different
> levels of experience to contribute.

Welcome, Ross!  2011 is turning out to be a good year for new 
committers, if I do say so myself (and being one myself).  I hope you 
get the opportunity to meet others here in person, too, to hear and see 
our barks.

It was good to meet some more of you all in person, if all too briefly, 
at the language summit on Thursday.  Alas, I was unable to stay for the 
rest of PyCon but Ronald and I had a chance to spend some quality time 
together again brainstorming issues in the OS X area.  Onward!

-- 
 Ned Deily,
 n...@acm.org

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Éric Araujo]

>> You can’t combine the -r option with other options.  (Yes, it’s a known
>> bug.)
> Are you sure? It just worked here: [...]
> (perhaps you're thinking of -R instead?)

Exactly.

>> On an unrelated note, you can use “-r .” to tell Mercurial to find the
>> branch name from the working directory instead of having to remember and
>> retype it.
> Does it work with merges?

It does.  hg help revs:

The reserved name "." indicates the working directory parent. If no
working directory is checked out, it is equivalent to null. If an
uncommitted merge is in progress, "." is the revision of the first
parent.

Gotta love consistency, DTRT, Just Working and all that :)

Regards
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)

[Glenn Linderman]

On 3/12/2011 3:43 AM, Nick Coghlan wrote:

I posted my rough notes and additional write-ups for Wednesday's VM
summit and Thursday's language summit:

http://www.boredomandlaziness.org/2011/03/python-vm-summit-rough-notes.html



2.7 to 3.2
  - treat PyPy Python 3 dialect like a major Python library (e.g. 
sponsored by PSF)


Can someone expound on what this means?  (context in above link)

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Callable, non-descriptor class attributes.

[Thomas Wouters]

On Sat, Mar 12, 2011 at 17:08, Greg Ewing wrote:

> Guido van Rossum wrote:
>
>  IIUC Thomas found that this breaks some current use of staticmethod.
>>
>
> I*I*UC, it wasn't making it callable that was the problem,
> it was changing the behaviour so that the staticmethod
> wrapper returns itself instead of the underlying object
> when accessed as a descriptor.
>
> The reason for doing *that* appears to be so that you
> can transplant a staticmethod-wrapped object from one
> class to another without surprises. But is that really
> a common enough thing to do that it warrants special
> attention? Would it be so bad to require applying
> staticmethod() again in such a situation?
>

There's two different things in play: the traditional use of staticmethod()
when you are storing something that isn't in your control as a class
attribute, and wrapping a Python function to have it be treated like a
PyCFunction, where you have no control (and no desire to control) what
happens with the function. Of course, if you do contor both the 'consumer'
and the 'producer' side, you can fix it either way you want.

For the emulate-builtins wrapper to do a proper job (and not leave subtle
differences lying around), the wrapper should not be a descriptor (or it
should return itself from __get__.) I suspect people will also want it to
expose the original function's attributes, like bound method wrappers do. If
we just have staticmethods be callable and tell people to use that, there
will still be cases where PyCFunctions and wrapped Python functions do
different things. But changing staticmethod to do these things would make
code that relies on staticmethod simply encapsulating objects (like, all my
own uses of staticmethod) break in fascinating ways. (I know, I tried.) And
I don't think having staticmethod do something different depending on the
type of object it's passed is worth considering.

I would much rather loudly warn people to fix their code, instead of forcing
other implementations (and, more importantly to me personally, future
CPython changes :) to deal with the distinction forever. But if we declare a
wrapper to be the right way to deal with this, let's at least do it right,
not half-assed again. And if we do add the nondescriptor wrapper for that, I
wonder if there are still cases where staticmethods should be callable; it
seems to me it would have no real benefit, and it would lure people into a
false sense of security when mistakenly using staticmethod as the wrapper
(which is, after all, a lot more visible.)

-- 
Thomas Wouters 

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Antoine Pitrou]

On Sat, 12 Mar 2011 23:43:52 +0100
Éric Araujo  wrote:
> > hg revert -ar default
> 
> You can’t combine the -r option with other options.  (Yes, it’s a known
> bug.)

Are you sure? It just worked here:

$ hg rev -ar default
reverting README

(perhaps you're thinking of -R instead?)

> On an unrelated note, you can use “-r .” to tell Mercurial to find the
> branch name from the working directory instead of having to remember and
> retype it.

Does it work with merges?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Glenn Linderman]

On 3/12/2011 1:55 PM, Fredrik Johansson wrote:

Consider sorting a list of pairs representing fractions. This can be
done easily in Python 2.x with the comparison function lambda
(p,q),(r,s): cmp(p*s, q*r). In Python 2.6, this is about 40 times
faster than using fractions.Fraction as a key function.


Am I correct in concluding that various ideas to eliminate or limit the 
size of the key= cache would not help this use case at all?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Nadeem Vawda]

On Sun, Mar 13, 2011 at 12:43 AM, Éric Araujo  wrote:
>> hg revert -ar default
>
> You can’t combine the -r option with other options.  (Yes, it’s a known bug.)

It seems to work for me (Mercurial 1.6.3 on Ubuntu). But I suppose it
wouldn't hurt
to split the options up.

Regards,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Martin v. Löwis]

But in Python 3 this solution is no longer available. How bad is that?
I'm not sure. But I'd like to at least get the issue out in the open.


Rather than reintroducing cmp=, I'd add a cached=True parameter.
If this is set to False, the key results wouldn't be put into a
list, but recreated every time two objects need to be compared.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Nick Coghlan]

On Sat, Mar 12, 2011 at 5:41 PM, "Martin v. Löwis"  wrote:
> Why not? IIUC, the current key function creates three objects: the tuple,
> the short string, and the int. With the class

Yeah, I misread the example. Using cmp_to_key would indeed save quite
a lot of memory in this case.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Éric Araujo]

> hg revert -ar default

You can’t combine the -r option with other options.  (Yes, it’s a known
bug.)

On an unrelated note, you can use “-r .” to tell Mercurial to find the
branch name from the working directory instead of having to remember and
retype it.

Regards
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Peter Otten]

Guido van Rossum wrote:

> I was just reminded that in Python 3, list.sort() and sorted() no
> longer support the cmp (comparator) function argument. The reason is
> that the key function argument is always better. But now I have a
> nagging doubt about this:
> 
> I recently advised a Googler who was sorting a large dataset and
> running out of memory. My analysis of the situation was that he was
> sorting a huge list of short lines of the form "shortstring,integer"
> with a key function that returned a tuple of the form ("shortstring",
> integer). Using the key function argument, in addition to N short
> string objects, this creates N tuples of length 2, N more slightly
> shorter string objects, and N integer objects. (Not to count a
> parallel array of N more pointers.) Given the object overhead, this
> dramatically increased the memory usage. It so happens that in this
> particular Googler's situation, memory is constrained but CPU time is
> not, and it would be better to parse the strings over and over again
> in a comparator function.
> 
> But in Python 3 this solution is no longer available. How bad is that?
> I'm not sure. But I'd like to at least get the issue out in the open.

While there are other arguments to reintroduce cmp (or less_than instead?) 
the memory problem could also be addressed with a dont_cache_keys flag or 
max_cache_keys limit.

Peter

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)

[Benjamin Peterson]

2011/3/12 Maciej Fijalkowski :
> On Sat, Mar 12, 2011 at 1:14 PM, "Martin v. Löwis"  wrote:
>>>
>>> http://www.boredomandlaziness.org/2011/03/python-vm-summit-rough-notes.html
>>>
>>> http://www.boredomandlaziness.org/2011/03/python-vm-summit-somewhat-coherent.html
>>
>> Wrt. the remark that other implementations should be referenced more
>> prominently: I added them to
>>
>> http://www.python.org/download/
>
> Cool. Would you mind mentioning JIT for PyPy in description?

Done.



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Martin v. Löwis]

Am 12.03.11 16:58, schrieb Nick Coghlan:

On Sat, Mar 12, 2011 at 4:50 PM, Reid Kleckner  wrote:

They should be able to use a slotted cmp_to_key style class:
http://docs.python.org/howto/sorting.html

That will allocate 1 Python object with no dict per key, but that
might not be good enough.


Tuples are already slotted, so that isn't likely to help in this case.


Why not? IIUC, the current key function creates three objects: the 
tuple, the short string, and the int. With the class


class cmp_to_key:
  __slots__=['obj']
  def __init__(self, obj):
  self.obj = obj
  def __lt__(self):
  ...

you would only create a single object, so you save the string and the 
integer. In addition, on a 64-bit system, the size of a cmp_to_key

instance is 56 bytes, whereas a two-tuple is 72 bytes, so you also
save 16 bytes per object. Whether that would already create a sufficient
reduction for the case at hand, I don't know.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Antoine Pitrou]

On Sun, 13 Mar 2011 00:24:07 +0200
Nadeem Vawda  wrote:
> On Sat, Mar 12, 2011 at 1:29 PM, "Martin v. Löwis"  wrote:
> > Isn't that command correct only if you are merging into default?
> > ISTM that it should be "hg revert -ar ".
> 
> In general, yes. However, the existing text refers specifically to the
> case of merging 3.2
> into default, so I was trying to be consistent with that.
> 
> The current text, after my proposed change, will be:
> 
> hg update default
> hg merge 3.2
> hg revert -ar default
> hg commit

Thanks for pointing this out. I've fixed the error.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Eugene Toder]

Can sort have an option (and/or try to figure it itself) to calculate
key for every comparison instead of caching them? This will have the
same memory requirements as with cmp, but doesn't require rewriting
code if you decide to trade speed for memory. Will this be much slower
than with cmp?

If going that route sort can also cache a limited amount of keys
(instead of all or nothing), using, for example, a LRU cache with
fixed size.

Eugene
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Nadeem Vawda]

On Sat, Mar 12, 2011 at 1:29 PM, "Martin v. Löwis"  wrote:
> Isn't that command correct only if you are merging into default?
> ISTM that it should be "hg revert -ar ".

In general, yes. However, the existing text refers specifically to the
case of merging 3.2
into default, so I was trying to be consistent with that.

The current text, after my proposed change, will be:

hg update default
hg merge 3.2
hg revert -ar default
hg commit

For full generality, you might want to say:

hg update 
hg merge 
hg revert -ar 
hg commit

However, the entire section on forward-porting is framed in terms of
concrete examples
(and that specific subsection refers explicitly to the "3.2 ->
default" case). The existing
text makes sense to me, so I don't see any need to change this.

Regards,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)

[Glenn Linderman]

On 3/12/2011 10:42 AM, Allison Randal wrote:
I might convert it directly into a Q&A blog post. 


I'd like to see that, or a summary, posted here.  As a Perl-to-Python 
convertee, I'm curious about the problematic semantic differences.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Reid Kleckner]

On Sat, Mar 12, 2011 at 4:58 PM, Nick Coghlan  wrote:
> On Sat, Mar 12, 2011 at 4:50 PM, Reid Kleckner  
> wrote:
>> They should be able to use a slotted cmp_to_key style class:
>> http://docs.python.org/howto/sorting.html
>>
>> That will allocate 1 Python object with no dict per key, but that
>> might not be good enough.
>
> Tuples are already slotted, so that isn't likely to help in this case.

It's three allocations vs. one.  The first is tuple + str + int, while
the adapter is just one object.  I'm not sure how it eventually shakes
out, though.

That said, it's still worse than Python 2, which is zero allocations.  :)

Reid
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Terry Reedy]

On 3/12/2011 3:44 PM, Guido van Rossum wrote:

I was just reminded that in Python 3, list.sort() and sorted() no
longer support the cmp (comparator) function argument. The reason is
that the key function argument is always better. But now I have a
nagging doubt about this:

I recently advised a Googler who was sorting a large dataset and
running out of memory. My analysis of the situation was that he was
sorting a huge list of short lines of the form "shortstring,integer"
with a key function that returned a tuple of the form ("shortstring",
integer).


I believe that if the integer field were padded with leading blanks as 
needed so that all are the same length, then no key would be needed.


ll = ['a,1', 'ab,3', 'a,1', 'a,  111']
ll.sort()
print(ll)
>>>
['a,1', 'a,  111', 'a,1', 'ab,3']

If most ints are near the max len, this would add little space, and be 
even faster than with the key.


> Using the key function argument, in addition to N short

string objects, this creates N tuples of length 2, N more slightly
shorter string objects, and N integer objects. (Not to count a
parallel array of N more pointers.) Given the object overhead, this
dramatically increased the memory usage. It so happens that in this
particular Googler's situation, memory is constrained but CPU time is
not, and it would be better to parse the strings over and over again
in a comparator function.


Was 3.2 used? It has a patch that reduces the extra memory that might 
not be in the last 3.1 release.



But in Python 3 this solution is no longer available. How bad is that?
I'm not sure. But I'd like to at least get the issue out in the open.


This removal has been one of the more contentious issues about (not) 
using 3.x. I believe Raymond had been more involved in the defense of 
the decision than I. However, the discussion/complaint has mostly been 
about the relative difficulty of writing a key function versus a compare 
function.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Callable, non-descriptor class attributes.

[Greg Ewing]

Guido van Rossum wrote:


IIUC Thomas found that this breaks some current use of staticmethod.


I*I*UC, it wasn't making it callable that was the problem,
it was changing the behaviour so that the staticmethod
wrapper returns itself instead of the underlying object
when accessed as a descriptor.

The reason for doing *that* appears to be so that you
can transplant a staticmethod-wrapped object from one
class to another without surprises. But is that really
a common enough thing to do that it warrants special
attention? Would it be so bad to require applying
staticmethod() again in such a situation?

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Nick Coghlan]

On Sat, Mar 12, 2011 at 4:50 PM, Reid Kleckner  wrote:
> They should be able to use a slotted cmp_to_key style class:
> http://docs.python.org/howto/sorting.html
>
> That will allocate 1 Python object with no dict per key, but that
> might not be good enough.

Tuples are already slotted, so that isn't likely to help in this case.

The basic point is that cmp vs key is actually a classic space vs
speed trade-off, and by removing the "cmp" option, we prevent people
from making that trade-off for themselves.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Fredrik Johansson]

On Sat, Mar 12, 2011 at 9:44 PM, Guido van Rossum  wrote:
> I was just reminded that in Python 3, list.sort() and sorted() no
> longer support the cmp (comparator) function argument. The reason is
> that the key function argument is always better. But now I have a
> nagging doubt about this:
>
> I recently advised a Googler who was sorting a large dataset and
> running out of memory. My analysis of the situation was that he was
> sorting a huge list of short lines of the form "shortstring,integer"
> with a key function that returned a tuple of the form ("shortstring",
> integer). Using the key function argument, in addition to N short
> string objects, this creates N tuples of length 2, N more slightly
> shorter string objects, and N integer objects. (Not to count a
> parallel array of N more pointers.) Given the object overhead, this
> dramatically increased the memory usage. It so happens that in this
> particular Googler's situation, memory is constrained but CPU time is
> not, and it would be better to parse the strings over and over again
> in a comparator function.
>
> But in Python 3 this solution is no longer available. How bad is that?
> I'm not sure. But I'd like to at least get the issue out in the open.

The removal of cmp and especially sort(cmp=...) was probably my least
favorite change in Python 3.

Sorting with a key only makes sense when the data can be mapped onto
builtin types (typically ints, strings, and tuples thereof) in an
order-preserving way. When this is not possible, you have to map it
onto a custom type and effectively implement a cmp function via the
comparison methods of that type. This is an ugly and slow substitute
for something that used to be elegant and fast.

Consider sorting a list of pairs representing fractions. This can be
done easily in Python 2.x with the comparison function lambda
(p,q),(r,s): cmp(p*s, q*r). In Python 2.6, this is about 40 times
faster than using fractions.Fraction as a key function.

Fredrik
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Reid Kleckner]

They should be able to use a slotted cmp_to_key style class:
http://docs.python.org/howto/sorting.html

That will allocate 1 Python object with no dict per key, but that
might not be good enough.

Reid

On Sat, Mar 12, 2011 at 3:44 PM, Guido van Rossum  wrote:
> I was just reminded that in Python 3, list.sort() and sorted() no
> longer support the cmp (comparator) function argument. The reason is
> that the key function argument is always better. But now I have a
> nagging doubt about this:
>
> I recently advised a Googler who was sorting a large dataset and
> running out of memory. My analysis of the situation was that he was
> sorting a huge list of short lines of the form "shortstring,integer"
> with a key function that returned a tuple of the form ("shortstring",
> integer). Using the key function argument, in addition to N short
> string objects, this creates N tuples of length 2, N more slightly
> shorter string objects, and N integer objects. (Not to count a
> parallel array of N more pointers.) Given the object overhead, this
> dramatically increased the memory usage. It so happens that in this
> particular Googler's situation, memory is constrained but CPU time is
> not, and it would be better to parse the strings over and over again
> in a comparator function.
>
> But in Python 3 this solution is no longer available. How bad is that?
> I'm not sure. But I'd like to at least get the issue out in the open.
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/reid.kleckner%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[Antoine Pitrou]

On Sat, 12 Mar 2011 13:08:26 -0500
Raymond Hettinger  wrote:
> I would like to withdraw my suggestion for the recursive constant folding 
> patch to be reverted.  I value social harmony much more than a decision about 
> whether a particular patch is a good idea.  I apologize to anyone who is 
> upset over the discussion.

Thank you Raymond.
Of course, if there turns out to be a technical issue that can't
be fixed, the patch should still be backed out.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About raising NotPortableWarning for CPython specific code

[Terry Reedy]

On 3/12/2011 8:33 AM, Laura Creighton wrote:


The thread with the whole gory details begins here:
http://codespeak.net/pipermail/pypy-dev/2011q1/006958.html


The second, multiplication issue does appears to be the same issue.
Augmenting my previous test:

class C(object):
def __iter__(self):
yield 'yes!'
def __radd__(self, other):
other.append('bug!')
return other
def __rmul__(self, other):
other *= 2
return other
def __index__(self):
  return 3

class L(list):
def __iadd__(self, other):
list.__iadd__(self,other)
return self
def __mul__(self, other):
return L(list.__imul__(self,other))

z1, z2, c = [], L(), C()
z1 += c
z2 += c
print(z1, z2, [1]*c, L([1])*c)
>>>
['bug!'] ['yes!'] [1, 1] [1, 1, 1] # PyPy prints ['yes!'], [1, 1, 1]

I opened http://bugs.python.org/issue11477

--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[Antoine Pitrou]

Hello,

> Finally: There appears to be some disagreement on who said what, in
> particular Raymond claims that he told Antoine not to commit whereas
> Antoine claims he did not hear this feedback. I'm guessing it happened
> in IRC (#python-dev), which is intentionally not logged anywhere.

Raymond did tell me he was objecting on a patch, but unless I
misunderstood him he was talking about http://bugs.python.org/issue11462
(which he indeed closed later), not the present issue.

I agree IRC is a poor platform for making decisions (and I've probably
been guilty of this too).

Regards

Antoine.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About raising NotPortableWarning for CPython specific code

[Terry Reedy]

On 3/12/2011 3:04 PM, Thomas Wouters wrote:


It should be fixed, yes, but breaking existing code is going to piss off
a lot of people (like me) who already have enough worries when upgrading
Python. It is apparent that there *is* code out there that relies on
this behaviour, we shouldn't break it wlily-nilly. A warning when
CPython encounters the case that will change behaviour in the next
CPython release and in other implementations is almost as good as
changing the behaviour.


Every bug fix breaks code that depends on the bug. That said, we do not 
backport every bug fix. I think in the long run it would be good to fix 
at least 2.7 and 3.2, but that could be debated.


Greg Price, who started the thread wanting PyPy to replicate the 
somewhat undefined and crazy behavior of CPython, admits
"I think a strict reading of the language reference would clearly 
identify this as wrong behavior by CPython."

http://codespeak.net/pipermail/pypy-dev/2011q1/006966.html
So he can hardly complain if we fix the 'wrong behavior'.

He is not a newbie: he is the one who diagnosed why CPython behaves as 
it does. He also knows how to fix their code: simply add to each 
appended class an __iter__ method just like I did, except yield the same 
thing as the __radd__ method appends rather than something different. 
With such a fix, his software would work with buggy and fixed CPython as 
well PyPy. He should just do that.


He apparently is not the original author. Too bad *that* programmer did 
not recognize and report that he had discovered a bug. I doubt that too 
many other people have encountered it because I expect someone would 
have posted something.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python3 regret about deleting list.sort(cmp=...)

[Guido van Rossum]

I was just reminded that in Python 3, list.sort() and sorted() no
longer support the cmp (comparator) function argument. The reason is
that the key function argument is always better. But now I have a
nagging doubt about this:

I recently advised a Googler who was sorting a large dataset and
running out of memory. My analysis of the situation was that he was
sorting a huge list of short lines of the form "shortstring,integer"
with a key function that returned a tuple of the form ("shortstring",
integer). Using the key function argument, in addition to N short
string objects, this creates N tuples of length 2, N more slightly
shorter string objects, and N integer objects. (Not to count a
parallel array of N more pointers.) Given the object overhead, this
dramatically increased the memory usage. It so happens that in this
particular Googler's situation, memory is constrained but CPU time is
not, and it would be better to parse the strings over and over again
in a comparator function.

But in Python 3 this solution is no longer available. How bad is that?
I'm not sure. But I'd like to at least get the issue out in the open.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About raising NotPortableWarning for CPython specific code

[Thomas Wouters]

On Sat, Mar 12, 2011 at 14:52, Terry Reedy  wrote:

> On 3/12/2011 8:59 AM, Nick Coghlan wrote:
>
>> (obviously, history now means that changing our behaviour would
>
> require a deprecation period)
>>
>
> I disagree. Behavior that contradicts intent and doc is a bug and should be
> fixed.


It should be fixed, yes, but breaking existing code is going to piss off a
lot of people (like me) who already have enough worries when upgrading
Python. It is apparent that there *is* code out there that relies on this
behaviour, we shouldn't break it wlily-nilly. A warning when CPython
encounters the case that will change behaviour in the next CPython release
and in other implementations is almost as good as changing the behaviour.

-- 
Thomas Wouters 

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About raising NotPortableWarning for CPython specific code

[Terry Reedy]

On 3/12/2011 8:59 AM, Nick Coghlan wrote:

On Sat, Mar 12, 2011 at 8:33 AM, Laura Creighton  wrote:

For those of you not at the Language Summit at PyCON the day before yesterday,
there was talk of identifying non-portable behaviour, such as relying on
CPython's reference counting garbage collector to close files for you as
soon as they become unreachable.  And then warning about them.

We have a real live user who has a large code base that relies on
the CPython behaviour that an object's __radd__ method will take precedence
over a list's inplace add behaviour.

The thread with the whole gory details begins here:
http://codespeak.net/pipermail/pypy-dev/2011q1/006958.html

My inclination is to declare code that relies on this as broken, rather
than patch every wretched container type in PyPy.  Can this become
blessed as a 'you shouldn't have done this'?


Given that the meat of the difference in semantics lies in the
CPython-specific distinction between nb_add and sq_concat, I'm
inclined to agree with you.


Me too. Consider filing a tracker issue with the += and * examples from 
the link above as new unit tests.



I'm actually tempted to call the current CPython semantics flatout
*wrong*. When a given operation has multiple C level slots, shouldn't
we be checking all the LHS slots before checking any of the RHS slots?


Yes.


There's nothing in the language spec that I'm aware of that justifies
us doing otherwise.


The current CPython behavior strikes me as a buggy optimization. If 
class 'list' were written in Python, it would have an '__iadd__' method 
that would implement +=. No question. I gather that this is what PyPy 
(correctly) does. The CPython implementation makes a (false) distinction 
between numeric + and * and sequence + and * that does not exist in 
Python the language. The reference manual only lists __add__, and not 
__nadd__ and __sadd__ as separate special methods. The fact that CPython 
has both (with different names) should *not* leak into Python semantics 
and behavior.



(obviously, history now means that changing our behaviour would
require a deprecation period)


I disagree. Behavior that contradicts intent and doc is a bug and should 
be fixed. Otherwise, the doc should be changed to say that for 
C-implmented objects, the precedence between forward and reverse methods 
depends on the category of object. This would mean that subclasses of 
builtins *should* act differently from their parent, as they can now. 
Example derived from the link:


# 3.2
class C(object):
def __iter__(self):
yield 'yes!'
def __radd__(self, other):
other.append('bug!')
return other

class L(list):
def __iadd__(self, other):
list.__iadd__(self,other)
return self

z1,z2 = [], L()
z1 += C()
z2 += C()
print(z1, z2)

>>>
['bug!'] ['yes!']

Such a change would make the definition of the language more 
C-dependent, which is what Laura is rightly complaining about, rather 
than less. It would also work against efforts to make Python and C 
versions of stdlib modules work the same and pass the same tests.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[Guido van Rossum]

On Sat, Mar 12, 2011 at 11:05 AM, Raymond Hettinger
 wrote:
> There are separate social, strategic, and tactical questions.
>
> The social question:  if the person who designed, implemented, and maintained 
> the optimizer recommends against a patch and another committer just checks it 
> in anyway, do we care?  I've taken responsibility for this code and have 
> treated it will a good deal of care.  I don't believe the patch should go in 
> because the risk/reward ratio is too low and because a better solution exists.

Hm. A few comments:

- you're dangerously close here to putting your ego ahead of things
- further review of the patch may give a better understanding of the risk
- a "better solution" that doesn't exist yet shouldn't be too strong
an argument against "good enough"

> The strategic question:  constant folding in the peephole optimizer pre-dates 
> the AST branch.  When AST landed, there was general agreement among myself 
> and those involved that any future optimizations which could be done with the 
> AST, should be done there.  It is the correct technological solution.
>
> At one point, we had strategic agreement to stop building-out the peepholer 
> in any significant way.   In fairness to Antoine, the conversations took 
> place several years before he became a committer.  The strategic question is 
> do we want to come to that agreement again?  Can we agree to not have 
> significant changes to the peepholer, to treat it with great care and 
> restraint?   This patch doesn't have to go in.  Py3.3 won't be out for 18 to 
> 24 months anyway.  There is a lot of time to do the job right and not add 
> weight to a house of cards.

Feel free to go ahead and start it. Write a PEP, or just write a patch
and get people to review and understand it. If it's successful, we can
delete peephole.c and the effect of Antoine's checkin will be
nullified. If it's not successful, Antoine's code has the advantage of
possibly being good enough, and will have had those same 18-24 months
to have its tires kicked a bit.

> The tactical question:  is this particular patch correct?  Maybe it is.  I 
> don't know, I didn't get past all the new macros.  I do know that I could 
> have whipped up something similar years ago but chose not to based on advice 
> from Thomas, Andrew, Tim, and Guido.   The only change since then is that 
> there is a newer committer doesn't buy into that reasoning.

A recurring problem I hear about is that there is too much resistance
to new ideas coming from new developers. Clearly new developers may
not be aware of where the skeletons are hiding. But just as clearly
the old guard isn't always right. Antoine is by now not a newbie any
more. So please consider Antoine's patch on its own merit and not just
from the perspective of a decision made years ago.

> Constant folding is the most complex part of the optimizer.  Why would we add 
> to it, when a better and more reliable approach exists?  The patch hasn't 
> even been demonstrated to be necessary in any real-world code.

peephole.c is now 773 lines; it was 700 lines, so it grew by about
10%. But that's not very large: ceval.c is 4500 lines (many of which
are much deeper magic which I personally don't feel I understand at
all :-). I am not at all sure that you can create an AST-based
optimizer in less than 700 lines. (Nor am I sure that you can't -- nor
am I sure it matters. I'm, just saying. :-)

All in all I think it would be more productive to try and understand
Antoine's patch than to try to get it revert based on "because I [or
Tim Peters] said so" arguments.

Finally: There appears to be some disagreement on who said what, in
particular Raymond claims that he told Antoine not to commit whereas
Antoine claims he did not hear this feedback. I'm guessing it happened
in IRC (#python-dev), which is intentionally not logged anywhere. This
is an illustration of what is wrong with using IRC for important
decisions. I would have been much happier if the discussion about the
desirability of the patch could have happened *before* it was
committed, and I think maybe we need to have a stricter rule requiring
code reviews and possibly cool-off periods.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About raising NotPortableWarning for CPython specific code

[Nick Coghlan]

On Sat, Mar 12, 2011 at 1:41 PM, Guido van Rossum  wrote:
> Yeah, indeed, on everything you said. The code dispatching based on
> internal slots is horribly ad-hoc and likely wrong in subtle ways. Has
> any of this improved in Python 3?

Not that I know of - I was definitely able to replicate the weirdness in 3.2.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The purpose of SETUP_LOOP, BREAK_LOOP, CONTINUE_LOOP

[P.J. Eby]

At 08:25 AM 3/12/2011 -0500, Eugene Toder wrote:

Right, I'm not suggesting to remove all blocks, only SETUP_LOOP
blocks. Do you see the problem in that case?


I think you guys are forgetting about FOR_ITER, listcomps, and the like.

That is, IIRC, the reason loops use the block stack is because they 
put things on the regular stack, that need to be cleared off the 
stack when the loop is exited (whether normally or via an exception).


In other words, just jumping out of a loop without popping the block 
stack would leave junk on the regular stack, thereby failing to 
deallocate the loop iterator.  In the case of a nested loop, this 
would also mean that the outer loop would start using the inner 
loop's iterator, and all sorts of hilarity would then ensue.




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)

[Maciej Fijalkowski]

On Sat, Mar 12, 2011 at 1:14 PM, "Martin v. Löwis"  wrote:
>>
>> http://www.boredomandlaziness.org/2011/03/python-vm-summit-rough-notes.html
>>
>> http://www.boredomandlaziness.org/2011/03/python-vm-summit-somewhat-coherent.html
>
> Wrt. the remark that other implementations should be referenced more
> prominently: I added them to
>
> http://www.python.org/download/

Cool. Would you mind mentioning JIT for PyPy in description?

>
> Regards,
> Martin
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Introductions

[Terry Reedy]

On 3/12/2011 9:41 AM, Ross Lagerwall wrote:

Hi,

I have been offered commit rights for Python after making a few patches
on subprocess and the os module.

Antoine suggested that I should introduce myself on the python-dev list
so here we go:

I am a student from South Africa and decided to do some work on Python
in my spare time. I think I chose the Python project because it
definitely seems to be one of the friendliest projects around and the
easiest to get started with - the dev guide helps a lot! Also, with the
big standard library, there is place for people with all different
levels of experience to contribute.


Great to have you with us!

--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Introductions

[Guido van Rossum]

Welcome Ross! Glad you like the new dev guide. (Brett and the
community should be proud of their recent work on that.) Remember,
there are no stupid questions. And while occasionally it may appear as
if someone tries to bite your head off, our bite is not as bad as our
bark.

--Guido

On Sat, Mar 12, 2011 at 9:41 AM, Ross Lagerwall  wrote:
> Hi,
>
> I have been offered commit rights for Python after making a few patches
> on subprocess and the os module.
>
> Antoine suggested that I should introduce myself on the python-dev list
> so here we go:
>
> I am a student from South Africa and decided to do some work on Python
> in my spare time. I think I chose the Python project because it
> definitely seems to be one of the friendliest projects around and the
> easiest to get started with - the dev guide helps a lot! Also, with the
> big standard library, there is place for people with all different
> levels of experience to contribute.
>
> Cheers
> Ross
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)

[Allison Randal]

On 03/12/2011 11:17 AM, Nick Coghlan wrote:

About this bit from the VM meeting notes:
  - original Python-on-Parrot ran into problems due to semantic
  mismatches between Perl 6 and Python - reached the limits of the
  degree of difference the Perl 6 toolchain was willing to tolerate)

Would you have any pointers to more info about that?


No, it was something Allison Randal said during the summit. I don't if
anything more detailed has been written about the topic anywhere else.


I haven't posted/written about it anywhere yet. Happy to answer any 
questions, probably off-list so as not to bore people here. :) If the 
questions are particularly insightful, I might convert it directly into 
a Q&A blog post.


Allison
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About raising NotPortableWarning for CPython specific code

[Guido van Rossum]

On Sat, Mar 12, 2011 at 8:59 AM, Nick Coghlan  wrote:
> On Sat, Mar 12, 2011 at 8:33 AM, Laura Creighton  wrote:
>> For those of you not at the Language Summit at PyCON the day before 
>> yesterday,
>> there was talk of identifying non-portable behaviour, such as relying on
>> CPython's reference counting garbage collector to close files for you as
>> soon as they become unreachable.  And then warning about them.
>>
>> We have a real live user who has a large code base that relies on
>> the CPython behaviour that an object's __radd__ method will take precedence
>> over a list's inplace add behaviour.
>>
>> The thread with the whole gory details begins here:
>> http://codespeak.net/pipermail/pypy-dev/2011q1/006958.html
>>
>> My inclination is to declare code that relies on this as broken, rather
>> than patch every wretched container type in PyPy.  Can this become
>> blessed as a 'you shouldn't have done this'?
>
> Given that the meat of the difference in semantics lies in the
> CPython-specific distinction between nb_add and sq_concat, I'm
> inclined to agree with you.
>
> I'm actually tempted to call the current CPython semantics flatout
> *wrong*. When a given operation has multiple C level slots, shouldn't
> we be checking all the LHS slots before checking any of the RHS slots?
> There's nothing in the language spec that I'm aware of that justifies
> us doing otherwise.
>
> (obviously, history now means that changing our behaviour would
> require a deprecation period)

Yeah, indeed, on everything you said. The code dispatching based on
internal slots is horribly ad-hoc and likely wrong in subtle ways. Has
any of this improved in Python 3?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)

[Martin v. Löwis]

http://www.boredomandlaziness.org/2011/03/python-vm-summit-rough-notes.html
http://www.boredomandlaziness.org/2011/03/python-vm-summit-somewhat-coherent.html


Wrt. the remark that other implementations should be referenced more 
prominently: I added them to


http://www.python.org/download/

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[Raymond Hettinger]

I would like to withdraw my suggestion for the recursive constant folding patch 
to be reverted.  I value social harmony much more than a decision about whether 
a particular patch is a good idea.  I apologize to anyone who is upset over the 
discussion.


Raymond
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)

[Nick Coghlan]

On Sat, Mar 12, 2011 at 11:08 AM, Éric Araujo  wrote:
> Hi,
>
>> I posted my rough notes and additional write-ups for Wednesday's VM
>> summit and Thursday's language summit:
>
> Thanks for doing that!
>
> About this bit from the VM meeting notes:
>  - original Python-on-Parrot ran into problems due to semantic
>  mismatches between Perl 6 and Python - reached the limits of the
>  degree of difference the Perl 6 toolchain was willing to tolerate)
>
> Would you have any pointers to more info about that?

No, it was something Allison Randal said during the summit. I don't if
anything more detailed has been written about the topic anywhere else.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Builtin open() too slow

[Christian Heimes]

Am 12.03.2011 16:13, schrieb Lukas Lueg:
> i've a storage engine that stores a lot of files (e.g. > 10.000) in
> one path. Running the code under cProfile, I found that with a total
> CPU-time of 1,118 seconds, 121 seconds are spent in 27.013 calls to
> open(). The number of calls is not the problem; however I find it
> *very* discomforting that Python spends about 2 minutes out of 18
> minutes of cpu time just to get a file-handle after which it can spend
> some other time to read from them.
> 
> May this be a problem with the way Python 2.7 gets filehandles from
> the OS or is it a problem with large directories itself?

Your issue is most like not a Python bug. The open() function in 2.7 is
a thin wrapper around fopen(3). You didn't tell us how you profiled your
program and what's your operating system, configuration (file system)
and hardware. Are you sure you have measured two CPU minutes and not two
minutes runtime mostly spent in I/O?

Recently I've seen a system that sometimes takes more than a minute or
two just to create and remove 16 files in 16 directories on a remote NFS
cluster. 10k files in one directory are not a bottleneck, if you use a
good file system (XFS, ext* with hashed btree index). I've over a
million files in one directory because it's as fast as 1000 directories
with 1000 files each for all my performance relevant operations --
sometimes even faster.

Christian

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4 version in the tracker

[Eric Smith]

On 03/12/2011 10:55 AM, Éric Araujo wrote:

I have a deprecation warning that I need to make an error in 3.4.


A neat trick to remember to do those changes is using a test that fails
if something does not raise a DeprecationWarning if sys.version_info[:2]
== (3, 3), or an error if sys.version_info[:3] == (3, 4).  You write
those tests once and let “make test” remind you as soon as the Python
version changes.


I like the idea, but it seems a little hostile to the person who 
actually changes the version number!


Eric.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)

[Éric Araujo]

Hi,

> I posted my rough notes and additional write-ups for Wednesday's VM
> summit and Thursday's language summit:

Thanks for doing that!

About this bit from the VM meeting notes:
  - original Python-on-Parrot ran into problems due to semantic
  mismatches between Perl 6 and Python - reached the limits of the
  degree of difference the Perl 6 toolchain was willing to tolerate)

Would you have any pointers to more info about that?

Regards
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[Raymond Hettinger]

There are separate social, strategic, and tactical questions.

The social question:  if the person who designed, implemented, and maintained 
the optimizer recommends against a patch and another committer just checks it 
in anyway, do we care?  I've taken responsibility for this code and have 
treated it will a good deal of care.  I don't believe the patch should go in 
because the risk/reward ratio is too low and because a better solution exists.

The strategic question:  constant folding in the peephole optimizer pre-dates 
the AST branch.  When AST landed, there was general agreement among myself and 
those involved that any future optimizations which could be done with the AST, 
should be done there.  It is the correct technological solution.  

At one point, we had strategic agreement to stop building-out the peepholer in 
any significant way.   In fairness to Antoine, the conversations took place 
several years before he became a committer.  The strategic question is do we 
want to come to that agreement again?  Can we agree to not have significant 
changes to the peepholer, to treat it with great care and restraint?   This 
patch doesn't have to go in.  Py3.3 won't be out for 18 to 24 months anyway.  
There is a lot of time to do the job right and not add weight to a house of 
cards.

The tactical question:  is this particular patch correct?  Maybe it is.  I 
don't know, I didn't get past all the new macros.  I do know that I could have 
whipped up something similar years ago but chose not to based on advice from 
Thomas, Andrew, Tim, and Guido.   The only change since then is that there is a 
newer committer doesn't buy into that reasoning.

Constant folding is the most complex part of the optimizer.  Why would we add 
to it, when a better and more reliable approach exists?  The patch hasn't even 
been demonstrated to be necessary in any real-world code.  


Raymond

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Builtin open() too slow

[Eugene Toder]

Hi

What OS and what file system you are using? Many file systems (e,g.
ext2/3fs) handle large directories very poorly.
A quick way to check if this has anything to do with Python is writing
a small C program that opens these files and time it.

Eugene

On Sat, Mar 12, 2011 at 10:13 AM, Lukas Lueg  wrote:
> Hi,
>
> i've a storage engine that stores a lot of files (e.g. > 10.000) in
> one path. Running the code under cProfile, I found that with a total
> CPU-time of 1,118 seconds, 121 seconds are spent in 27.013 calls to
> open(). The number of calls is not the problem; however I find it
> *very* discomforting that Python spends about 2 minutes out of 18
> minutes of cpu time just to get a file-handle after which it can spend
> some other time to read from them.
>
> May this be a problem with the way Python 2.7 gets filehandles from
> the OS or is it a problem with large directories itself?
>
> Best regards
> Lukas
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/eltoder%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4 version in the tracker

[Antoine Pitrou]

On Sat, 12 Mar 2011 16:55:30 +0100
Éric Araujo  wrote:
> > I have a deprecation warning that I need to make an error in 3.4.
> 
> A neat trick to remember to do those changes is using a test that fails
> if something does not raise a DeprecationWarning if sys.version_info[:2]
> == (3, 3), or an error if sys.version_info[:3] == (3, 4).  You write
> those tests once and let “make test” remind you as soon as the Python
> version changes.

And magically break all the buildbots :)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Builtin open() too slow

[Lukas Lueg]

Hi,

i've a storage engine that stores a lot of files (e.g. > 10.000) in
one path. Running the code under cProfile, I found that with a total
CPU-time of 1,118 seconds, 121 seconds are spent in 27.013 calls to
open(). The number of calls is not the problem; however I find it
*very* discomforting that Python spends about 2 minutes out of 18
minutes of cpu time just to get a file-handle after which it can spend
some other time to read from them.

May this be a problem with the way Python 2.7 gets filehandles from
the OS or is it a problem with large directories itself?

Best regards
Lukas
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4 version in the tracker

[Éric Araujo]

> I have a deprecation warning that I need to make an error in 3.4.

A neat trick to remember to do those changes is using a test that fails
if something does not raise a DeprecationWarning if sys.version_info[:2]
== (3, 3), or an error if sys.version_info[:3] == (3, 4).  You write
those tests once and let “make test” remind you as soon as the Python
version changes.

Regards
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4 version in the tracker

[Martin v. Löwis]

Am 12.03.11 09:26, schrieb Eric Smith:

Could someone with the right access add a "Python 3.4" version to the
tracker?


Done.


I'd also like to make it a release blocker in 3.4 so I don't forget
about it. If I do that, will it screw up any release workflow?


I don't think so. As release managers can easily find out, they
tend to switch those to "deferred blocker" just before a release,
and switch them back afterwards.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Introductions

[Ross Lagerwall]

Hi,

I have been offered commit rights for Python after making a few patches
on subprocess and the os module.

Antoine suggested that I should introduce myself on the python-dev list
so here we go:

I am a student from South Africa and decided to do some work on Python
in my spare time. I think I chose the Python project because it
definitely seems to be one of the friendliest projects around and the
easiest to get started with - the dev guide helps a lot! Also, with the
big standard library, there is place for people with all different
levels of experience to contribute.

Cheers
Ross

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python 3.4 version in the tracker

[Eric Smith]

Could someone with the right access add a "Python 3.4" version to the 
tracker? I have a deprecation warning that I need to make an error in 3.4.


I'd also like to make it a release blocker in 3.4 so I don't forget 
about it. If I do that, will it screw up any release workflow?


Thanks.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About raising NotPortableWarning for CPython specific code

[Nick Coghlan]

On Sat, Mar 12, 2011 at 8:33 AM, Laura Creighton  wrote:
> For those of you not at the Language Summit at PyCON the day before yesterday,
> there was talk of identifying non-portable behaviour, such as relying on
> CPython's reference counting garbage collector to close files for you as
> soon as they become unreachable.  And then warning about them.
>
> We have a real live user who has a large code base that relies on
> the CPython behaviour that an object's __radd__ method will take precedence
> over a list's inplace add behaviour.
>
> The thread with the whole gory details begins here:
> http://codespeak.net/pipermail/pypy-dev/2011q1/006958.html
>
> My inclination is to declare code that relies on this as broken, rather
> than patch every wretched container type in PyPy.  Can this become
> blessed as a 'you shouldn't have done this'?

Given that the meat of the difference in semantics lies in the
CPython-specific distinction between nb_add and sq_concat, I'm
inclined to agree with you.

I'm actually tempted to call the current CPython semantics flatout
*wrong*. When a given operation has multiple C level slots, shouldn't
we be checking all the LHS slots before checking any of the RHS slots?
There's nothing in the language spec that I'm aware of that justifies
us doing otherwise.

(obviously, history now means that changing our behaviour would
require a deprecation period)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] About raising NotPortableWarning for CPython specific code

[Laura Creighton]

For those of you not at the Language Summit at PyCON the day before yesterday,
there was talk of identifying non-portable behaviour, such as relying on
CPython's reference counting garbage collector to close files for you as
soon as they become unreachable.  And then warning about them.

We have a real live user who has a large code base that relies on
the CPython behaviour that an object's __radd__ method will take precedence
over a list's inplace add behaviour.

The thread with the whole gory details begins here:
http://codespeak.net/pipermail/pypy-dev/2011q1/006958.html

My inclination is to declare code that relies on this as broken, rather
than patch every wretched container type in PyPy.  Can this become 
blessed as a 'you shouldn't have done this'?

Laura

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The purpose of SETUP_LOOP, BREAK_LOOP, CONTINUE_LOOP

[Eugene Toder]

> There are also "with" blocks :-) (which use separate opcodes, although
> they are similar in principle to try/finally blocks)

IIUC they use separate opcode, but the same block type (SETUP_FINALLY).

> There may be complications with nested try/finally blocks. You either
> need to generate separate bytecode for when the "finally" clause is
> entered following a given continue/break (so as to hardcode the jump
> offset at the end of the clause), or save the jump offsets somewhere on
> a stack for each finally clause to pop, IMO.

Right, I'm not suggesting to remove all blocks, only SETUP_LOOP
blocks. Do you see the problem in that case?

Eugene
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[John Arbash Meinel]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


...

> I have always felt uncomfortable with *any* kind of optimization --
> whether AST-based or bytecode-based. I feel the cost in code
> complexity is pretty high and in most cases the optimization is not
> worth the effort. Also I don't see the point in optimizing expressions
> like "3 * 4 * 5" in Python. In C, this type of thing occurs frequently
> due to macro expansion and the like, but in Python your code usually
> looks pretty silly if you write that. 

Just as a side comment here, I often do this sort of thing when dealing
with time based comments, or large constants. It is a bit more obvious that:
 10*1024*1024 vs 10485760 is 10MiB
especially since you can't put commas in your constants. 10,485,760
would at least make it immediately obvious it was 10M not 104M or
something else.

Similarly is 10,800s 3 hours? 3*60*60 certainly is.

I don't think I've done that sort of thing in anything performance
critical. But I did want to point out that writing "X*Y*Z" as a constant
isn't always "pretty silly".

John
=:->
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk17c0sACgkQJdeBCYSNAAMUrQCdEhissWuvTElIc6Wy/2qotzaU
xz4AnRO+ND/3NkKWC7Bbu78ACjs2X920
=QR/2
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)

[Nick Coghlan]

I posted my rough notes and additional write-ups for Wednesday's VM
summit and Thursday's language summit:

http://www.boredomandlaziness.org/2011/03/python-vm-summit-rough-notes.html
http://www.boredomandlaziness.org/2011/03/python-vm-summit-somewhat-coherent.html
http://www.boredomandlaziness.org/2011/03/python-language-summit-rough-notes.html
http://www.boredomandlaziness.org/2011/03/python-language-summit-highlights.html

I believe Brian Curtin will also be posting a write-up for Thursday,
but I don't know if that is online anywhere yet.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] public visibility of python-dev decisions "before it's too late" (was: PyCObject_AsVoidPtr removed from python 3.2 - is this documented?)

[Doug Hellmann]

On Mar 11, 2011, at 7:12 PM, Paul Moore wrote:

> On 11 March 2011 23:24, Guido van Rossum  wrote:
>>> I'm interested in the task and I guess I'll follow-up with Doug Hellman. I
>>> don't follow -ideas close enough to summarize it, but I'd contribute to a
>>> -dev blog.
>> 
>> Awesome! (And we don't need to stop at one blogger. Many hands make light 
>> work.)
> 
> I'm interested but probably haven't got time to take something like
> this on "properly". Helping out occasionally would probably be
> something I could do, though. (Was that tentative enough? :-)) I'm on
> -ideas, too, so I could do something there as well.

Thanks Brian and Paul! I have sent you both invitations to join the 
communications team mailing list so we can work out the details.

Doug

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Martin v. Löwis]

Am 12.03.11 04:03, schrieb Nadeem Vawda:

Hmm... it seems that the given instructions don't actually work. "hg
revert -a" fails,
saying that a specific revision needs to be provided. The command should be
"hg revert -ar default".


Isn't that command correct only if you are merging into default?
ISTM that it should be "hg revert -ar ".

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[Nick Coghlan]

On Sat, Mar 12, 2011 at 5:07 AM, Mark Dickinson  wrote:
> I can also see the case for ripping out the peepholer entirely.  But
> reverting Antoine's patch seems like a step backwards.

+1 to what Mark says here.

If the day comes when the peepholer can be ripped out in favour of AST
based optimisation, then yay. In the meantime, having it work as
consistently as possible in picking up and optimising literal
expressions makes for potentially valuable micro-optimisations that
people don't have to worry about doing by hand.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The purpose of SETUP_LOOP, BREAK_LOOP, CONTINUE_LOOP

[Antoine Pitrou]

Hello,

> Am I missing something? Does SETUP_LOOP serve any other purpose?

Not to my knowledge.

> Similarly, it looks like BREAK_LOOP and CONTINUE_LOOP are just jumps
> that respect try/finally blocks (i.e. jumping out of try executes
> finally). Is there more semantics to them than this?

There are also "with" blocks :-) (which use separate opcodes, although
they are similar in principle to try/finally blocks)

> 1) If not in try/finally, simply generate a direct jump outside of the
> loop (break) or to the start of the loop (continue).

Right.

> 2) If in try/finally, generate a new instruction JUMP_FROM_TRY which
> replaces both BREAK_LOOP and CONTINUE_LOOP. It behaves the same way as
> CONTINUE_LOOP but without restriction to only jump backwards (could
> reuse CONTINUE_LOOP, but the name would be misleading).

There may be complications with nested try/finally blocks. You either
need to generate separate bytecode for when the "finally" clause is
entered following a given continue/break (so as to hardcode the jump
offset at the end of the clause), or save the jump offsets somewhere on
a stack for each finally clause to pop, IMO.

(the current code does the latter with various complications, see
WITH_CLEANUP for some aspects of the fun)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[Mark Dickinson]

FWIW, I'm -1 on backing out Antoine's patch.  In addition to fixing
the minor optimization regression, it makes the peepholer
significantly more consistent in what it can and can't fold.  One of
the first times that I delved into the peepholer code was to try to
understand why expressions like:  2 * 3 * 4 and 3**2 * 7 were constant
folded, when 2 * (3 * 4) and 7 * 3**2 were not;  with Antoine's patch,
they're all folded.  That both Antoine and Eugene are happy with the
code gives me confidence in its correctness.

I can also see the case for ripping out the peepholer entirely.  But
reverting Antoine's patch seems like a step backwards.

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggest reverting today's checkin (recursive constant folding in the peephole optimizer)

[Antoine Pitrou]

Hello,

> I recall several occasions where the peephole optimizer was subtly
> buggy -- on one occasion the bug remained undetected for at least a
> whole release cycle. (Sorry, I can't recall the details.) In fact, the
> bug reported in http://bugs.python.org/issue11244 is another example
> of how subtle the peephole optimizer is.

Well, the regression stems from a commit in the ast code, not in the
peepholer code (and it's only a regression as in "misses an
optimization opportunity", not "produces bogus code").

The peepholer is actually not very subtle, and that's why it missed the
obvious optimization here.

> As far as Antoine's commit goes, it looks like it went in hastily,
> without review, and before any discussion. Clearly it *is*
> controversial, and controversial fixes always need to be held back
> until at least rough consensus is reached or until a BDFL decision.

I didn't get any comments against the patch before committing it, and
I trusted Eugene's comment that the second patch was fine (Eugene
proposed other, decent patches for the peepholer and thus seems
acquainted with the code). Also, as Eugene pointed out, running the
whole test suite in itself stresses the peepholer quite a bit if you
take care of removing pyc files beforehand (which the buildbots do).

Any further review is obviously welcome, as usual.

> Also it is not a fix to the bug reported in the issue, so the
> Misc/NEWS text is misleading. A much simpler fix was proposed in the
> bug and even approved by Antoine.

Well, the "much simpler fix" fixes a separate issue with -0; it is
orthogonal to the original issue, and doesn't fix it (Eugene should
probably have filed it it in a separate issue).

The patch I committed, OTOH, *is* a fix to the original issue (there's
even a test for that ;).
The fact that the fix implies a more general improvement to the
peepholer might be seen as an annoyance, or a benefit, depending on
your point of view (although I'm not sure why a general improvement
would be an annoyance).

> (One note on the patch: it allocates
> an extra stack which is dynamically grown; but there is no unittest to
> exercise the stack-growing code. This note does constitute a full
> review.)

As Eugene answered, there is actually an unittest for that.

> I have always felt uncomfortable with *any* kind of optimization --
> whether AST-based or bytecode-based. I feel the cost in code
> complexity is pretty high and in most cases the optimization is not
> worth the effort. Also I don't see the point in optimizing expressions
> like "3 * 4 * 5" in Python.

But then, as others pointed out, why don't we rip out the peepholer?
If it's fragile and doesn't serve a purpose, it doesn't deserve staying.

There is, by the way, recent history for adding optimizations to the
peepholer, which was even approved by Raymond:
http://bugs.python.org/issue6690
... so I'm not sure why there is opposition to fixing a regression,
when apparently new optimizations are uncontroversial.

> In C, this type of thing occurs frequently
> due to macro expansion and the like, but in Python your code usually
> looks pretty silly if you write that. So this is basically a solution
> to a non-problem.

The original issue was about constant-folding of tuples containing
negative numbers, which doesn't sound like an exotic situation. The
obvious solution (enabling propagation of constant-folding) also
entails constant-folding of arbitrarily complex expressions, which is
why I have added tests for it, and suitably described it in the
Misc/NEWS. Having more tests is always worthwhile IMO (even if we later
choose to change them for whatever reason).

> Now, should Antoine's checkin be rolled back? I think the case is in
> Antoine's court to convince us that his patch is correct *and* to help
> at least one or two others feel comfortable they understand how it
> works -- if only Antoine understands it, it is too complex.

I really don't think the patch is complicated. It adds a stack to
manage the current run of constants. This replaces the previous code
where that run of constants was implicitly managed through the "lastlc"
and "cumlc" variables and hand-computed indices into the bytecode; the
new approach seems less clumsy to me.

(Eugene apparently understood the patch, so hopefully I'm not alone :-))

Now, if a majority is in favour of rolling it back ("backout" in
hg terminology), let's do it. That regression obviously isn't
killing anyone. Should we ask for a vote?

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Nadeem Vawda]

Hmm... it seems that the given instructions don't actually work. "hg
revert -a" fails,
saying that a specific revision needs to be provided. The command should be
"hg revert -ar default".
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Nadeem Vawda]

On Sat, Mar 12, 2011 at 9:32 AM, Eli Bendersky  wrote:
> The devguide's recommendation is to "forward-port" changes withing a major
> release line, i.e. if I need something in all 3.[123], then start with 3.1
> and forward-port (by "hg merge ") to 3.2 and then 3.3
>
> Just to clarify - does this mean that all changesets that are applied to 3.2
> eventually get to 3.3 as well? Since we just do "hg merge 3.2" in the 3.3
> clone, I guess this must be true. But then what happens if there's a change
> in 3.2 that shouldn't get into 3.3? (say a bug fix for a feature that has
> disappeared)?

The "Note:" box two paragraphs down from

explains how prevent a changeset from propagating: you do a dummy merge
where you revert the changes before committing. This marks the changeset from
3.2 as having been merged into default/3.3 without actually applying
the changes.

Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com