Re: [Python-Dev] A new JIT compiler for a faster CPython?

[Jeffrey Yasskin]

On Tue, Jul 17, 2012 at 3:20 PM, Victor Stinner
 wrote:
>> It's the JIT compiler of Unladen Swallow that "failed"; in
>> my understanding because LLVM is crap (i.e. it is slow, memory-consuming,
>> and buggy) - as a low-level virtual machine; it may be ok as a compiler
>> backend (but I still think it is buggy there as well).
>
> What is the status of LLVM nowadays? Is it not a good solution to
> write a portable JIT?

Its code generator is still fairly slow. You could probably get a
faster one committed, but you'd have to write it. LLVM also still
doesn't have great profile-guided optimizations (what you need in a
JIT), although the infrastructure is starting to be built. You'd
probably have to contribute to that too. It's probably a better use of
your time to contribute to PyPy.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 418: Add monotonic clock

[Jeffrey Yasskin]

FWIW, I'm not sure you're the right person to drive time PEPs. You
don't seem to have come into it with much knowledge of time, and it's
taken several repetitions for you to take corrections into account in
both this discussion and the Decimal/datetime representation PEP.

On Mon, Mar 26, 2012 at 4:32 PM, Victor Stinner
 wrote:
> Hi,
>
> I started to write the PEP 418 to clarify the notions of monotonic and
> steady clocks.
>
> The PEP is a draft and everyone is invited to contribute!
>
> http://www.python.org/dev/peps/pep-0418/
> http://hg.python.org/peps/file/tip/pep-0418.txt
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rename time.steady(strict=True) to time.monotonic()?

[Jeffrey Yasskin]

On Fri, Mar 23, 2012 at 4:25 PM, Victor Stinner
 wrote:
> Hi,
>
> time.steady(strict=True) looks to be confusing for most people, some
> of them don't understand the purpose of the flag and others don't like
> a flag changing the behaviour of the function.
>
> I propose to replace time.steady(strict=True) by time.monotonic().
> That would avoid the need of an ugly NotImplementedError: if the OS
> has no monotonic clock, time.monotonic() will just not exist.
>
> So we will have:
>
> - time.time(): realtime, can be adjusted by the system administrator
> (manually) or automatically by NTP
> - time.clock(): monotonic clock on Windows, CPU time on UNIX
> - time.monotonic(): monotonic clock, its speed may or may not be
> adjusted by NTP but it only goes forward, may raise an OSError
> - time.steady(): monotonic clock or the realtime clock, depending on
> what is available on the platform (use monotonic in priority). may be
> adjusted by NTP or the system administrator, may go backward.

Please don't use the word "steady" for something different from what
C++ means by it. C++'s term means "may not be adjusted at all, even by
NTP; proceeds at as close to the rate of real time as the system can
manage" (paraphrased). If the consensus in the Python community is
that a C++-style "steady" clock is unnecessary, then feel free not to
define it. If the consensus is that "monotonic" already means
everything C++ means by "steady", that's fine with me too. I mentioned
it because I thought it might be worth looking at what other languages
were doing in this space, not because I thought it was a nice word
that you should attach your own definitions to.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Jeffrey Yasskin]

On Tue, Mar 13, 2012 at 6:10 PM, Nadeem Vawda  wrote:
> On Wed, Mar 14, 2012 at 3:03 AM, Victor Stinner
>  wrote:
>> I suppose that most libraries and programs will have to implement a
>> similar fallback.
>>
>> We may merge both functions with a flag to be able to disable the
>> fallback. Example:
>>
>>  - time.realtime(): best-effort monotonic, with a fallback
>>  - time.realtime(monotonic=True): monotonic, may raise OSError or
>> NotImplementedError
>
> This was my suggestion - I think it's useful to have the fallback
> available (since most users will want it), but at the same time we
> should also cater to users who need a clock that is *guaranteed* to
> be monotonic.
>
> As an aside, I think "monotonic" is a better name than "realtime";
> it conveys the functions purpose more clearly. Then we could call
> the flag "strict".

While you're bikeshedding:

Some of the drafts of the new C++ standard had a monotonic_clock,
which was guaranteed to only go forwards, but which could be affected
by system clock updates that went forwards. Because of problems in
defining timeouts using an adjustable clock, C++11 instead defines a
"steady_clock", which ticks as steadily as the machine/OS/library can
ensure, and is definitely not affected by any time adjustments:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3191.htm. I
realize that Unix's clock_gettime(CLOCK_MONOTONIC) already includes
the steadiness criterion, but the word itself doesn't actually include
the meaning.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Jeffrey Yasskin]

On Tue, Mar 13, 2012 at 5:03 PM, Michael Foord
 wrote:
>
> On 13 Mar 2012, at 16:57, Victor Stinner wrote:
>
>> Hi,
>>
>> I added two functions to the time module in Python 3.3: wallclock()
>> and monotonic(). I'm unable to explain the difference between these
>> two functions, even if I wrote them :-) wallclock() is suppose to be
>> more accurate than time() but has an unspecified starting point.
>> monotonic() is similar except that it is monotonic: it cannot go
>> backward. monotonic() may not be available or fail whereas wallclock()
>> is available/work, but I think that the two functions are redundant.
>>
>> I prefer to keep only monotonic() because it is not affected by system
>> clock update and should help to fix issues on NTP update in functions
>> implementing a timeout.
>>
>> What do you think?
>>
>
>
> I am in the middle of adding a feature to unittest that involves timing of 
> individual tests. I want the highest resolution cross platform measure of 
> wallclock time - and time.wallclock() looked ideal. If monotonic may not 
> exist or can fail why would that be better?
>

Isn't the highest resolution cross platform measure of "wallclock"
time spelled "time.clock()"? Its docs say "this is the function to use
for benchmarking Python or timing algorithms", and it would be a shame
to add and teach a new function rather than improving clock()'s
definition.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: New timestamp formats

[Jeffrey Yasskin]

On Fri, Feb 3, 2012 at 11:17 AM, Antoine Pitrou  wrote:
> On Fri, 3 Feb 2012 11:04:14 -0800
> Jeffrey Yasskin  wrote:
>> On Thu, Feb 2, 2012 at 4:59 PM, Nick Coghlan  wrote:
>> > datetime.datetime
>> >
>> > - real problem with the idea is that not all timestamps can be easily
>> > made absolute (e.g. some APIs may return "time since system started"
>> > or "time since process started")
>>
>> I think this is an argument for returning the appropriate one of
>> datetime or timedelta from all of these functions: users need to keep
>> track of whether they've got an absolute time, or an offset from an
>> unspecified starting point, and that's a type-like distinction.
>
> Keep in mind timedelta has a microsecond resolution. The use cases
> meant for the PEP imply nanosecond resolution (POSIX' clock_gettime(),
> for example).

Yes, I think someone had noted that datetime and timedelta would need
to be extended to support nanosecond resolution.

>> A plain number of seconds is superficially simpler, but it forces more
>> complexity onto the user, who has to track what that number
>> represents.
>
> If all you are doing is comparing timestamps (which I guess is most of
> what people do with e.g. st_mtime), a number is fine.

Sure. I don't think the argument for datetime is totally convincing,
just that it's stronger than the PEP currently presents.

> If you want the current time and date in a high-level form, you can
> already use datetime.now() or datetime.utcnow() (which "only" has
> microsecond resolution as well :-)). We don't need another way to spell
> it.

Whoops, yes, there's no need to extend time() to return a datetime.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: New timestamp formats

[Jeffrey Yasskin]

On Thu, Feb 2, 2012 at 4:59 PM, Nick Coghlan  wrote:
> datetime.datetime
>
> - real problem with the idea is that not all timestamps can be easily
> made absolute (e.g. some APIs may return "time since system started"
> or "time since process started")

I think this is an argument for returning the appropriate one of
datetime or timedelta from all of these functions: users need to keep
track of whether they've got an absolute time, or an offset from an
unspecified starting point, and that's a type-like distinction.

> - the complexity argument used against timedelta also applies

A plain number of seconds is superficially simpler, but it forces more
complexity onto the user, who has to track what that number
represents.

datetime and timedelta are even available from a C module, which I had
expected to be a blocking issue.

The biggest problem I see with using datetime and timedelta for
everything is that switching to them is very backwards-incompatible.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: New timestamp formats

[Jeffrey Yasskin]

On Fri, Feb 3, 2012 at 3:57 AM, Victor Stinner
 wrote:
>> datetime.datetime
>>
>> - as noted earlier in the thread, total_seconds() actually gives you a
>> decent timestamp value and always returning UTC avoids timezone issues
>
> os.stat() and time.time() use the local time.

The documentation disagrees with you.
http://docs.python.org/dev/library/time.html#time.time says "Return
the time as a floating point number expressed in seconds since the
epoch, in UTC."

os.stat is documented less clearly, but it's implemented by forwarding
to the system stat(), and that's defined at
http://www.opengroup.org/sud/sud1/xsh/sysstat.h.htm#sysstat.h-file-desc-stru
to return times since the Epoch.
http://pubs.opengroup.org/onlinepubs/95399/functions/localtime.html
says "January 1, 1970 0:00 UTC (the Epoch)"

> I don't see datetime without tzinfo as an issue.
>
> Being unable to convert a datetime to an epoch timestamp is also not
> an issue: if you need an epoch timestamp, just use float or Decimal
> types.

Your PEP still says, 'there is no easy way to convert it into "seconds
since the epoch"'. If you don't actually think it's an issue (which
it's not, because there is an easy way to convert it), then take that
out of the PEP.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: New timestamp formats

[Jeffrey Yasskin]

On Wed, Feb 1, 2012 at 5:03 PM, Victor Stinner
 wrote:
> datetime.datetime
> -
>
> datetime.datetime only supports microsecond resolution, but can be enhanced
> to support nanosecond.
>
> datetime.datetime has issues:
>
> - there is no easy way to convert it into "seconds since the epoch"

Not true:

>>> import datetime, time
>>> epoch = datetime.datetime(1970, 1, 1, 0, 0, 0)
>>> (datetime.datetime.utcnow() - epoch).total_seconds()
1328219742.385039
>>> time.time()
1328219747.640937
>>>

> - any broken-down time has issues of time stamp ordering in the
>  duplicate hour of switching from DST to normal time

Only if you insist on putting it in a timezone. Use UTC, and you should be fine.

> - time zone support is flaky-to-nonexistent in the datetime module

Why do you need time zone support for system interfaces that return
times in UTC?


I think I saw another objection that datetime represented points in
time, while functions like time.time() and os.stat() return an offset
from the epoch. This objection seems silly to me: the return value of
the system interfaces intends to represent points in time, even though
it has to be implemented as an offset since an epoch because of
limitations in C, and datetime is also implemented as an offset from
an epoch (year 0).

On the other hand, the return value of functions like time.clock() is
_not_ intended to represent an exact point in time, and so should be
either a timedelta or Decimal.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hash collision security issue (now public)

[Jeffrey Yasskin]

On Wed, Dec 28, 2011 at 5:37 PM, Jesse Noller  wrote:
>
>
> On Wednesday, December 28, 2011 at 8:28 PM, Michael Foord wrote:
>
>> Hello all,
>>
>> A paper (well, presentation) has been published highlighting security 
>> problems with the hashing algorithm (exploiting collisions) in many 
>> programming languages Python included:
>>
>> http://events.ccc.de/congress/2011/Fahrplan/attachments/2007_28C3_Effective_DoS_on_web_application_platforms.pdf
>>
>> Although it's a security issue I'm posting it here because it is now public 
>> and seems important.
>>
>> The issue they report can cause (for example) handling an http post to 
>> consume horrible amounts of cpu. For Python the figures they quoted:
>>
>> reasonable-sized attack strings only for 32 bits Plone has max. POST size of 
>> 1 MB
>> 7 minutes of CPU usage for a 1 MB request
>> ~20 kbits/s → keep one Core Duo core busy
>>
>> This was apparently reported to the security list, but hasn't been responded 
>> to beyond an acknowledgement on November 24th (the original report didn't 
>> make it onto the security list because it was held in a moderation queue).
>>
>> The same vulnerability was reported against various languages and web 
>> frameworks, and is already fixed in some of them.
>>
>> Their recommended fix is to randomize the hash function.
>>
>> All the best,
>>
>> Michael
>>
> Back up link for the PDF:
> http://dl.dropbox.com/u/1374/2007_28C3_Effective_DoS_on_web_application_platforms.pdf
>
> Ocert disclosure:
> http://www.ocert.org/advisories/ocert-2011-003.html

Discussion of hash functions in general:
http://burtleburtle.net/bob/hash/doobs.html
Two of the best hash functions that currently exist:
http://code.google.com/p/cityhash/ and
http://code.google.com/p/smhasher/wiki/MurmurHash.

I'm not sure exactly what problem the paper is primarily complaining about:
1. Multiply+add and multiply+xor hashes are weak: this would be solved
by changing to either of the better-and-faster hashes I linked to
above. On the other hand:
http://mail.python.org/pipermail/python-3000/2007-September/010327.html
2. It's possible to find collisions in any hash function in a 32-bit
space: only solved by picking a varying seed at startup or compile
time.

If you decide to change to a stronger hash function overall, it might
also be useful to change the advice "to somehow mix together (e.g.
using exclusive or) the hash values for the components" in
http://docs.python.org/py3k/reference/datamodel.html#object.__hash__.
hash(tuple(components)) will likely be better if tuple's hash is
improved.

Hash functions are already unstable across Python versions. Making
them unstable across interpreter processes (multiprocessing doesn't
share dicts, right?) doesn't sound like a big additional problem.
Users who want a distributed hash table will need to pull their own
hash function out of hashlib or re-implement a non-cryptographic hash
instead of using the built-in one, but they probably need to do that
already to allow themselves to upgrade Python.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Format factories (was Re: sWAPcASE Was: transform() and untransform() methods, and the codec registry)

[Jeffrey Yasskin]

On Sat, Dec 11, 2010 at 11:21 AM, Brett Cannon  wrote:
> On Thu, Dec 9, 2010 at 16:26, Nick Coghlan  wrote:
>> On Fri, Dec 10, 2010 at 9:29 AM, Antoine Pitrou  wrote:
>>> On Thu, 09 Dec 2010 18:10:38 -0500
>>> Eric Smith  wrote:
 If we're looking to reduce the number of methods on str, I wouldn't mind
 seeing center() and zfill() also go away, since they're trivially
 replaced by format().
>>>
>>> Well, it's only trivial when you know format()'s wicked mini-language by
>>> heart :/  center() is easy and obvious. zfill() is arguably a bit too
>>> specialized.
>>
>> I've occasionally wondered if string formatting [1] and the struct
>> module [2] would benefit from format building functions that made them
>> easier to use without necessarily learning the cryptic mini-languages
>> off by heart.
>>
>> For example, a "string.make_format_spec" function might have a signature 
>> like:
>>
>> def make_format_spec(fill=None, align=None, sign=None, width=None,
>> precision=None, display_type='s', alternate=False, commas=False,
>> numeric_case=None)
>>
>> "align" would accept not only the actual format symbols ('<', '>',
>> '=', '^'), but also the corresponding names ('left', 'right',
>> 'numeric', 'center').
>>
>> "numeric_case" would accept None, 'upper' or 'lower' (affecting the
>> formatting of hex and floating point values). If the stated numeric
>> case differs from that specified by the display type, raise a
>> ValueError. For an unspecified numeric case, the affected display
>> types would default to 'lower'.
>>
>> Similarly, "display_type" would accept long names in addition to the
>> character codes:
>>
>> 's': 'str'
>> 'b': 'binary'
>> 'c': 'chr'
>> 'd': 'int'
>> 'o': 'octal'
>> 'x', 'X': 'hex' (numeric case controls display of digits A-F)
>> 'n': 'locale'
>> 'e', 'E': 'exponent' (numeric case controls display of exponent as
>> well as infinite and NaN values)
>> 'f', 'F': 'float' (numeric case controls display of exponent as well
>> as infinite and NaN values)
>> 'g', 'G': 'general' (numeric case controls display of exponent as well
>> as infinite and NaN values)
>> '%': 'percent' (numeric case controls display of exponent as well as
>> infinite and NaN values)
>>
>> There could also be a corresponding parse_format_spec that produced a
>> named tuple with the appropriate details.
>
> But is this worth it since once you write it you won't be changing it
> again,

Having a make_format_spec() would also help people trying to read the code.

> suggesting that taking the time to look up the formatting rules
> isn't that much harder and wouldn't burden us w/ writing such
> functions and trying to come up with a good API. I suspect what would
> be more helpful is either have a rather detailed docstring for
> str.format or to at least put the URL to the docs which explains the
> mini-language to make it easier to find.
>>
>> Cheers,
>> Nick.
>>
>> [1] 
>> http://docs.python.org/dev/library/string#format-specification-mini-language
>> [2] http://docs.python.org/dev/library/struct#format-strings
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[Jeffrey Yasskin]

On Wed, Oct 20, 2010 at 2:55 PM, Glyph Lefkowitz
 wrote:
>
> On Oct 20, 2010, at 12:31 AM, Jeffrey Yasskin wrote:
>
> No comment on the rest of your claim, but this is a silly argument.
> The standard says the same thing about at least fcntl.h, signal.h,
> pthread.h, and ucontext.h, which clearly are useful.
>
> It was meant to be tongue-in-cheek :).  Perhaps I should not have assumed
> that everyone else was as familiar with the POSIX documentation; I figured
> that most readers would know that most pages say that.

Oops, sorry. My joke-detector failed.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[Jeffrey Yasskin]

On Tue, Oct 19, 2010 at 8:37 PM, Glyph Lefkowitz
 wrote:
> I'd like to echo this sentiment.  This is not about providing a 'safe'
> wrapper to hide some powerful feature of these APIs: the POSIX aio_*
> functions are really completely useless.
> To quote the relevant standard
> :
>
> APPLICATION USAGE
> None.
> RATIONALE
> None.
> FUTURE DIRECTIONS
> None.


No comment on the rest of your claim, but this is a silly argument.
The standard says the same thing about at least fcntl.h, signal.h,
pthread.h, and ucontext.h, which clearly are useful.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Behaviour of max() and min() with equal keys

[Jeffrey Yasskin]

On Tue, Sep 7, 2010 at 2:40 PM, Jeffrey Yasskin  wrote:
> On Tue, Sep 7, 2010 at 1:44 PM, Mark Dickinson  wrote:
>> On Tue, Sep 7, 2010 at 8:34 PM, Matthew Woodcraft
>>  wrote:
>>> In CPython, the builtin max() and min() have the property that if there
>>> are items with equal keys, the first item is returned. From a quick look
>>> at their source, I think this is true for Jython and IronPython too.
>>
>> It's actually not clear to me that this behaviour is ideal;  it might
>> make more sense to have max() return the first item among equal
>> largest elements, and min() return the last item.  That way, the
>> special case of two operands has the nice property that (max(x, y),
>> min(x, y)) is always either (x, y) or (y, x), rather than being
>> possibly (x, x) or (y, y).  (That is, id(max(x, y)) and id(min(x, y))
>> are id(x) and id(y) in one order or the other.)
>>
>> The max and min methods for the Decimal module take care to work this
>> way, for example:
>>
>>>>> x, y = Decimal(2), Decimal('2.0')
>>>>> x.max(y), x.min(y)
>> (Decimal('2'), Decimal('2.0'))
>>
>> But:
>>
>>>>> max(x, y), min(x, y)
>> (Decimal('2'), Decimal('2'))
>>
>
> Decimal may actually have this backwards. The idea would be that
> min(*lst) == sorted(lst)[0], and max(*lst) == sorted(lst)[-1]. Given a
> stable sort, then, max of equivalent elements would return the last
> element, and min the first. According to Alex Stepanov, this helps
> some with composing these small order statistics into larger
> stably-ordered functions.
>
> I do think Decimal.max's answer is better than the builtin's answer,
> and that the incremental benefit from returning the last instead of
> first is pretty small.

Actually, Decimal isn't doing anything along these lines. At least in
Python 2.6, I get:

>>> Decimal('2').max(Decimal('2.0'))
Decimal('2')
>>> Decimal('2.0').max(Decimal('2'))
Decimal('2')
>>> Decimal('2.0').min(Decimal('2'))
Decimal('2.0')
>>> Decimal('2').min(Decimal('2.0'))
Decimal('2.0')

indicating that '2' is considered larger than '2.0' in the min/max
comparison. It's ignoring the order of the arguments. It also creates
a new Decimal object for the return value, so I can't use id() to
check which one of identical elements it returns.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Behaviour of max() and min() with equal keys

[Jeffrey Yasskin]

On Tue, Sep 7, 2010 at 1:44 PM, Mark Dickinson  wrote:
> On Tue, Sep 7, 2010 at 8:34 PM, Matthew Woodcraft
>  wrote:
>> In CPython, the builtin max() and min() have the property that if there
>> are items with equal keys, the first item is returned. From a quick look
>> at their source, I think this is true for Jython and IronPython too.
>
> It's actually not clear to me that this behaviour is ideal;  it might
> make more sense to have max() return the first item among equal
> largest elements, and min() return the last item.  That way, the
> special case of two operands has the nice property that (max(x, y),
> min(x, y)) is always either (x, y) or (y, x), rather than being
> possibly (x, x) or (y, y).  (That is, id(max(x, y)) and id(min(x, y))
> are id(x) and id(y) in one order or the other.)
>
> The max and min methods for the Decimal module take care to work this
> way, for example:
>
 x, y = Decimal(2), Decimal('2.0')
 x.max(y), x.min(y)
> (Decimal('2'), Decimal('2.0'))
>
> But:
>
 max(x, y), min(x, y)
> (Decimal('2'), Decimal('2'))
>

Decimal may actually have this backwards. The idea would be that
min(*lst) == sorted(lst)[0], and max(*lst) == sorted(lst)[-1]. Given a
stable sort, then, max of equivalent elements would return the last
element, and min the first. According to Alex Stepanov, this helps
some with composing these small order statistics into larger
stably-ordered functions.

I do think Decimal.max's answer is better than the builtin's answer,
and that the incremental benefit from returning the last instead of
first is pretty small.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New regex module for 3.2?

[Jeffrey Yasskin]

On Fri, Jul 9, 2010 at 7:06 AM, anatoly techtonik  wrote:
> On Thu, Jul 8, 2010 at 10:52 PM, MRAB  wrote:
>> Hi all,
>>
>> I re-implemented the re module, adding new features and speed
>> improvements. It's available at:
>>
>>    http://pypi.python.org/pypi/regex
>>
>> under the name "regex" so that it can be tried alongside "re".
>>
>> I'd be interested in any comments or feedback. How does it compare with
>> "re" in terms of speed on real-world data? The benchmarks suggest it
>> should be faster, or at worst comparable.
>
> And where are the benchmarks?
> In particular it would be interesting to see it compared both to re
> from stdlib and re2 from  http://code.google.com/p/re2/

While the re2 comparison might be interesting from an abstract
standpoint, it intentionally supports a different regex language from
Python so that it can run faster and use less memory. Since re2 can
never replace Python's re module, it doesn't make sense to hold MRAB's
new module to that standard.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] what environment variable should contain compiler warning suppression flags?

[Jeffrey Yasskin]

On Sun, Jun 27, 2010 at 1:04 PM, Mark Dickinson  wrote:
> On Sun, Jun 27, 2010 at 6:46 AM, Jeffrey Yasskin  wrote:
>> AC_PROG_CC is the macro that sets CFLAGS to -g -O2 on gcc-based
>> systems 
>> (http://www.gnu.org/software/hello/manual/autoconf/C-Compiler.html#index-AC_005fPROG_005fCC-842).
>> If Python's configure.in sets an otherwise-empty CFLAGS to -g before
>> calling AC_PROG_CC, AC_PROG_CC won't change it. Or we could just
>> preserve the users CFLAGS setting across AC_PROG_CC regardless of
>> whether it's set, to let the user set CFLAGS on the configure line
>> without stomping any defaults.
>
> I think saving and restoring CFLAGS across AC_PROG_CC was attempted in
> http://bugs.python.org/issue8211 . It turned out that it broke OS X
> universal builds.

Thanks for the link to the issue. http://bugs.python.org/issue8366
says Ronald Oussoren fixed the universal builds without reverting the
CFLAGS propagation.

> I'm not sure I understand the importance of allowing AC_PROG_CC to set
> CFLAGS (if CFLAGS is undefined at the point of the AC_PROG_CC);  can
> someone give an example of why this is necessary?

Marc-Andre's argument seems to be "it's possible that AC_PROG_CC adds
other flags as well (it currently doesn't, but that may well change in
future versions of autoconf)." That seems a little weak to constrain
fixing actual problems today. If it ever adds more arguments, we'll
need to inspect them anyway to see if they're more like -g or -O2
(wanted or harmful).

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] what environment variable should contain compiler warning suppression flags?

[Jeffrey Yasskin]

On Sat, Jun 26, 2010 at 4:37 PM, M.-A. Lemburg  wrote:
> Brett Cannon wrote:
>> On Wed, Jun 23, 2010 at 14:53, Brett Cannon  wrote:
>>> I finally realized why clang has not been silencing its warnings about
>>> unused return values: I have -Wno-unused-value set in CFLAGS which
>>> comes before OPT (which defines -Wall) as set in PY_CFLAGS in
>>> Makefile.pre.in.
>>>
>>> I could obviously set OPT in my environment, but that would override
>>> the default OPT settings Python uses. I could put it in EXTRA_CFLAGS,
>>> but the README says that's for stuff that tweak binary compatibility.
>>>
>>> So basically what I am asking is what environment variable should I
>>> use? If CFLAGS is correct then does anyone have any issues if I change
>>> the order of things for PY_CFLAGS in the Makefile so that CFLAGS comes
>>> after OPT?
>>>
>>
>> Since no one objected I swapped the order in r82259. In case anyone
>> else uses clang to compile Python, this means that -Wno-unused-value
>> will now work to silence the warning about unused return values that
>> is caused by some macros. Probably using -Wno-empty-body is also good
>> to avoid all the warnings triggered by the UCS4 macros in cjkcodecs.
>
> I think you need to come up with a different solution and revert
> the change...
>
> OPT has historically been the only variable to use for
> adjusting the Python C compiler settings.
>
> As the name implies this was usually used to adjust the
> optimizer settings, including raising the optimization level
> from the default or disabling it.
>
> With your change CFLAGS will always override OPT and thus
> any optimization definitions made in OPT will no longer
> have an effect.
>
> Note that CFLAGS defines -O2 on many platforms.
>
> In your particular case, you should try setting OPT to
> "... -Wno-unused-value ..." (ie. replace -Wall with your
> setting).

The python configure environment variables are really confused. If OPT
is intended to be user-overridden for optimization settings, it
shouldn't be used to set -Wall and -Wstrict-prototypes. If it's
intended to set warning options, it shouldn't also set optimization
options. Setting the user-visible customization option on the
configure command line shouldn't stomp unrelated defaults.

In configure-based systems, CFLAGS is traditionally
(http://sources.redhat.com/automake/automake.html#Flag-Variables-Ordering)
the way to tack options onto the end of the command line. Python
breaks this by threading flags through CFLAGS in the makefile, which
means they all get stomped if the user sets CFLAGS on the make command
line. We should instead use another spelling ("CFlags"?) for the
internal variable, and append $(CFLAGS) to it.

AC_PROG_CC is the macro that sets CFLAGS to -g -O2 on gcc-based
systems 
(http://www.gnu.org/software/hello/manual/autoconf/C-Compiler.html#index-AC_005fPROG_005fCC-842).
If Python's configure.in sets an otherwise-empty CFLAGS to -g before
calling AC_PROG_CC, AC_PROG_CC won't change it. Or we could just
preserve the users CFLAGS setting across AC_PROG_CC regardless of
whether it's set, to let the user set CFLAGS on the configure line
without stomping any defaults.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Jeffrey Yasskin]

On Thu, May 27, 2010 at 8:06 PM, Brian Quinlan  wrote:
>
> On May 28, 2010, at 11:57 AM, Reid Kleckner wrote:
>
>> On Thu, May 27, 2010 at 4:13 AM, Brian Quinlan  wrote:
>>>
>>> Keep in mind that this library magic is consistent with the library magic
>>> that the threading module does - unless the user sets Thread.daemon to
>>> True,
>>> the interpreter does *not* exit until the thread does.
>>
>> Is there a compelling to make the threads daemon threads?  If not,
>> perhaps they can just be normal threads, and you can rely on the
>> threading module to wait for them to finish.
>
> Did you read my explanation of the reasoning behind my approach?

You should try to link to explanations. These have been long threads,
and it's often hard to find the previous message where a subject was
addressed.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Jeffrey Yasskin]

On Wed, May 26, 2010 at 3:57 AM, Greg Ewing  wrote:
> Having read through the PEP again, here are my thoughts.
>
> * I'm bothered by the term "future". To my mind, it's
> too long on cleverness and too short on explanativeness.
>
> I think that the standard library is no place for
> cuteness of naming. The name of a stdlib module should
> reflect its functionality in some straightforward and
> obvious way. If I were looking for a thread pool or
> process pool implementation, the word "future" is not
> something that would spring readily to mind.

Please go re-read the older threads on this. For example,
http://mail.python.org/pipermail/python-dev/2010-March/098279.html.

> * It seems unnecessarily verbose to tack "Executor"
> onto the end of every Executor subclass. They could
> simply be called ThreadPool and ProcessPool without
> losing anything.

+0

> * I don't see a strong reason to put this module
> inside a newly-created namespace. If there were a
> namespace called "concurrent", I would expect to find
> other existing concurrency-related modules there as
> well, such as threading and multiprocessing. But we
> can't move them there without breaking existing code.

Again, please re-read the older thread (which you participated in).
For example, http://mail.python.org/pipermail/python-dev/2010-March/098200.html.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Jeffrey Yasskin]

On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan  wrote:
> Rename "executor" => "executer"

-1 for consistency with Java.

> Rename "submit" to "apply"

"apply" focuses attention on the function object, while "submit"
focuses attention, properly I think, on the fact that you're handing
something to the executor to run. So -1.

> Rename "done" to "finished"

"done" is nice and short, and I don't think "finished" or "completed"
will be any less prone to people thinking the task actually ran. So
-1.

> Rename "not_finished" to "pending"

+0.5. Doesn't matter that much, but pending is used elsewhere in the
proposal for this concept. On the other hand, "pending" could be
thought to refer to the state before "running". Possibly "finished"
should be renamed to "done" here, since it's described as '"finished",
contains the futures that completed (finished or were cancelled)',
which uses "finished" for two different concepts.

> Rename "FIRST_COMPLETED" to "ONE_COMPLETED"

"ONE_COMPLETED" could imply that the first result set must contain
exactly one element, but in fact, if multiple tasks finish before the
waiting thread has a chance to wake up, multiple futures could be
returned as done. So -1.


And like my other post, I won't argue about these, leaving the actual
decision up to Brian and Jesse.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Jeffrey Yasskin]

I think the PEP's overall API is good to go.

On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan  wrote:
>
> On 22 May 2010, at 23:59, R. David Murray wrote:
>> If there is still discussion then perhaps the PEP isn't ready for
>> pronouncement yet.  At some point someone can decide it is all
>> bikeshedding and ask for pronouncement on that basis, but I don't
>> think it is appropriate to cut off discussion by saying "it's ready for
>> pronouncement" unless you want increase the chances of its getting
>> rejected.
>
> Here are the new proposed non-documentation changes that I've collected (let
> me know if I've missed any):
>
> ...

I propose to rename the Future.result method to Future.get. "get" is
what Java 
(http://java.sun.com/javase/7/docs/api/java/util/concurrent/Future.html)
and C++ (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3092.pdf
section 30.6.6 para 12) use, and the word "result" doesn't seem
particularly better or worse than "get" for our purposes, which
inclines me to stay consistent.

> We can discuss naming for all eternity and never reach a point where even
> half of the participants are satisfied.

Agreed. To reduce the length of the discussion, I'm not going to reply
to counter-arguments to my proposal, but I think it'll be useful to
Jesse if people who agree or disagree speak up briefly. I'll reply the
other naming proposals in another message.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0

[Jeffrey Yasskin]

On Wed, May 12, 2010 at 6:39 AM, James Y Knight  wrote:
> I think you'll be a lot happier just modifying Psyco than making everyone
> else in the world change their compiler flags.

Aye, there's the rub. Nobody's happier modifying Psyco. :)  But that
just means people will gradually have to stop using psyco in favor of
maintainable JITs. Gcc's not going to change its stack requirements
just because some people think they know what the ABI should be better
than the people defining the ABI. Btw, this has been a problem since
at least gcc-4.4.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mixing float and Decimal -- thread reboot

[Jeffrey Yasskin]

On Wed, Mar 24, 2010 at 2:09 PM, Mark Dickinson  wrote:
> Slight change of topic.  I've been implementing the extra comparisons
> required for the Decimal type and found an anomaly while testing.
> Currently in py3k, order comparisons (but not ==, !=) between a
> complex number and another complex, float or int raise TypeError:
>
 z = complex(0, 0)
 z < int()
> Traceback (most recent call last):
>  File "", line 1, in 
> TypeError: unorderable types: complex() < int()
 z < float()
> Traceback (most recent call last):
>  File "", line 1, in 
> TypeError: unorderable types: complex() < float()
 z < complex()
> Traceback (most recent call last):
>  File "", line 1, in 
> TypeError: unorderable types: complex() < complex()
>
> But Fraction is the odd man out:  a comparison between a Fraction and
> a complex raises a TypeError for complex numbers with nonzero
> imaginary component, but returns a boolean value if the complex number
> has zero imaginary component:
>
 z < Fraction()
> False
 complex(0, 1) < Fraction()
> Traceback (most recent call last):
>  File "", line 1, in 
> TypeError: unorderable types: complex() < Fraction()
>
> I'm tempted to call this Fraction behaviour a bug, but maybe it arises
> from the numeric integration themes of PEP 3141.  Any ideas?

I'd call it a bug.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mixing float and Decimal -- thread reboot

[Jeffrey Yasskin]

On Sat, Mar 20, 2010 at 4:20 PM, Greg Ewing  wrote:
> Adam Olsen wrote:
>
>> For a little context, we have this numeric tower:
>>
>> int -> Fraction -> float -> complex
>>
>> Decimal is more precise, and pays a performance cost for it.  It also
>> seems odd to stick it between float and complex (nobody's planning a
>> ComplexDecimal, right?)  That suggests it should go between Fraction
>> and float.  Decimal/float -> float.
>
> There are two ways in which that linear tower is overly
> simplistic:
>
> * It conflates the notions of exactness and width. They're
> really orthogonal concepts, and to reflect this you would
> need two parallel towers, with exact and inexact versions
> of each type.

It's representing the mathematical concepts
  Integral -> Rational -> Real -> Complex

When designing it, I tried to include a notion of exact/inexact types,
but we couldn't find anything practical to do with them, so we took
them out. It's reasonably easy to design inexact Integral and Rational
types, but pretty hard to design a useful, exact Real type (things
like '==' get uncomputable quickly), so we probably couldn't actually
implement two whole parallel towers.

> * Decimal and float really belong side-by-side in the
> tower, rather than one above the other.

Yep.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C++

[Jeffrey Yasskin]

On Fri, Mar 12, 2010 at 7:54 PM,   wrote:
>    Antoine> s...@pobox.com a écrit :
>    >>
>    >> Traditionally Python has run on some (minority) platforms where C++
>    >> was unavailable.
>
>    Antoine> Is this concern still valid? We are in the 2010s now.
>
> Like I said, *minority* platforms.  Here are some which come to mind as
> quite possibly not supporting C++ well, if at all, yet all have some dialect
> of Python available:
>
>    Windows CE: http://sourceforge.net/projects/pythonce/

http://www.wirelessdevnet.com/channels/pda/training/vcce.html

>    Palm: http://pippy.sourceforge.net/

http://prc-tools.sourceforge.net/

>    iPod: http://ciberjacobo.com/python-ipod

http://ipodlinux.org/wiki/Toolchain

>    OS/2: http://www.andymac.org/

I can't find a modern c++ compiler for OS/2. Then again, your link
only provides python 2.4.

>    QNX: http://sourceforge.net/projects/pyqnx/

http://www.qnx.com/products/tools/

> If all you care about are mainstream Windows, Unix/Linux and Mac OSX
> environments then C++ is no problem.

If you care about any of these minority platforms except OS/2, C++ is
also available, as a google search for " C++" quickly
finds. OS/2 might be available too; I just didn't look for very long.
If you know of platforms that don't support particular features of
C++, please link to documentation of that like Neil Hodgson did. If
not, please stop spreading FUD.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C++

[Jeffrey Yasskin]

On Fri, Mar 12, 2010 at 2:18 PM, Neil Hodgson  wrote:
> Antoine Pitrou:
>
>> Is this concern still valid? We are in the 2010s now.
>> I'm not saying I want us to put some C++ in the core interpreter, but
>> the portability argument sounds a little old...
>
>   There are still viable platforms which only support subsets of C++.
> IIRC, Android does not support exceptions in C++.

Looks like they'll be getting exceptions "soon":
http://groups.google.com/group/android-ndk/browse_thread/thread/89db67ed1fbf6450/4de3dd6105eb26ce?#4de3dd6105eb26ce

But yeah, thanks for the concrete example, and I'd agree that Python
should compile with -fno-exceptions, for a couple more years at least.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jeffrey Yasskin]

On Sun, Mar 7, 2010 at 9:57 AM, P.J. Eby  wrote:
> At 08:39 AM 3/7/2010 -0800, Jeffrey Yasskin wrote:
>> Do you have an example of a language or library that uses the term
>> "future" to refer to what you're talking about? I'm curious to see
>> what it looks like.
>
> The wikipedia page menetioned earlier in the discussion has several
> examples.  Twisted's Deferreds are probably the best example of such a
> system in Python, though, as far as I know.  The features proposed in the
> PEP are basically akin to Twisted's deferToThread(), i.e. "return a future
> for running this code in some other thread".

Nearly all of the wikipedia page's examples refer to exactly this kind
of system, but with syntax to support it instead of putting it in a
library. The distinction between futures and deferreds is that futures
are designed to block, while deferreds are designed to continue
through callbacks. We could say that futures are to deferreds as
threading.Lock is to mutex.mutex.

> That way you
> can write synchronous-looking code that nonetheless executes asynchronously.

 "Asynchronous" in the PEP refers to how the task behind the future
runs, not how users wait for the future. Is there a better way to say
that?

> Twisted uses Deferreds extensively, both for I/O and IPC, and of course for
> database access.  However, the 'defer' module itself is conceptually
> distinct from both the I/O system, event loop, and threads: it's a
> largely-generic framework for managing asynchronous computation.  Parallel
> task queueing is simply *one* possible application of Deferreds; their
> actual purpose is to provide a common API for working with values that
> aren't yet known.
>
> That is to say, futures.
>
> In contrast, the PEP manages only a very limited sort of "future" - the
> completion of a single task, which must be synchronously waited for.  If one
> "future" in the PEP needs to wait for another, you tie up that thread or
> process in a busy loop...  which rather limits how much parallelism you can
> have.

So is it that you just don't like the idea of blocking, and want to
stop anything that relies on it from getting into the standard
library?

Given the set_result and set_exception methods, it's pretty
straightforward to fill in the value of a future from something that
isn't purely computational. Given a way to register "on-done"
callbacks with the future, it would be straightforward to wait for a
future without blocking, too.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jeffrey Yasskin]

On Sun, Mar 7, 2010 at 7:48 AM, P.J. Eby  wrote:
> At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote:
>>
>> P.J. Eby wrote:
>> > (Personally, I think it would be better to just drop the ambitious title
>> > and scope, and go for the "nice task queue" scope.  I imagine, too, that
>> > in that case Jean-Paul wouldn't need to worry about it being raised as a
>> > future objection to Deferreds or some such getting into the stdlib.)
>>
>> This may be a terminology thing - to me futures *are* just a nice way to
>> handle farming tasks out to worker threads or processes. You seem to see
>> them as something more comprehensive than that.
>
> Actual futures are, yes.  Specifically, futures are a mechanism for
> asynchronous computation, whereas the PEP seems to be all about
> synchronously managing parallel tasks.  That's a huge difference.
>
> Technically, the things in the PEP (and by extension, Java's futures) match
> the letter of the definition of a future, but not (IMO) the spirit.  There's
> no clean way to compose them, and at base they're more about parallelism
> than asynchrony.

Do you have an example of a language or library that uses the term
"future" to refer to what you're talking about? I'm curious to see
what it looks like.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jeffrey Yasskin]

On Fri, Mar 5, 2010 at 9:47 PM, Stephen J. Turnbull  wrote:
> Guido van Rossum writes:
>
>  > "Future" is a pretty standard CS term for this concept (as noted
>  > "promise" is another),
>
> I like the term "promise" better.  "Future" is very generic ("not now,
> but later"), whereas a "promise" is something I don't get from you
> now, but you will give me later.
>
> The wikipedia article is not very helpful on the implicit vs. explicit
> distinction.  As far as I can tell from it, that distinction isn't
> really attached to "future" vs "promise."  The only distinction the
> article described was in the context of the Alice language, where a
> future = promise (read-only) plus resolver (mutator).  IMO that's not
> a compelling reason for adopting "future" in Python.

It seems like a good idea to follow the choice other languages have
used for the name (if they tend to agree) regardless of whether the
evil Java followed it too. So let's take a poll:

Io: Uses "future" to refer to the implicit kind
(http://www.iolanguage.com/scm/io/docs/IoGuide.html#Concurrency-Futures)
Alice ML: Uses "future" to refer to the implicit kind, and "promise"
to refer to a handle that can fill in the future
(http://www.ps.uni-saarland.de/alice/manual/futures.html)
Java: Uses "future" to refer to the explicit kind.
(http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html)
AmbientTalk: Uses "future" to refer to something more like a deferred:
you register callbacks to run when the future is resolved.
(http://soft.vub.ac.be/amop/at/tutorial/actors#futures)
C++0x: Uses "future" to refer to the explicit kind; "promise"
similarly to AliceML, and "packaged_task" to get a future from a
callable. (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3035.pdf
section 30.6)
E: Uses "promise" to refer to the implicit kind, and "resolver" to
refer to a handle that can fill in the promise.
(http://wiki.erights.org/wiki/Promise)
Oz: A "future" is a read-only logic variable. I'm not entirely sure
what that means.
(http://www.mozart-oz.org/documentation/dstutorial/node2.html#label61)
PLT Scheme: Uses "future" to refer to the explicit kind.
(http://docs.plt-scheme.org/futures/index.html)
C#: Uses "Task" to refer to the explicit kind.
(http://msdn.microsoft.com/en-us/library/dd321424(VS.100).aspx)
Id, from 1991: Used "I-structure" to refer to the implicit kind.
(http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.4920)
Scala: Uses "Future" to refer to the explicit kind.
(http://www.scala-lang.org/docu/files/api/scala/actors/Actor.html#%21%21%28Any%2CPartialFunction%5BAny%2CA%5D%29)

What languages did I miss? From this list, "future" seems to be the
most popular choice, and it doesn't seem to distinguish between the
implicit and explicit kinds.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jeffrey Yasskin]

On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby  wrote:
> I'm somewhat concerned that, as described, the proposed API ... [creates] yet 
> another alternative (and
> mutually incompatible) event loop system in the stdlib ...

Futures are a blocking construct; they don't involve an event loop.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jeffrey Yasskin]

On Fri, Mar 5, 2010 at 2:54 PM, Antoine Pitrou  wrote:
> Le Fri, 5 Mar 2010 17:03:02 +1100,
> Brian Quinlan  a écrit :
>>
>> The PEP lives here:
>> http://python.org/dev/peps/pep-3148/
>
> Ok, here is my take on it:
>
>> cancel()
>>
>> Attempt to cancel the call. If the call is currently being executed
>> then it cannot be cancelled and the method will return False,
>> otherwise the call will be cancelled and the method will return
>> True.
>
> I think it shouldn't return anything, and raise an exception if
> cancelling failed. It is really an error condition, and ignoring the
> result doesn't seem right.

The caller can't avoid the error here by querying the future, because
of the problem you point out below, so I'm inclined to think that "the
future was already started" should be a return value rather than an
exception (although that may be my C++ background showing through).
Would calling the method try_cancel() work better?

>> Future.running()
>>
>> Return True if the call is currently being executed and cannot be
>> cancelled.
>>
>> Future.done()
>>
>> Return True if the call was successfully cancelled or finished
>> running.
>
> These don't really make sense since the future is executing
> concurrently. By the time the result is returned, it can already be
> wrong. I advocate removing those two methods.

"done()" can only be wrong in one direction though. If it returns
True, it'll never again return False, which can be useful (although
perhaps only for polling, which might be an argument for removing it
anyway).

"running()" becomes True and then False again, so I agree with your
objection. A "started()" function would only go from False to True
once. Maybe that's a better function?

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

[Jeffrey Yasskin]

+1 overall. I'm certainly not concerned with replacing pyc clutter
with pyr clutter. I do like that you haven't _increased_ the number of
extraneous siblings of .py files.

I have a couple bikesheddy or "why didn't you do this" comments. I'll
be perfectly satisfied with an answer or a line in the pep.

1. Why the -R flag? It seems like this is a uniform improvement, so it
should be the default. Have faith in your design! ;-)

2. Vitor's suggestion to make 1 "pyr" directory per directory and
stick all the .pyc's there would solve the "pyc clutter" problem. Any
reason not to do that? Trying to make it 1-pyr-per-directory-hierarchy
as Ben suggested seems unworkable.  The one problem with this would
seem to be filename length limits; do we care about those anymore?

3. It seems like .pyr directories are nicely forward-compatible with
other uses like version-specific .so's or JIT caches. I don't think
this PEP needs to flesh out any of those other possibilities though.

4. -1 to a moratorium on bytecode changes. No moratorium can last
forever, and then packagers will be back to the same problem. The
rationale for 3003 doesn't seem to apply here.

On Sat, Jan 30, 2010 at 4:00 PM, Barry Warsaw  wrote:
> PEP: 3147
> Title: PYC Repository Directories
> Version: $Revision$
> Last-Modified: $Date$
> Author: Barry Warsaw 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 2009-12-16
> Python-Version: 3.2
> Post-History:
>
>
> Abstract
> 
>
> This PEP describes an extension to Python's import mechanism which
> improves sharing of Python source code files among multiple installed
> different versions of the Python interpreter.  It does this by
> allowing many different byte compilation files (.pyc files) to be
> co-located with the Python source file (.py file).  The extension
> described here can also be used to support different Python
> compilation caches, such as JIT output that may be produced by an
> Unladen Swallow [1]_ enabled C Python.
>
>
> Rationale
> =
>
> Linux distributions such as Ubuntu [2]_ and Debian [3]_ provide more
> than one Python version at the same time to their users.  For example,
> Ubuntu 9.10 Karmic Koala can install Python 2.5, 2.6, and 3.1, with
> Python 2.6 being the default.
>
> In order to ease the burden on operating system packagers for these
> distributions, the distribution packages do not contain Python version
> numbers [4]_; they are shared across all Python versions installed on
> the system.  Putting Python version numbers in the packages would be a
> maintenance nightmare, since all the packages - *and their
> dependencies* - would have to be updated every time a new Python
> release was added or removed from the distribution.  Because of the
> sheer number of packages available, this amount of work is infeasible.
>
> For pure Python modules, sharing is possible because upstream
> maintainers typically support multiple versions of Python in a source
> compatible way.  In practice though, it is well known that pyc files
> are not compatible across Python major releases.  A reading of
> import.c [5]_ in the Python source code proves that within recent
> memory, every new CPython major release has bumped the pyc magic
> number.
>
> Even C extensions can be source compatible across multiple versions of
> Python.  Compiled extension modules are usually not compatible though,
> and PEP 384 [6]_ has been proposed to address this by defining a
> stable ABI for extension modules.
>
> Because the distributions cannot share pyc files, elaborate mechanisms
> have been developed to put the resulting pyc files in non-shared
> locations while the source code is still shared.  Examples include the
> symlink-based Debian regimes python-support [7]_ and python-central
> [8]_.  These approaches make for much more complicated, fragile,
> inscrutable, and fragmented policies for delivering Python
> applications to a wide range of users.  Arguably more users get Python
> from their operating system vendor than from upstream tarballs.  Thus,
> solving this pyc sharing problem for CPython is a high priority for
> such vendors.
>
> This PEP proposes a solution to this problem.
>
>
> Proposal
> 
>
> Python's import machinery is extended to search for byte code cache
> files in a directory co-located with the source file, but with an
> extension 'pyr'.  The pyr directory contains individual files with the
> cached byte compilation of the source code, identical to current pyc
> and pyo files.  The files inside the pyr directory retain their file
> extensions, but the base name is replaced by the hexlified [10]_ magic
> number of the Python version the byte code is compatible with.
>
> The file extension pyr was chosen because 'r' is a mnemonic for
> 'repository', and there appears to be no prior uses of the extension
> [9]_.
>
> For example, a module `foo` with source code in `foo.py` and byte
> compiled with Python 2.5, Python 2.6, Python 2.6 `-O`, Pyth

Re: [Python-Dev] PEP 3147: PYC Repository Directories

[Jeffrey Yasskin]

On Sat, Jan 30, 2010 at 8:22 PM, Vitor Bosshard  wrote:
> 2010/1/31 Nick Coghlan :
>
>>> Can't a VCS be configured to ignore a .pyr directory just as easily as
>>> it can be configured to ignore a .pyc file?
>>
>> Yes they can.
>
>
> Of course they can, but not out of the box. It was just an example off
> the top of my head.

Mercurial, at least, doesn't ignore .pyc files out of the box either.
I'm sure it will be a terrible hardship for people to add one extra
line to their .ignore files.

> A trickier case: My GUI app offers scripting facilities. The
> associated open file dialog hides all .pyc files, and users select
> just from .py files. if subfolders are introduced, however, they can't
> be hidden away as easily. Filtering by extension is standard
> functionality in all GUI toolkits. Hiding a fraction of subfolders is
> not.

You're saying you can filter files by extension, but you cannot filter
directories by extension?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Jeffrey Yasskin]

On Wed, Jan 27, 2010 at 2:14 PM, Daniel Stutzbach
 wrote:
> On Wed, Jan 27, 2010 at 3:49 PM, Raymond Hettinger
>  wrote:
>>
>> Also, am not sure if this affects psyco or the other
>> implementations such as Jython which may implement
>> lists in terms of existing Java containers.
>
> Or Unladen Swallow.  I'm -1 on mucking with any of the fundamental data
> structures until the Unladen Swallow patch lands (assuming it lands).

Don't block internal changes for our sake. For the most part, we've
already made plans to make them non-issues. In any cases we haven't,
we need to make them non-issues anyway to avoid slowing down python
development in the future.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Jeffrey Yasskin]

On Wed, Jan 27, 2010 at 11:16 AM, Collin Winter  wrote:
> We absolutely do not want CPython to include a copy of LLVM in its
> source tree. Unladen Swallow has done this to make it easier to pick
> up changes to LLVM's codebase as we make them, but this is not a
> viable model for CPython's long-term development. As mentioned in
> http://www.python.org/dev/peps/pep-3146/#managing-llvm-releases-c-api-changes,
> one of our full-time engineers is tasked with fixing all critical
> issues in LLVM before LLVM's 2.7 release so that CPython can simply
> use that release.

I'm now tracking my to-do list for LLVM 2.7 in
http://code.google.com/p/unladen-swallow/issues/detail?id=131.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Jeffrey Yasskin]

On Mon, Jan 25, 2010 at 1:50 PM, "Martin v. Löwis"  wrote:
>> We really do need precise descriptions of the problems so we can avoid them.
>
> One family of problems is platform lack of initializer support in the
> object file format; any system with traditional a.out (or b.out) is
> vulnerable (also, COFF is, IIRC).
>
> The solution e.g. g++ came up with is to have the collect2 linker
> replacement combine all such initializers into a synthesized function
> __main; this function then gets "magically" called by main(), provided
> that main() itself gets compiled by a C++ compiler. Python used to have
> a ccpython.cc entry point to support such systems.
>
> This machinery is known to fail in the following ways:
> a) main() is not compiled with g++: static objects get not constructed
> b) code that gets linked into shared libraries (assuming the system
>   supports them) does not get its initializers invoked.
> c) compilation of main() with a C++ compiler, but then linking with ld
>   results in an unresolved symbol __main.

Thank you for the details. I'm pretty confident that (a) and (c) will
not be a problem for the Unladen Swallow merge because we switched
python.c (which holds main()) and linking to use the C++ compiler when
LLVM's enabled at all:
http://code.google.com/p/unladen-swallow/source/browse/trunk/Makefile.pre.in.
Python already had some support for this through the LINKCC configure
variable, but it wasn't being used to compile main(), just to link.

(b) could be a problem if we depend on LLVM as a shared library on one
of these platforms (and, of course, if LLVM's JIT supports these
systems at all). The obvious answers are: 1) --without-llvm on these
systems, 2) link statically on these systems, 3) eliminate the static
constructors. There may also be less obvious answers.

> Not sure whether U-S has any global C++ objects that need construction
> (but I would be surprised if it didn't).

I checked with them, and LLVM would welcome patches to remove these if
we need to. They already have a llvm::ManagedStatic class
(http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Support/ManagedStatic.h?view=markup)
with no constructor that we can replace most of them with.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Jeffrey Yasskin]

On Mon, Jan 25, 2010 at 10:44 AM, Tres Seaver  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Collin Winter wrote:
>
>> For reference, what are these "obscure platforms" where static
>> initializers cause problems?
>
> It's been a long while since I had to deal with it, but the "usual
> suspets" back in the day were HP-UX, AIX, and Solaris with non-GCC
> compilers, as well as Windows when different VC RT libraries got into
> the mix.

So then the question is, will this cause any problems we care about?
Do the problems still exist, or were they eliminated in the time
between "back in the day" and now? In what circumstances do static
initializers have problems? What problems do they have? Can the
obscure platforms work around the problems by configuring with
--without-llvm? If we eliminate static initializers in LLVM, are there
any other problems?

We really do need precise descriptions of the problems so we can avoid them.

Thanks,
Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Jeffrey Yasskin]

On Thu, Jan 21, 2010 at 3:49 PM, Barry Warsaw  wrote:
> Martin's follow up reminds me what the issues with C++ here are.  They center
> around which C++ compilers you use on which platforms.  Solaris, and to some
> extent Windows IIRC, were the most problematic for the work I was doing 3+
> years ago.  Usually, everything's fine if you're compiling all the code with
> the same compiler.  The problem comes if you want to mix say C++ libraries
> compiled with Sun's C++ compiler and Python compiled with g++.  Unlike the C
> world, where it doesn't matter much, once C++ is in the mix you have to be
> very careful about how all your libraries and core binary are compiled and
> linked.

I'm not an expert here, but this is why we've tried to avoid exposing
C++ functions into the public API. As long as the public API stays C,
we only require that LLVM and Python are compiled with the same
compiler, right? Extension modules, since they only call C API
functions, can be compiled with different compilers.

I could imagine a problem if Python+LLVM link in one libstdc++, and an
extension module links in a different one, even if no C++ objects are
passed across the boundary. Does that cause problems in practice? We'd
have the same problems as from having two different C runtimes in the
same application.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Jeffrey Yasskin]

On Thu, Jan 21, 2010 at 10:09 AM, Hanno Schlichting  wrote:
> I'm a relative outsider to core development (I'm just a Plone release
> manager), but'll allow myself a couple of questions. Feel free to
> ignore them, if you think they are not relevant at this point :-) I'd
> note that I'm generally enthusiastic and supportive of the proposal :)
> As a data point, I can add that all tests of Zope 2.12 / Plone 4.0 and
> their dependency set run fine under Unladen Swallow.

Hi, thanks for checking that!

> On Wed, Jan 20, 2010 at 11:27 PM, Collin Winter  
> wrote:
>> We have chosen to reuse a set of existing compiler libraries called LLVM
>> [#llvm]_ for code generation and code optimization.
>
> Would it be prudent to ask for more information about the llvm
> project? Especially in terms of its non-code related aspects. I can
> try to hunt down this information myself, but as a complete outsider
> to the llvm project this takes much longer, compared to someone who
> has interacted with the project as closely as you have.
>
> Questions like:
>
> - Who holds the copyright, is there a foundation or is there a risk of
> the project getting into trouble because of copyright issues?

See http://llvm.org/docs/DeveloperPolicy.html#clp. The University of
Illinois holds the copyright.

> - What licence is the code and the tools under and what affect does
> that have on the code generated by the JIT compiler?

LLVM's under the University of Illinois/NCSA license, which is
BSD-like: http://www.opensource.org/licenses/UoI-NCSA.php or
http://llvm.org/releases/2.6/LICENSE.TXT. If I understand correctly,
code transformed by a compiler isn't a work derived from that
compiler, so the compiler's license has no bearing on the license of
the output code.

> - What's the bus factor of the project? Is it in practice dependent on
> a single BDFL or a single company like Apple?

I believe the project would survive either Chris Lattner or Apple
getting bored. Even though Apple provides much of the development
effort right now, LLVM has enough users (http://llvm.org/Users.html)
to pick up the slack if they need to. Adobe, Cray, Google, NVidia, and
Rapidmind all contribute patches back to LLVM. (And Unladen Swallow
isn't the only reason Google cares.)

> - What's the status of the documentation or web presence? Does the
> project consist only of developers or has it been able to attract
> people from other backgrounds?

The documentation's quite good: http://llvm.org/docs/. What other
backgrounds are you asking about? I don't think they have any
sculptors on the project. ;)

> On the code related side you described general enthusiasm and support
> of the llvm community, but noted that some aspect of it aren't as
> mature as others. You also noted that given Python's good test
> coverage you have been able to find problems in llvm itself. This
> raises some questions to me:
>
> - Does the llvm project employ automated test driven development, are
> there functioning nightly test runs?

Yep: http://google1.osuosl.org:8011/console. We've been adding tests
as we fix bugs so they don't regress. They also have a larger
"nightly" test suite that runs whole programs,
http://llvm.org/docs/TestingGuide.html#quicktestsuite, but it isn't
actually run every night.

> - What do they do to ensure a good quality of their software, how many
> critical regressions have their been in final releases?

In addition to the unit tests and "nightly" tests, they have beta and
release candidate phases during which they ask the community to test
out the new release with our projects.
Nick Lewycky (cc'ed) answered your second question with: "back in ...
1.7-ish? we had to do a 1.7.1 release within 24 hours due to an
accidental release with a critical regression". Since LLVM's now on
2.6, I take that as "not very many".

> - Is there a clearly distinguished core and stable part of the project
> that is separate from more experimental ideas?

Not really.

> - What's the status of their known bugs? Do "important" issues get
> resolved in a reasonable time?

I'm not entirely sure how to answer that. The bugs database is at
http://llvm.org/bugs, and they've tended to fix bugs I ran into pretty
quickly, when those bugs affected the static compiler. JIT bugs get
handled somewhat more slowly, but I've also been able to fix those
bugs myself fairly easily.

> - Did they have security problems related to their codebase, have
> those been handled in a professional manner?

I haven't been exposed to their resolution of any security problems,
so I can't really answer that either. Inside of Python, LLVM will be
pretty insulated from the outside world, so the risk exposure
shouldn't be _too_ great, but I'm not really qualified to evaluate
that. If a Python user could send arbitrary IR into LLVM, they could
probably exploit the process, but Unladen doesn't give users a way to
do that. Nick can talk more about this.

> There's probably more of these questions you tend to ask yoursel

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Jeffrey Yasskin]

On Wed, Jan 20, 2010 at 5:56 PM, Collin Winter  wrote:
> On Wed, Jan 20, 2010 at 5:14 PM, Terry Reedy  wrote:
>> Given the slight benefits compared to the costs, I think this, in its
>> current state, should be optional, such as is psyco.
>>
>> Psyco has a similar time-space tradeoff, except that the benefit is much
>> greater (10x+ has been reported by happy users) for certain types of code
>> (lots of integer arithmethic, which psyco unboxes). Why should your changes
>> be blessed and psyco not? While now 2.x only, like UnSw, there are
>> apparently people working on a 3.x version. Pysco makes its tradeoffs
>> voluntary, and easy to switch on and off where the benefits are worth the
>> cost to the particular user.

I think you're right that users should be able to turn off the JIT at
runtime (not just configure time), and get pretty close to no slowdown
or memory overhead compared to pre-Unladen CPython. We currently have
a command-line switch to do just that, but I suspect it doesn't get as
close to pre-Unladen performance/memory use as it could. I've filed
http://code.google.com/p/unladen-swallow/issues/detail?id=123 to keep
track of this.

>> I do not think that standard CPython should be
>> made incompatible with a module that greatly benefits certain users who have
>> been using it for years.

Standard CPython is made incompatible with psyco at every release. ;)
Someone had to update psyco to support CPython 2.5 and 2.6, and
they'll have to update it to support each 3.x as well. I don't think
there's anything we've done in Unladen that will make it impossible to
update psyco to coexist with it; it's just that psyco turned out to be
too complicated for us to keep it working through our bytecode
changes.

> - While Psyco provides large benefits to numerical workloads, the
> benefits to other systems we have at Google are much, much smaller, in
> the 15-30% range.

I don't think it's just Google workloads that Psyco provides a
smaller-than-advertised benefit to.

>> Your results suggest that speeding up garden-variety Python code is harder
>> than it sometimes seems. I wonder how your results from fancy codework
>> compare, for instance, with simply making built-in names reserved, so that,
>> for instance, len =  is illegal, and all such names get
>> dereferenced at compile time.
>
> Yes, if you change the language, certain optimizations become simpler
> or some code becomes faster. However, changing the language in subtle
> ways -- like changing when builtins are bound -- increases the barrier
> to adoption, which is why we chose to implement the language as
> specified. Auditing Google's entire Python codebase to correct for
> these subtle changes would be prohibitive, as would the need to
> retrain all our engineers who use Python to educate them on the
> differences between the two languages.

Again, this isn't just Google. Do you really want to audit your code
and all the libraries you use to make sure they don't depend on late
builtin binding?

We have considered making language changes like this at some -O level
to support people who want every ounce of speed (in the same way that
gcc supports a -ffast-math option for people who don't care about
floating point accuracy), and we think it'll be a good thing to
investigate in the future, but it's not the kind of change I'd want to
make to the default language.

>> I guess what I am mainly saying is that there are several possible ways to
>> speed up Python 3 execution (including others not mentioned here) and it is
>> not at all clear to me that this particular one is in any sense 'best of
>> breed'. If it disables other approaches, I think it should be optional for
>> the standard PSF distribution.

As far as I can tell, it does not disable other approaches.

Thanks for the comments!
Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Jeffrey Yasskin]

On Mon, Nov 2, 2009 at 4:15 AM, Antoine Pitrou  wrote:
> Martin v. Löwis  v.loewis.de> writes:
>>
> [gil_drop_request]
>> Even if it is read from memory, I still wonder what might happen on
>> systems that require explicit memory barriers to synchronize across
>> CPUs. What if CPU 1 keeps reading a 0 value out of its cache, even
>> though CPU 1 has written an 1 value a long time ago?
>>
>> IIUC, any (most?) pthread calls would cause synchronization in that
>> case, which is why applications that also use locks for reading:
>>
>> http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap04.html#tag_04_10
>>
>> Of course, on x86, you won't see any issues, because it's cache-coherent
>> anyway.
>
> I think there are two things here:
> - all machines Python runs on should AFAIK be cache-coherent: CPUs synchronize
> their views of memory in a rather timely fashion.
> - memory ordering: writes made by a CPU can be seen in a different order by
> another CPU (i.e. CPU 1 writes A before B, but CPU 2 sees B written before 
> A). I
> don't see how this can apply to gil_drop_request, since it's a single variable
> (and, moreover, only a single bit of it is significant).
>
> (there's an explanation of memory ordering issues here:
> http://www.linuxjournal.com/article/8211)
>
> As a side note, I remember Jeffrey Yasskin trying to specify an ordering model
> for Python code
> (see http://code.google.com/p/unladen-swallow/wiki/MemoryModel).

Note that that memory model was only for Python code; the C code
implementing it is subject to the C memory model, which is weaker (and
not even fully defined until the next C standard comes out).

To be really safe, we ought to have a couple primitives implementing
"atomic" rather than just "volatile" instructions, but until then a
signal that's just saying "do something" rather than "here's some data
you should look at" should be ok as a volatile int.

I'd like to look at the patch in detail, but I can't guarantee that
I'll get to it in a timely manner. I'd say check it in and let more
threading experts look at it in the tree. We've got some time before a
release for people to fix problems and make further improvements. +1
to Martin's request for detailed documentation though. :)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [unladen-swallow] PEP 384: Defining a Stable ABI

[Jeffrey Yasskin]

On Wed, May 20, 2009 at 10:34 AM, Antoine Pitrou  wrote:
> Jeffrey Yasskin  gmail.com> writes:
>>
>> Sorry, I didn't mean to get into a GIL debate. All I'm saying is that
>> I don't think changing the definition of Py_INCREF and Py_DECREF
>> justifies going to Python 4.0, so I don't think their definitions
>> should be part of the ABI. If that's not what the ABI means, that's ok
>> too.
>
> Consider, though, that if Py_INCREF and Py_DECREF are not part of the ABI,
> enabling the ABI-specific preprocessor symbol will hide them, which might (or
> might not!) annoy a lot of extension writers.

Yes, that's my intention. (Well, not the annoying part, but making
them use Py_IncRef instead for ABI compatibility is, I think, a good
thing.) If they don't want ABI compatibility, they shouldn't ask for
it. Giving them something else useful to ask for is why I mentioned an
API compatibility mode.

To decrease the annoyance of having to change source code, we could
have Py_INCREF(x) expand to Py_IncRef(x) in ABI-compatibility mode.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [unladen-swallow] PEP 384: Defining a Stable ABI

[Jeffrey Yasskin]

On Wed, May 20, 2009 at 10:14 AM, Antoine Pitrou  wrote:
> Jeffrey Yasskin  gmail.com> writes:
>>
>> Over an 8-year lifetime for Python 3, Moore's law predicts that
>> desktop systems will have up to 64 cores, at which point even the
>> simplest GIL-removal strategy of making refcounts atomic will be a
>> win, despite the 2x performance loss for a single thread.
>
> That's only if you think all workloads parallelize easily (and with little 
> work
> from the average programmer), which sounds a bit presumptuous. When you have a
> GUI application and the perceived performance is driven by UI responsivity,
> spawning dozens of threads can little to improve the picture ("GUI 
> application"
> here can mean a feature-rich Web application, too).
>
> As for desktop systems having 64 cores, that's unless the available die space
> gets used for something else instead, e.g. an integrated GPU. Or unless the
> desktop dies in favor of something else (e.g. laptops with small tightly
> integrated chips). The former is already in AMD's and Intel's plans. The 
> latter
> could be happening right now.
>
> And we're not even talking about embedded platforms, or virtual machines 
> where a
> 64-core server is partitioned into 64 "single-core" systems.
>
> (and then there's the whole threading vs processing debate ;-))
>
> Endly, removing the GIL means you have to make all base types (especially
> containers) thread-safe without sacrificing their performance. I don't think
> it's just about reference-counting.
>
>
> That said, the Py_Incref() and Py_Decref() functions already exist. Perhaps 
> they
> should be advertised a bit more in the documentation. The day a hypothetical
> Python implementation gets rid of reference-counting while remaining binary
> compatible with the rest of the API (which rules out PyPy), and gets much 
> faster
> in the process, I think people will happily suffer a small recompile.

Sorry, I didn't mean to get into a GIL debate. All I'm saying is that
I don't think changing the definition of Py_INCREF and Py_DECREF
justifies going to Python 4.0, so I don't think their definitions
should be part of the ABI. If that's not what the ABI means, that's ok
too.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Fwd: Re: PEP 384: Defining a Stable ABI]

[Jeffrey Yasskin]

On Tue, May 19, 2009 at 2:09 AM, William Reade
 wrote:
> (for example, all(?) PyCxx modules use PyCode_New and
> PyFrame_New to get nicer tracebacks)

Specifically for this, I think it'd be nice to expose a function to do
this directly. I recently added PyCode_NewEmpty
(http://svn.python.org/view?view=rev&revision=72487) to go part of the
way here. I didn't go farther because I didn't have a big enough
picture. If most uses of PyFrame_New are really just to call into
Python with a nice traceback, I think it'd be a good idea to add such
a function to ceval.h next to PyEval_Call*(). We can only credibly
tell people to use only the ABI functions when we have an ABI
replacement for the (sane uses of) non-ABI calls.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [unladen-swallow] PEP 384: Defining a Stable ABI

[Jeffrey Yasskin]

A couple thoughts:

I'm with the people who think the refcount should be accessed through
functions by apps that want ABI compatibility. In particular,
GIL-removal efforts are guaranteed to change how the refcount is
modified, but there's a good chance they wouldn't have to change the
API. (We have some ideas for how to maintain source compatibility in
the absence of a GIL:
http://code.google.com/p/unladen-swallow/wiki/ExtensionModules#Reference_Counting)
Over an 8-year lifetime for Python 3, Moore's law predicts that
desktop systems will have up to 64 cores, at which point even the
simplest GIL-removal strategy of making refcounts atomic will be a
win, despite the 2x performance loss for a single thread. I wouldn't
want an ABI to rule that out.

I do think the refcounting macros should remain present in the API
(not ABI) for apps that only need source compatibility and want the
extra speed.

I wonder if it makes sense to specify an API compatibility mode in
this PEP too.

"Py_LIMITED_API" may not be the right macro name—it didn't imply
anything about an ABI when I first saw it. Might it make sense to use
Py_ABI_COMPATIBILITY=### instead? (Where ### could be an ISO date like
20090520.) That would put "ABI" in the macro name and make it easier
to define new versions later if necessary. (New versions would help
people compile against a new version of Python and be confident they
had something that would run against old versions.) If we never define
a new version, defining it to a number instead of just anything
doesn't really hurt.

It's probably worth pointing out in the PEP that the fact that
PyVarObject.ob_size is part of the ABI means that PyObject cannot
change size, even by adding fields at the end.

Right now, the globals representing types are defined like
"PyAPI_DATA(PyTypeObject) PyList_Type;". To allow the core to use the
new type creation functions, it might be useful to make the ABI type
objects PyTypeObject* constants instead.

In general, this looks really good. Thanks!

Jeffrey

On Sun, May 17, 2009 at 1:54 PM, "Martin v. Löwis"  wrote:
>
> Thomas Wouters reminded me of a long-standing idea; I finally
> found the time to write it down.
>
> Please comment!
>
> Regards,
> Martin
>
> PEP: 384
> Title: Defining a Stable ABI
> Version: $Revision: 72754 $
> Last-Modified: $Date: 2009-05-17 21:14:52 +0200 (So, 17. Mai 2009) $
> Author: Martin v. Löwis 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 17-May-2009
> Python-Version: 3.2
> Post-History:
>
> Abstract
> 
>
> Currently, each feature release introduces a new name for the
> Python DLL on Windows, and may cause incompatibilities for extension
> modules on Unix. This PEP proposes to define a stable set of API
> functions which are guaranteed to be available for the lifetime
> of Python 3, and which will also remain binary-compatible across
> versions. Extension modules and applications embedding Python
> can work with different feature releases as long as they restrict
> themselves to this stable ABI.
>
> Rationale
> =
>
> The primary source of ABI incompatibility are changes to the lay-out
> of in-memory structures. For example, the way in which string interning
> works, or the data type used to represent the size of an object, have
> changed during the life of Python 2.x. As a consequence, extension
> modules making direct access to fields of strings, lists, or tuples,
> would break if their code is loaded into a newer version of the
> interpreter without recompilation: offsets of other fields may have
> changed, making the extension modules access the wrong data.
>
> In some cases, the incompatibilities only affect internal objects of
> the interpreter, such as frame or code objects. For example, the way
> line numbers are represented has changed in the 2.x lifetime, as has
> the way in which local variables are stored (due to the introduction
> of closures). Even though most applications probably never used these
> objects, changing them had required to change the PYTHON_API_VERSION.
>
> On Linux, changes to the ABI are often not much of a problem: the
> system will provide a default Python installation, and many extension
> modules are already provided pre-compiled for that version. If additional
> modules are needed, or additional Python versions, users can typically
> compile them themselves on the system, resulting in modules that use
> the right ABI.
>
> On Windows, multiple simultaneous installations of different Python
> versions are common, and extension modules are compiled by their
> authors, not by end users. To reduce the risk of ABI incompatibilities,
> Python currently introduces a new DLL name pythonXY.dll for each
> feature release, whether or not ABI incompatibilities actually exist.
>
> With this PEP, it will be possible to reduce the dependency of binary
> extension modules on a specific Python feature release, and applications
> embedding Python can be made work with differen

[Python-Dev] Documenting lnotab

[Jeffrey Yasskin]

Hi all.

I've got a patch to add some documentation for lnotab and its use in
tracing at http://bugs.python.org/issue6042. I think it's correct, but
it's complicated so I'm looking for someone who was around when it was
designed to check. I'm also proposing a change to the semantics of
PyCode_CheckLineNumber and want to know whether I should consider it
public.

Thanks to anyone who takes a look!
Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rethinking intern() and its data structure

[Jeffrey Yasskin]

On Thu, Apr 9, 2009 at 6:24 PM, John Arbash Meinel
 wrote:
> Greg Ewing wrote:
>> John Arbash Meinel wrote:
>>> And the way intern is currently
>>> written, there is a third cost when the item doesn't exist yet, which is
>>> another lookup to insert the object.
>>
>> That's even rarer still, since it only happens the first
>> time you load a piece of code that uses a given variable
>> name anywhere in any module.
>>
>
> Somewhat true, though I know it happens 25k times during startup of
> bzr... And I would be a *lot* happier if startup time was 100ms instead
> of 400ms.

I think you have plenty of a case to try it out. If you code it up and
it doesn't speed anything up, well then we've learned something, and
maybe it'll be useful anyway for the memory savings. If it does speed
things up, well then Python's faster. I wouldn't waste time arguing
about it before you have the change written.

Good luck!
Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyDict_SetItem hook

[Jeffrey Yasskin]

On Thu, Apr 2, 2009 at 5:57 PM, Guido van Rossum  wrote:
> On Thu, Apr 2, 2009 at 3:07 PM, Raymond Hettinger  wrote:
>>> Wow. Can you possibly be more negative?
>>
>> I think it's worse to give the poor guy the run around
>
> Mind your words please.
>
>> by making him run lots of random benchmarks.  In
>> the end, someone will run a timeit or have a specific
>> case that shows the full effect.  All of the respondents so far seem to have
>> a clear intuition that hook is right in the middle of a critical path.
>>  Their intuition matches
>> what I learned by spending a month trying to find ways
>> to optimize dictionaries.
>>
>> Am surprised that there has been no discussion of why this should be in the
>> default build (as opposed to a compile time option).  AFAICT, users have not
>> previously
>> requested a hook like this.
>
> I may be partially to blame for this. John and Stephan are requesting
> this because it would (mostly) fulfill one of the top wishes of the
> users of Wingware. So the use case is certainly real.
>
>> Also, there has been no discussion for an overall strategy
>> for monitoring containers in general.  Lists and tuples will
>> both defy this approach because there is so much code
>> that accesses the arrays directly.  Am not sure whether the
>> setitem hook would work for other implementations either.
>
> The primary use case is some kind of trap on assignment. While this
> cannot cover all cases, most non-local variables are stored in dicts.
> List mutations are not in the same league, as use case.
>
>> It seems weird to me that Collin's group can be working
>> so hard just to get a percent or two improvement in specific cases for
>> pickling while python-dev is readily entertaining a patch that slows down
>> the entire language.
>
> I don't actually believe that you can know whether this affects
> performance at all without serious benchmarking. The patch amounts to
> a single global flag check as long as the feature is disabled, and
> that flag could be read from the L1 cache.

When I was optimizing the tracing support in the eval loop, we started
with two memory loads and an if test. Removing the whole thing saved
about 3% of runtime, although I think that had been as high as 5% when
Neal measured it a year before. (That indicates that the exact
arrangement of the code can affect performance in subtle and annoying
ways.) Removing one of the two loads saved about 2% of runtime. I
don't remember exactly which benchmark that was; it may just have been
pybench.

Here, we're talking about introducing a load+if in dicts, which is
less critical than the eval loop, so I'd guess that the effect will be
less than 2% overall. I do think the real-life benchmarks are worth
getting for this, but they may not predict the effect after other code
changes. And I don't really have an opinion on what performance hit
for normal use is worth better debugging.

>> If my thoughts on the subject bug you, I'll happily
>> withdraw from the thread.  I don't aspire to be a
>> source of negativity.  I just happen to think this proposal isn't a good
>> idea.
>
> I think we need more proof either way.
>
>> Raymond
>>
>>
>>
>> - Original Message - From: "Guido van Rossum" 
>> To: "Raymond Hettinger" 
>> Cc: "Thomas Wouters" ; "John Ehresman"
>> ; 
>> Sent: Thursday, April 02, 2009 2:19 PM
>> Subject: Re: [Python-Dev] PyDict_SetItem hook
>>
>>
>> Wow. Can you possibly be more negative?
>>
>> 2009/4/2 Raymond Hettinger :
>>>
>>> The measurements are just a distractor. We all already know that the hook
>>> is being added to a critical path. Everyone will pay a cost for a feature
>>> that few people will use. This is a really bad idea. It is not part of a
>>> thorough, thought-out framework of container hooks (something that would
>>> need a PEP at the very least). The case for how it helps us is somewhat
>>> thin. The case for DTrace hooks was much stronger.
>>>
>>> If something does go in, it should be #ifdef'd out by default. But then, I
>>> don't think it should go in at all.
>>>
>>>
>>> Raymond
>>>
>>>
>>>
>>>
>>> On Thu, Apr 2, 2009 at 04:16, John Ehresman  wrote:

 Collin Winter wrote:
>
> Have you measured the impact on performance?

 I've tried to test using pystone, but am seeing more differences between
 runs than there is between python w/ the patch and w/o when there is no
 hook
 installed. The highest pystone is actually from the binary w/ the patch,
 which I don't really believe unless it's some low level code generation
 affect. The cost is one test of a global variable and then a switch to
 the
 branch that doesn't call the hooks.

 I'd be happy to try to come up with better numbers next week after I get
 home from pycon.
>>>
>>> Pystone is pretty much a useless benchmark. If it measures anything, it's
>>> the speed of the bytecode dispatcher (and it doesn't measure it
>>> particularly
>>> well.) PyBench isn't any better, in my 

Re: [Python-Dev] PyDict_SetItem hook

[Jeffrey Yasskin]

On Fri, Apr 3, 2009 at 12:28 PM, Michael Foord
 wrote:
> Collin Winter wrote:
>>
>> On Fri, Apr 3, 2009 at 2:27 AM, Antoine Pitrou 
>> wrote:
>>
>>>
>>> Thomas Wouters  python.org> writes:
>>>

 Pystone is pretty much a useless benchmark. If it measures anything,
 it's the

>>>
>>> speed of the bytecode dispatcher (and it doesn't measure it particularly
>>> well.)
>>> PyBench isn't any better, in my experience.
>>>
>>> I don't think pybench is useless. It gives a lot of performance data
>>> about
>>> crucial internal operations of the interpreter. It is of course very
>>> little
>>> real-world, but conversely makes you know immediately where a performance
>>> regression has happened. (by contrast, if you witness a regression in a
>>> high-level benchmark, you still have a lot of investigation to do to find
>>> out
>>> where exactly something bad happened)
>>>
>>> Perhaps someone should start maintaining a suite of benchmarks,
>>> high-level and
>>> low-level; we currently have them all scattered around (pybench, pystone,
>>> stringbench, richard, iobench, and the various Unladen Swallow
>>> benchmarks; not
>>> to mention other third-party stuff that can be found in e.g. the Computer
>>> Language Shootout).
>>>
>>
>> Already in the works :)
>>
>> As part of the common standard library and test suite that we agreed
>> on at the PyCon language summit last week, we're going to include a
>> common benchmark suite that all Python implementations can share. This
>> is still some months off, though, so there'll be plenty of time to
>> bikeshed^Wrationally discuss which benchmarks should go in there.
>>
>
> Where is the right place for us to discuss this common benchmark and test
> suite?

Dunno. Here, by default, but I'd subscribe to a tests-sig or
commonlibrary-sig or benchmark-sig if one were created.

> As the benchmark is developed I would like to ensure it can run on
> IronPython.

We want to ensure the same thing for the current unladen swallow
suite. If you find ways it currently doesn't, send us patches (until
we get it moved to the common library repository at which point you'll
be able to submit changes yourself). You should be able to check out
http://unladen-swallow.googlecode.com/svn/tests independently of the
rest of the repository. Follow the instructions at
http://code.google.com/p/unladen-swallow/wiki/Benchmarks to run
benchmarks though perf.py. You'll probably want to select benchmarks
individually rather than accepting the default of "all" because it's
currently not very resilient to tests that don't run on one of the
comparison pythons.

Personally, I'd be quite happy moving our performance tests into the
main python repository before the big library+tests move, but I don't
know what directory to put it in, and I don't know what Collin+Thomas
think of that.

> The test suite changes will need some discussion as well - Jython and
> IronPython (and probably PyPy) have almost identical changes to tests that
> currently rely on deterministic finalisation (reference counting) so it
> makes sense to test changes on both platforms and commit a single solution.

IMHO, any place in the test suite that relies on deterministic
finalization but isn't explicitly testing that CPython-specific
feature is a bug and should be fixed, even before we export it to the
new repository.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Evaluated cmake as an autoconf replacement

[Jeffrey Yasskin]

On Sun, Mar 29, 2009 at 1:14 PM, David Cournapeau  wrote:
> About cmake: I haven't looked at it recently, but I have a bit of hard
> time believing python requires more from a build system than KDE. The
> lack of autoheader is not accurate, if
> only because kde projects have it:
>
> http://www.cmake.org/Wiki/CMake_HowToDoPlatformChecks

That page says, "... So we write a source file named config.h.in...".
The purpose of autoheader is to write that file automatically. I might
have missed something, but you'll have to provide a more precise link.

The problems I found were enough to convince me that it wasn't worth
continuing to a full translation of configure.in and Makefile.pre.in.
If you or someone else wants to do that translation, I'd be happy to
look at the result in case it turns out not to be as inconvenient as I
currently expect.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Evaluated cmake as an autoconf replacement

[Jeffrey Yasskin]

I've heard some good things about cmake — LLVM, googletest, and Boost
are all looking at switching to it — so I wanted to see if we could
simplify our autoconf+makefile system by using it. The biggest wins I
see from going to cmake are:
 1. It can autogenerate the Visual Studio project files instead of
needing them to be maintained separately
 2. It lets you write functions and modules without understanding
autoconf's mix of shell and M4.
 3. Its generated Makefiles track header dependencies accurately so we
might be able to add private headers efficiently.

However, after trying it out during PyCon and looking over the
previous attempt at
, I
can't recommend switching to it.

 A. It has no equivalent of autoheader, so we'd have to maintain
pyconfig.h.in by hand.
 B. It does not allow the CMakeLists.txt file control the --help
output. This appears to be an intentional decision
(http://www.cmake.org/pipermail/cmake-promote/2006-May/95.html).
To replace it, they have an interactive mode (which asks you about
each possible option) and a -LH flag (which runs the whole configure
process and then tells you about the flags you could have set if you
knew about them).
 C. It does not allow the CMakeLists.txt file to see the command line,
so we can't stay backward compatible with the configure script, and
we'd have to replace flags like --with-pydebug with
-Dpydebug:bool=true. We could write a separate configure script to
mitigate this and the previous problem, but we'd have to keep that in
sync with CMakeLists.txt manually.
 D. It doesn't have an expression language, so to replace
ac_md_system=`echo $ac_sys_system |
   tr -d '[/ ]' | tr '[[A-Z]]' '[[a-z]]'`
  you have to write:
string(REGEX REPLACE "[/ ]" "" ac_md_system ${ac_sys_system})
string(TOLOWER ${ac_md_system} ac_md_system)

So, at the very least, it doesn't look like a big enough win to
justify switching, and may not be a win at all.


The other popular configure+make replacement is scons. The biggest
objection to scons before even looking at it is that it requires a
working Python interpreter in order to build Python, causing a
bootstrap problem. However, Brett Cannon and I talked, and we think
this is surmountable. First, nearly every desktop system comes with a
Python interpreter, which would avoid the bootstrap for ordinary
development. Second, we need a cross compilation story anyway for
embedded systems, and the same mechanism could be used to get Python
onto a new desktop platform. Third, Jython and IronPython are pretty
mature and either can run scons now or should be able to run it after
some relatively easy tweaks. They could be used to build CPython on a
new platform. I don't intend to look into scons myself any time soon,
but I'd be interested in anyone's experiences who does try it.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ABCs and MRO

[Jeffrey Yasskin]

On Mon, Mar 2, 2009 at 1:14 PM, Paul Moore  wrote:
> 2009/3/2 Jeffrey Yasskin :
>> I tend to think it's a bug in ABCs. You seem to have thought of
>> several possible ways to fix it, and I don't have strong preferences
>> between them.
>
> I've discussed ways of fixing simplegeneric, but not of fixing the
> issue with ABCs. I'm not sure the ABC "issue" is fixable - after all,
> it's by design that ABCs can implement __issubclass__ and "magically"
> become superclasses of arbitrary classes as a result.
>
> I'm not happy about fixing simplegeneric, though, as the whole point
> was just to expose an existing implementation, because it might be
> generally useful. If we start expanding and enhancing it, there are
> better implementations already available on PyPI (better in the sense
> of having seen real-world use). And if they don't handle ABCs, then
> that might indicate that needing to handle ABCs isn't as vital as this
> discussion would seem to imply (but I have no real-world data myself
> to make such a claim).
>
>> One way to deal with __subclasscheck__ definitions other than
>> ABCMeta.__subclasscheck__ would be to make the "someone registered a
>> subclass" interface public in addition to the "tell me when someone
>> registers a subclass" interface. Then other __subclasscheck__
>> definitions could call the "someone registered a subclass" method when
>> the class is defined if they want to be usable with generic functions.
>
> I'm not sure I follow this. Can you clarify? My intuition is that *no*
> notification hook can deal with ABCs that code arbitrary checks in
> their __subclasscheck__ method (so there's no registration involved,
> particular classes just "magically" become subclasses).

Hm, oops, you're right. So you'd have to scan the relevant abcs and
cache the list that turns up true ... probably with cooperation from
the ABCs so you'd know when to invalidate the cache. Ick.
Unfortunately, I think overloading functions on Number or Iterable
would be really useful, and constraining it to only look at base
classes would be unfortunate.

>> You can get something similar to the mro by checking whether ABCs are
>> subclasses of each other.
>
> The key point about the MRO is that you get it just from the initial
> class, via __mro__. You're not checking if another class is *in* the
> MRO, but rather you're *generating* the list of classes in the MRO.

I was trying to reply to PJE's statement that "The hairier issue for
these types of systems is method precedence". I agree that getting the
set of abcs for an instance is more difficult that I'd thought.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ABCs and MRO

[Jeffrey Yasskin]

On Mon, Mar 2, 2009 at 2:41 AM, Paul Moore  wrote:
> ...
> More generally, there is NO WAY to determine the list of classes for
> which issubclass(C, x) is true.
>
> This could be considered a limitation of, or a bug in, ABCs, I don't
> have a particular opinion on that, but it does mean that no code which
> relies on being able to traverse the class inheritance graph will see
> ABCs. One particular case of this is (any implementation I can think
> of, of) generic functions.
>
> In my view, this implies one of the following:
>
> 1) It should be a documented limitation of such code that it doesn't
> work with ABCs (and conversely, this limitation of ABCs should be
> documented in the ABC documentation)
> 2) Generic functions, and any other code requiring this type of
> introspection, is essentially useless unless it can support ABCs, and
> should not be used in the light of this limitation.
> 3) This is a bug in ABCs and should be fixed.
> 4) Something else I didn't think of :-)

I tend to think it's a bug in ABCs. You seem to have thought of
several possible ways to fix it, and I don't have strong preferences
between them.

One way to deal with __subclasscheck__ definitions other than
ABCMeta.__subclasscheck__ would be to make the "someone registered a
subclass" interface public in addition to the "tell me when someone
registers a subclass" interface. Then other __subclasscheck__
definitions could call the "someone registered a subclass" method when
the class is defined if they want to be usable with generic functions.

You can get something similar to the mro by checking whether ABCs are
subclasses of each other.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 374 (DVCS) now in reST

[Jeffrey Yasskin]

On Mon, Jan 26, 2009 at 2:00 PM, Guido van Rossum  wrote:
> On Mon, Jan 26, 2009 at 1:57 PM, Giovanni Bajo  wrote:
>> On Mon, 26 Jan 2009 10:31:55 -0800, Guido van Rossum wrote:
>>
>>> On Mon, Jan 26, 2009 at 8:08 AM, Paul Hummer 
>>> wrote:
 At a previous employer, we had this same discussion about switching to
 a DVCS, and the time and cost required to learn the new tool.  We
 switched to bzr, and while there were days where someone got lost in
 the DVCS, the overall advantages with merging allowed that cost to be
 offset by the fact that merging was so cheap (and we merged a lot).

 That's a big consideration to be made when you're considering a DVCS.
 Merges in SVN and CVS can be painful, where merging well is a core
 feature of any DVCS.
>>>
>>> I hear you. I for one have been frustrated (now that you mention it) by
>>> the inability to track changes across merges. We do lots of merges from
>>> the trunk into the py3k branch, and the merge revisions in the branch
>>> quotes the full text of the changes merged from the trunk, but not the
>>> list of affected files for each revision merged. Since merges typically
>>> combine a lot of revisions, it is very painful to find out more about a
>>> particular change to a file when that change came from such a merge --
>>> often even after reading through the entire list of descriptions you
>>> still have no idea which merged revision is responsible for a particular
>>> change. Assuming this problem does not exist in DVCS, that would be a
>>> huge bonus from switching to a DVCS!
>>
>> Well, not only it does not exist by design in any DVCS, but I have a
>> better news: it does not exist anymore in Subversion 1.5. You just need
>> to upgrade your SVN server to 1.5, migrate your merge history from the
>> format of svnmerge to the new builtin format (using the official script),
>> and you're done: say hello to "-g/--use-merge-history", to be use with
>> svn log and svn blame.
>>
>> This is a good writeup of the new features:
>> http://chestofbooks.com/computers/revision-control/subversion-svn/Merge-
>> Sensitive-Logs-And-Annotations-Branchmerge-Advanced-Lo.html
>
> Unfortunately I've heard we shouldn't upgrade to svn 1.5 until more
> Linux distributions ship with it by default.

Besides that, `svn merge` cannot handle parallel branches like
trunk/py3k without lots of handholding. Unlike svnmerge.py, when you
merge to and then from a branch, it tries to merge changes that came
from trunk and produces lots of conflicts. (Before you point me at
--reintegrate, note "In Subversion 1.5, once a --reintegrate merge is
done from branch to trunk, the branch is no longer usable for further
work." from the book.) In principle, the svn devs could fix this, but
they didn't in svn 1.5.

To keep this slighly on topic ... maybe the abilities and limits of
svnmerge.py and `svn merge` should be mentioned in the PEP?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reindenting the C code base?

[Jeffrey Yasskin]

On Sun, Dec 14, 2008 at 8:26 AM, Guido van Rossum  wrote:
> On Sat, Dec 13, 2008 at 2:11 PM, Antoine Pitrou  wrote:
>> Guido van Rossum  python.org> writes:
>>>
>>> I think we should not do this. We should use 4 space indents for new
>>> files, but existing files should not be reindented.
>>
>> Well, right now many files are indented with a mix of spaces and tabs, 
>> depending
>> on who did the edit and how their editor was configured at the time.
>
> That's  a shame. We used to have more rigorous standards than allowing that.
>
>> Perhaps a graceful policy would be to mandate that all new edits be made with
>> spaces without touching other functions in the file. Then hopefully the code
>> base would gradually converge to a tabless scheme.
>
> I don't think so. I find local consistency more important than global
> consistency. A file can become really hard to read when different
> indentation schemes are used in random parts of the code.
>
> If you have a problem configuring your editor, just say so and someone
> will explain how to do it.

I've never figured out how to configure emacs to deduce whether the
current file uses spaces or tabs and has a 4 or 8 space indent. I
always try to get it right anyway, but it'd be a lot more convenient
if my editor did it for me. If there are such instructions, perhaps
they should be added to PEPs 7 and 8?

Thanks,
Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Merging flow

[Jeffrey Yasskin]

On Thu, Dec 11, 2008 at 12:10 PM, "Martin v. Löwis"  wrote:
>> Yeah, that's why I asked. I tried what Martin suggested with r67698 by
>> just saying I'd resolved the conflict, which added the single revision
>> I was merging from to the svnmerge-integrated property. It didn't add
>> the two original revisions.
>
> Can you elaborate? What are the "two original revisions" it didn't add?
>
> If you are referring to the trunk revisions - that's fine. As far
> as svnmerge is concerned, we merge revisions from the 3k branch
> to the 3.0 maintenance branch. The original revisions don't exist
> on the 3k branch (they have an empty changeset), so it's not a
> problem that they didn't get recorded as merged.

Yes, I was referring to the trunk revisions. Sounds like this (marking
the conflicting property as resolved without changing it) is the way
to go then. Thanks!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Merging flow

[Jeffrey Yasskin]

On Thu, Dec 11, 2008 at 4:18 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote:
> Martin v. Löwis wrote:
>> Jeffrey Yasskin wrote:
>>> Was there ever a conclusion to this? I need to merge the patches
>>> associated with issue 4597 from trunk to all the maintenance branches,
>>> and I'd like to avoid messing anyone up if possible. If I don't hear
>>> back, I'll plan to svnmerge directly from trunk to each of the
>>> branches, and then block my merge to py3k from being merged again to
>>> release30-maint.
>>
>> No - you should merge from the py3k branch to the release30-maint branch.
>
> I believe that's difficult when you previously merged from the trunk to
> the py3k branch - the merged change to the svnmerge related properties
> on the root directory gets in the way when svnmerge attempts to update
> them on the maintenance branch.
>
> That's what started this thread, and so far nobody has come up with a
> workaround. It seems to me that svnmerge.py should just be able to do a
> svn revert on the affected properties in the maintenance branch before
> it attempts to modify them, but my svn-fu isn't strong enough for me to
> say that for sure.

Yeah, that's why I asked. I tried what Martin suggested with r67698 by
just saying I'd resolved the conflict, which added the single revision
I was merging from to the svnmerge-integrated property. It didn't add
the two original revisions. I don't know enough about how svnmerge
works to know if that's the right outcome or who it's going to cause
trouble for.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trap SIGSEGV and SIGFPE

[Jeffrey Yasskin]

On Thu, Dec 11, 2008 at 1:34 AM, Victor Stinner
<[EMAIL PROTECTED]> wrote:
> But if -as many people wrote-
> Python is totally broken after a segfault, it is maybe not a good idea :-)

While it's true that after a segfault or unexpected longjmp, there are
no guarantees whatsoever about the state of the python program, the
program will often just happen to work, and there are at least some
programs I've worked on that would rather take the risk in order to
try to shut down gracefully. For example, an interactive app may want
to give the user a chance to save her (not necessarily corrupted) work
into a new file rather than unconditionally losing it. Or a webserver
might want to catch the segfault, finish replying to the other
requests that were in progress at the time, maybe reply to the request
that caused the segfault, and then restart. Yes there's a possibility
that the events around the segfault exposed some secret internal data
(and they may do so even without segfaulting), but when the
alternative is not replying to the users at all, this may be a risk
the app wants to take. It would be nice for Python to at least expose
the option so that developers (who are consenting adults, remember)
can make their own decisions. It should _not_ be on by default, but
something like sys.dangerous_turn_C_crashes_into_exceptions() would be
useful.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Merging flow

[Jeffrey Yasskin]

Was there ever a conclusion to this? I need to merge the patches
associated with issue 4597 from trunk to all the maintenance branches,
and I'd like to avoid messing anyone up if possible. If I don't hear
back, I'll plan to svnmerge directly from trunk to each of the
branches, and then block my merge to py3k from being merged again to
release30-maint.

Thanks,
Jeffrey

On Thu, Dec 4, 2008 at 7:12 AM, Christian Heimes <[EMAIL PROTECTED]> wrote:
> Several people have asked about the patch and merge flow. Now that Python
> 3.0 is out it's a bit more complicated.
>
> Flow diagram
> 
>
> trunk ---> release26-maint
>   \->  py3k   ---> release30-maint
>
>
> Patches for all versions of Python should land in the trunk. They are then
> merged into release26-maint and py3k branches. Changes for Python 3.0 are
> merged via the py3k branch.
>
> Christian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Patch to speed up non-tracing case in PyEval_EvalFrameEx (2% on pybench)

[Jeffrey Yasskin]

Done: http://bugs.python.org/issue4477

On Sun, Nov 30, 2008 at 8:14 PM, Brett Cannon <[EMAIL PROTECTED]> wrote:
> Can you toss the patch into the issue tracker, Jeffrey, so that any
> patch comments can be done there?
>
> -Brett
>
> On Sun, Nov 30, 2008 at 17:54, Jeffrey Yasskin <[EMAIL PROTECTED]> wrote:
>> Tracing support shows up fairly heavily an a Python profile, even
>> though it's nearly always turned off. The attached patch against the
>> trunk speeds up PyBench by 2% for me. All tests pass. I have 2
>> questions:
>>
>> 1) Can other people corroborate this speedup on their machines? I'm
>> running on a Macbook Pro (Intel Core2 processor, probably Merom) with
>> a 32-bit build from Apple's gcc-4.0.1. (Apple's gcc consistently
>> produces a faster python than gcc-4.3.)
>>
>> 2) Assuming this speeds things up for most people, should I check it
>> in anywhere besides the trunk? I assume it's out for 3.0; is it in for
>> 2.6.1 or 3.0.1?
>>
>>
>>
>> Pybench output:
>>
>> ---
>> PYBENCH 2.0
>> ---
>> * using CPython 2.7a0 (trunk:67458M, Nov 30 2008, 17:14:10) [GCC 4.0.1
>> (Apple Inc. build 5488)]
>> * disabled garbage collection
>> * system check interval set to maximum: 2147483647
>> * using timer: time.time
>>
>> ---
>> Benchmark: pybench.out
>> ---
>>
>>Rounds: 10
>>Warp:   10
>>Timer:  time.time
>>
>>Machine Details:
>>   Platform ID:Darwin-9.5.0-i386-32bit
>>   Processor:  i386
>>
>>Python:
>>   Implementation: CPython
>>   Executable:
>> /Users/jyasskin/src/python/trunk-fast-tracing/build/python.exe
>>   Version:2.7.0
>>   Compiler:   GCC 4.0.1 (Apple Inc. build 5488)
>>   Bits:   32bit
>>   Build:  Nov 30 2008 17:14:10 (#trunk:67458M)
>>   Unicode:UCS2
>>
>>
>> ---
>> Comparing with: ../build_orig/pybench.out
>> ---
>>
>>Rounds: 10
>>Warp:   10
>>Timer:  time.time
>>
>>Machine Details:
>>   Platform ID:Darwin-9.5.0-i386-32bit
>>   Processor:  i386
>>
>>Python:
>>   Implementation: CPython
>>   Executable:
>> /Users/jyasskin/src/python/trunk-fast-tracing/build_orig/python.exe
>>   Version:2.7.0
>>   Compiler:   GCC 4.0.1 (Apple Inc. build 5488)
>>   Bits:   32bit
>>   Build:  Nov 30 2008 13:51:09 (#trunk:67458)
>>   Unicode:UCS2
>>
>>
>> Test minimum run-timeaverage  run-time
>> thisother   diffthisother   diff
>> ---
>>  BuiltinFunctionCalls:   127ms   130ms   -2.4%   129ms   132ms   
>> -2.1%
>>   BuiltinMethodLookup:90ms93ms   -3.2%91ms94ms   
>> -3.1%
>> CompareFloats:88ms91ms   -3.3%89ms93ms   
>> -4.3%
>> CompareFloatsIntegers:97ms99ms   -2.1%97ms   100ms   
>> -2.4%
>>   CompareIntegers:79ms82ms   -4.2%79ms85ms   
>> -6.1%
>>CompareInternedStrings:90ms92ms   -2.4%94ms94ms   
>> -0.9%
>>  CompareLongs:86ms83ms   +3.6%87ms84ms   
>> +3.5%
>>CompareStrings:80ms82ms   -3.1%81ms83ms   
>> -2.3%
>>CompareUnicode:   103ms   105ms   -2.3%   106ms   108ms   
>> -1.5%
>>ComplexPythonFunctionCalls:   139ms   137ms   +1.3%   140ms   139ms   
>> +0.1%
>> ConcatStrings:   142ms   151ms   -6.0%   156ms   154ms   
>> +1.1%
>> ConcatUnicode:87ms92ms   -5.4%89ms94ms   
>> -5.7%
>>   CreateInstances:   142ms   144ms   -1.4%   144ms   145ms   
>> -1.1%
>>CreateNewInstances:   107ms   109ms   -2.3%   108ms   111ms   
>> -2.1%
>>

[Python-Dev] Patch to speed up non-tracing case in PyEval_EvalFrameEx (2% on pybench)

[Jeffrey Yasskin]

Tracing support shows up fairly heavily an a Python profile, even
though it's nearly always turned off. The attached patch against the
trunk speeds up PyBench by 2% for me. All tests pass. I have 2
questions:

1) Can other people corroborate this speedup on their machines? I'm
running on a Macbook Pro (Intel Core2 processor, probably Merom) with
a 32-bit build from Apple's gcc-4.0.1. (Apple's gcc consistently
produces a faster python than gcc-4.3.)

2) Assuming this speeds things up for most people, should I check it
in anywhere besides the trunk? I assume it's out for 3.0; is it in for
2.6.1 or 3.0.1?



Pybench output:

---
PYBENCH 2.0
---
* using CPython 2.7a0 (trunk:67458M, Nov 30 2008, 17:14:10) [GCC 4.0.1
(Apple Inc. build 5488)]
* disabled garbage collection
* system check interval set to maximum: 2147483647
* using timer: time.time

---
Benchmark: pybench.out
---

Rounds: 10
Warp:   10
Timer:  time.time

Machine Details:
   Platform ID:Darwin-9.5.0-i386-32bit
   Processor:  i386

Python:
   Implementation: CPython
   Executable:
/Users/jyasskin/src/python/trunk-fast-tracing/build/python.exe
   Version:2.7.0
   Compiler:   GCC 4.0.1 (Apple Inc. build 5488)
   Bits:   32bit
   Build:  Nov 30 2008 17:14:10 (#trunk:67458M)
   Unicode:UCS2


---
Comparing with: ../build_orig/pybench.out
---

Rounds: 10
Warp:   10
Timer:  time.time

Machine Details:
   Platform ID:Darwin-9.5.0-i386-32bit
   Processor:  i386

Python:
   Implementation: CPython
   Executable:
/Users/jyasskin/src/python/trunk-fast-tracing/build_orig/python.exe
   Version:2.7.0
   Compiler:   GCC 4.0.1 (Apple Inc. build 5488)
   Bits:   32bit
   Build:  Nov 30 2008 13:51:09 (#trunk:67458)
   Unicode:UCS2


Test minimum run-timeaverage  run-time
 thisother   diffthisother   diff
---
  BuiltinFunctionCalls:   127ms   130ms   -2.4%   129ms   132ms   -2.1%
   BuiltinMethodLookup:90ms93ms   -3.2%91ms94ms   -3.1%
 CompareFloats:88ms91ms   -3.3%89ms93ms   -4.3%
 CompareFloatsIntegers:97ms99ms   -2.1%97ms   100ms   -2.4%
   CompareIntegers:79ms82ms   -4.2%79ms85ms   -6.1%
CompareInternedStrings:90ms92ms   -2.4%94ms94ms   -0.9%
  CompareLongs:86ms83ms   +3.6%87ms84ms   +3.5%
CompareStrings:80ms82ms   -3.1%81ms83ms   -2.3%
CompareUnicode:   103ms   105ms   -2.3%   106ms   108ms   -1.5%
ComplexPythonFunctionCalls:   139ms   137ms   +1.3%   140ms   139ms   +0.1%
 ConcatStrings:   142ms   151ms   -6.0%   156ms   154ms   +1.1%
 ConcatUnicode:87ms92ms   -5.4%89ms94ms   -5.7%
   CreateInstances:   142ms   144ms   -1.4%   144ms   145ms   -1.1%
CreateNewInstances:   107ms   109ms   -2.3%   108ms   111ms   -2.1%
   CreateStringsWithConcat:   114ms   137ms  -17.1%   117ms   139ms  -16.0%
   CreateUnicodeWithConcat:92ms   101ms   -9.2%95ms   102ms   -7.2%
  DictCreation:77ms81ms   -4.4%80ms85ms   -5.9%
 DictWithFloatKeys:91ms   107ms  -14.5%93ms   109ms  -14.6%
   DictWithIntegerKeys:95ms94ms   +1.4%   108ms96ms  +12.3%
DictWithStringKeys:83ms88ms   -5.8%84ms88ms   -4.7%
  ForLoops:72ms72ms   -0.1%79ms74ms   +5.8%
IfThenElse:83ms80ms   +3.9%85ms80ms   +5.3%
   ListSlicing:   117ms   118ms   -0.7%   118ms   121ms   -1.8%
NestedForLoops:   116ms   119ms   -2.4%   121ms   121ms   +0.0%
  NormalClassAttribute:   106ms   115ms   -7.7%   108ms   117ms   -7.7%
   NormalInstanceAttribute:96ms98ms   -2.3%97ms   100ms   -3.1%
   PythonFunctionCalls:92ms95ms   -3.7%94ms99ms   -5.2%
 PythonMethodCalls:   147ms   147ms   +0.1%   152ms   149ms   +2.1%
 Recursion:   135ms   136ms   -0.3%   140ms   144ms   -2.9%
  SecondImport:   101ms99ms   +2.1%   103ms   101ms   +2.2%
  

Re: [Python-Dev] [ANN] VPython 0.1

[Jeffrey Yasskin]

t;   UnicodeMappings:   222ms   244ms   -8.9%   241ms   257ms   -6.3%
> UnicodePredicates:   170ms   214ms  -20.6%   179ms   227ms  -21.2%
> UnicodeProperties:   136ms   159ms  -14.9%   154ms   206ms  -25.3%
>UnicodeSlicing:   142ms   215ms  -34.1%   171ms   248ms  -31.3%
>   WithFinally:   208ms   260ms  -20.1%   212ms   271ms  -21.9%
>   WithRaiseExcept:   175ms   193ms   -9.0%   186ms   209ms  -11.0%
> -----------
> Totals:  7682ms 10935ms  -29.8%  8468ms 12832ms  
> -34.0%
>
> (this=2008-10-22 20:45:22, other=/tmp/vanilla252.pybench)
>
> --
> David Ripton[EMAIL PROTECTED]
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com
>



-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mini-Pep: Simplifying the Integral ABC

[Jeffrey Yasskin]

Well, it seems like Integral instances should be able to be passed to
either int() or long(), so __long__ should probably stay. I have no
idea why I didn't include __int__, but its absence was probably the
only reason __index__ calls long() instead of int().

On Fri, Jun 6, 2008 at 3:23 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote:
> Both of these seem 2.6-specific quirks. Those lines wereJeffrey's;
> maybe he remembers? I'm guessing that adding __long__ was done since
> 2.6 supports it, and the removal of __int__ was an oversight.  I also
> think that there's no reason to change __index__ to call long(); int()
> will automatically return a long as needed. Maybe changing __long__
> back to __int__ is also harmless.
>
> On Fri, Jun 6, 2008 at 2:13 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
>> From: "Guido van Rossum" <[EMAIL PROTECTED]>
>>>
>>> Make that int() instead of long() and I'm okay with it.
>>
>> Does anyone know why Integral says that __long__ is a required abstract
>> method, but not __int__?
>>
>> Likewise, why is index() defined as long(self) instead of int(self)?
>>
>> There may be some design nuance that I'm not seeing.
>>
>>
>> Raymond
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
>



-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mini-Pep: Simplifying the Integral ABC

[Jeffrey Yasskin]

Because .numerator and .denominator are defined by the Rational ABC,
and Integral inherits from that, removing them from Integral is a
larger change than removing the other methods. You'd have to remove
them from Rational or break the inheritance relation.

Removing the bitwise operators sounds fine to me.

On Wed, Jun 4, 2008 at 1:54 AM, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> The only comment so far was to keep the __index__ method.
>
> Other than that, is this good to go?
>
> Raymond
>
>
> - Original Message -
>>
>> Target:  Py2.6 and Py3.0
>> Author: Raymond Hettinger
>> Date: May 31, 2008
>>
>> Motivation
>> --
>> The principal purpose of an abstract base class is to support multiple
>> implementations of an API; thereby allowing one concrete class to be
>> substitutable for another. This purpose is defeated when useful
>> substitutions
>> are precluded because the ABC is too aggressive in its behavioral
>> requirements
>> -- mandating too many methods, mandating methods that are difficult to
>> implement, or too closely following the full API of a single concrete
>> type.
>>
>> The Integral ABC is somewhat extensive and requires essentially every
>> behavior
>> exhibited by the int concrete class.  Usefully, it provides for basic
>> integer
>> behaviors like simple arithmetic and ordering relations.  However, the
>> current
>> ABC goes beyond that and requires bit-flipping and other operations that
>> are
>> not fundamental to the notion of being an integer.  That makes it
>> challenging
>> to define a new Integral class that isn't already an int.
>>
>> Proposal
>> 
>> Remove non-essential abstract methods like __index__, three argument
>> __pow__,
>> __lshift__, __rlshift__, __rshift__, __rrshift__, __and__, __rand__,
>> __xor__,
>> __rxor__, __or__, __ror__, and __invert__, numerator, and denominator.
>>
>> Discussion
>> --
>> The only known use cases for variants of int are types that limit the
>> range of
>> values to those that fit in a fixed storage width.
>>
>> One existing implementation is in numpy which has types like int0, int8,
>> int16, int32, and int16.  The numpy integral types implement all the
>> methods
>> present in Integral except for the new methods like __trunc__, __index__,
>> real, imag, conjugate, numerator, and denominator.  For the most part,
>> they
>> are fully substitutable into code that expects regular ints; however, they
>> do
>> have wrap-around behaviors such as int8(200)+int8(100) --> int8(44). The
>> wrap-around behaviors make the numpy types totally unsuitable for some
>> applications of Integral such as the fractions module which accepts any
>> integral numerator and denominator.
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/python%40rcn.com
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com
>



-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] test_signal on osx g4

[Jeffrey Yasskin]

I've tried to fix test_signal at least. For those particular calls,
EINTR means that the kill I launched finished before I was expecting,
so we can ignore it without retrying. Figuring out subprocess in
general is a worthy goal but shouldn't block making the test less
flaky. :) r62102.

On Tue, Apr 1, 2008 at 8:05 AM,  <[EMAIL PROTECTED]> wrote:
> test_signal is failing on osx g4:
>
> test test_signal failed -- Traceback (most recent call last):
>   File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", 
> line 151, in test_main
> self.fail(tb)
> AssertionError: Traceback (most recent call last):
>   File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", 
> line 134, in test_main
> self.run_test()
>   File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", 
> line 80, in run_test
> child = subprocess.Popen(['kill', '-HUP', str(pid)])
>   File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", line 
> 593, in __init__
> errread, errwrite)
>   File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", line 
> 1078, in _execute_child
> data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
> OSError: [Errno 4] Interrupted system call
>
>  This is the code which reads stderr from the child process:
>
> data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
> os.close(errpipe_read)
> if data != "":
> os.waitpid(self.pid, 0)
> child_exception = pickle.loads(data)
> raise child_exception
>
>  I'm not an expert in this stuff my any stretch of the imagination, but
>  shouldn't subprocess try harder to read this output?  Something like:
>
> while True:
> try:
> data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
> except OSError, err:
> if err.errno == errno.EINTR:
> continue
> else:
> raise
> else:
> os.close(errpipe_read)
> if data != "":
> os.waitpid(self.pid, 0)
> child_exception = pickle.loads(data)
> raise child_exception
> break
>
>  Maybe not while True, but try a few times at least.
>
>  Or is the system supposed to automatically restart interrupted system calls?
>
>  Skip
>  ___
>  Python-Dev mailing list
>  Python-Dev@python.org
>  http://mail.python.org/mailman/listinfo/python-dev
>  Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com
>



-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] the release gods are angry at python

[Jeffrey Yasskin]

On Fri, Mar 28, 2008 at 11:15 AM, Bill Janssen <[EMAIL PROTECTED]> wrote:
> > On Fri, Mar 28, 2008 at 3:31 AM,  <[EMAIL PROTECTED]> wrote:
>  > >
>  > > Neal> Anything that connects to a remote host is definitely flaky.
>  > >
>  > >  Would it maybe help to set up a dedicated host (or virtual host) to 
> serve as
>  > >  the sole target of all network tests?
>  >
>  > It would help, but not fix the problem.  There will still be transient
>  > network issues that come up.
>
>  We really need to be able to wrap a timeout around any individual
>  test, perhaps throwing a TimeoutException if the timeout is reached,
>  and have the ability to regard a TimeoutException as not being a
>  "failure".

Something like:

@contextmanager
def timeout(max_time):
def RaiseTimeout(signal, frame):
raise TimeoutException
# Why don't we have a context manager for the next line?
old_alarm = signal.signal(signal.SIGALRM, RaiseTimeout)
try:
signal.alarm(max_time)
try:
yield
finally:
signal.alarm(0)
finally:
signal.signal(signal.SIGALRM, old_alarm)

with appropriate fallbacks for Windows. I may not be poking at
test_support for a bit, so anyone else is welcome to check that in, or
a fixed version of it, whenever they're fixing a timing-out test.

Someone else will have to implement the support in regrtest, although
I wouldn't mind having it treat a timeout as a failure, as long as
it's easier to diagnose than the SIGKILL it currently uses for
timeouts.

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest's redundant assertions: asserts vs. failIf/Unlesses

[Jeffrey Yasskin]

On Wed, Mar 19, 2008 at 2:15 PM,  <[EMAIL PROTECTED]> wrote:
>
>  On 02:21 pm, [EMAIL PROTECTED] wrote:
>  >>OTOH, I'd rather there be OOWTDI so whatever the consensus is is fine
>  >>with me.
>  >
>  >This strikes me as a gratuitous API change of the kind Guido was
>  >warning about in his recent post: "Don't change your APIs incompatibly
>  >when porting to Py3k"
>
>  I agree emphatically.  Actually I think this is the most extreme case.
>  The unit test stuff should be as stable as humanly possible between 2
>  and 3, moreso than any other library.

This is convincing for me. Move my +1 back to 3.1.

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest's redundant assertions: asserts vs. failIf/Unlesses

[Jeffrey Yasskin]

+1 to assert* from me. the fail* variants always feel like
double-negatives. I also always use assertTrue instead of assert_. But
I don't care enough to argue about it. :)

On Wed, Mar 19, 2008 at 2:24 AM, Gabriel Grant <[EMAIL PROTECTED]> wrote:
> Hi all,
>
>  This gem from unittest.py is pretty much the opposite of "one obvious way":
>
> # Synonyms for assertion methods
>
> assertEqual = assertEquals = failUnlessEqual
>
> assertNotEqual = assertNotEquals = failIfEqual
>
> assertAlmostEqual = assertAlmostEquals = failUnlessAlmostEqual
>
> assertNotAlmostEqual = assertNotAlmostEquals = failIfAlmostEqual
>
> assertRaises = failUnlessRaises
>
> assert_ = assertTrue = failUnless
>
> assertFalse = failIf
>
>  Could these be removed for 3k?
>
>  There was a short discussion about this among some of those those
>  present in the Python Core sprint room at PyCon today and most
>  preferred the "assertEqual" form for [Not][Almost]Equal and Raises.
>
>  With assertFalse vs. failIf (and assertTrue vs. failUnless) there was
>  far less agreement. JUnit uses assertTrue exclusively, and most people
>  said they feel that using assertTrue would be more consistent, but
>  many (myself included) still think failUnless and failIf are much more
>  natural. Another issue with assertTrue is that it doesn't actually
>  test for 'True', strictly speaking, since it is based on equality, not
>  identity.
>
>  Its also interesting to note the original commit message:
>
>  > r34209 | purcell | 2003-09-22 06:08:12 -0500 (Mon, 22 Sep 2003)
>  > [...]
>  > - New assertTrue and assertFalse aliases for comfort of JUnit users
>  > [...]
>
>  assertEqual (and its cousins) were already present at that point.
>
>  In any case, if the decision is made to not use failUnless, something
>  still needs to be done with assert_ vs. assertTrue. assert_ seems
>  somewhat better to me, in that it has fewer characters, but I think
>  that a case could certainly be made to keep both of these.
>
>  I certainly don't have the authority to make a call on any of this,
>  but if someone else decides what colour to paint this bike shed, I can
>  try to get it done (hopefully with 2.6 warnings) tomorrow.
>
>  Cheers,
>
>  -Gabriel
>
>  P.S. If you were in the sprint room and feel terribly misrepresented,
>  please feel free to give me a swift kick both on-list and in person
>  tomorrow morning.
>  ___________
>  Python-Dev mailing list
>  Python-Dev@python.org
>  http://mail.python.org/mailman/listinfo/python-dev
>  Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com
>



-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Optimizing with.

[Jeffrey Yasskin]

I've got a patch in http://bugs.python.org/issue2179 that optimizes
the bytecode generated by a with statement by tucking the
context_manager.__exit__ method onto the stack. It saves 2 opcodes, 8
bytes, and about .5us for each with block at the cost of an extra
stack entry for the duration of the block. Since it's the first time
I've touched the core of the compiler and interpreter, I'm hoping that
someone can take a look before I check it in.

Thanks!
Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] signal.alarm(3) in trunk test_socketserver.py

[Jeffrey Yasskin]

On Tue, Mar 4, 2008 at 7:25 PM, Trent Nelson <[EMAIL PROTECTED]> wrote:
> > r61099 added the following to trunk/Lib/test/test_socketserver.py:
>  >
>  >   if __name__ == "__main__":
>  >   test_main()
>  > + signal.alarm(3)  # Shutdown shouldn't take more than 3 seconds.
>  >
>
>  Actually, signal.alarm() was introduced all over the place in that revision. 
>  I understand the intent of this commit was to speed up the runtime of this 
> test (something like 28s -> 0.3s was quoted in the commit log).  FWIW, 
> runtime of the test with the following patch on Windows is 0.125s:
>

Yep, the alarm is only there to prevent what would be deadlocks from
running forever. Sorry for breaking !unix. Your patch looks fine to
me. Do you want to submit it or shall I?

>  Index: test_socketserver.py
>  ===
>  --- test_socketserver.py(revision 61233)
>  +++ test_socketserver.py(working copy)
>  @@ -28,6 +28,9 @@
>   HAVE_UNIX_SOCKETS = hasattr(socket, "AF_UNIX")
>   HAVE_FORKING = hasattr(os, "fork") and os.name != "os2"
>
>  +def signal_alarm(n):
>  +if hasattr(signal, 'alarm'):
>  +signal.alarm(n)
>
>   def receive(sock, n, timeout=20):
>  r, w, x = select.select([sock], [], [], timeout)
>  @@ -99,7 +102,7 @@
>  """Test all socket servers."""
>
>  def setUp(self):
>  -signal.alarm(20)  # Kill deadlocks after 20 seconds.
>  +signal_alarm(20)  # Kill deadlocks after 20 seconds.
>  self.port_seed = 0
>  self.test_files = []
>
>  @@ -112,7 +115,7 @@
>  except os.error:
>  pass
>  self.test_files[:] = []
>  -signal.alarm(0)  # Didn't deadlock.
>  +signal_alarm(0)  # Didn't deadlock.
>
>  def pickaddr(self, proto):
>  if proto == socket.AF_INET:
>  @@ -267,4 +270,4 @@
>
>
>   if __name__ == "__main__":
>  test_main()
>  -signal.alarm(3)  # Shutdown shouldn't take more than 3 seconds.
>  +signal_alarm(3)  # Shutdown shouldn't take more than 3 seconds.
>
>
> ___
>  Python-Dev mailing list
>  Python-Dev@python.org
>  http://mail.python.org/mailman/listinfo/python-dev
>  Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com
>



-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Py_CLEAR to avoid crashes

[Jeffrey Yasskin]

On Feb 17, 2008 12:29 PM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Jeffrey Yasskin wrote:
> > On Feb 16, 2008 3:12 PM, Amaury Forgeot d'Arc <[EMAIL PROTECTED]> wrote:
> >> Should we however intensively search and correct all of them?
> >> Is there a clever way to prevent these problems globally, for example
> >> by delaying finalizers "just a little"?
> >
> > A simple way to do this would be to push objects whose refcounts had
> > reached 0 onto a list instead of finalizing them immediately, and have
> > PyEval_EvalFrameEx periodically swap in a new to-delete list and
> > delete the objects on the old one.
>
> -1. This would only soften the problem in a shallow way. It should still
> be possible to create cases where the final cleanup of the object comes
> "too early", e.g. when some caller of PyEval_EvalFrameEx still carries
> a pointer to some object that got deleted, and then still some code can
> get hold of the then-deleted object. It will just be more difficult to
> trigger such crashes, depending on the period in which objects are
> cleaned.

Right. Changing introducing these bugs from "easy" to "possible"
sounds like an improvement. Making them harder to trigger has both
good and bad effects: they're harder to exploit and harder to
reproduce. If we delete on every iteration, it only costs a couple
memory accesses more, and should eliminate most of the
non-determinism.

Anyway, I saw an approach like this work well on a server I used to
work on, but there were differences. In particular, there was no way
to call a function to trigger deletions; you had to return out to the
main loop, which made it a little more reliable than it would be in
Python. I suspect it would still fix nearly all of the bugs Amaury is
finding since Py_DECREF would no longer return control to the
interpreter, but I could be wrong.

> The only sane way to never touch deleted objects is to have no
> references to them on heap anymore, and to not touch stack references
> after the DECREF.
>
> Regards,
> Martin
>



-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Py_CLEAR to avoid crashes

[Jeffrey Yasskin]

On Feb 16, 2008 3:12 PM, Amaury Forgeot d'Arc <[EMAIL PROTECTED]> wrote:
> Should we however intensively search and correct all of them?
> Is there a clever way to prevent these problems globally, for example
> by delaying finalizers "just a little"?

A simple way to do this would be to push objects whose refcounts had
reached 0 onto a list instead of finalizing them immediately, and have
PyEval_EvalFrameEx periodically swap in a new to-delete list and
delete the objects on the old one. A linked list would cost an extra
pointer in PyObject_HEAD, but a growable array would only cost
allocations, which would be amortized over the allocations of the
objects you're deleting, so that's probably the way to go. A
fixed-size queue that just delayed finalization by a constant number
of objects would usually work without any allocations, but there would
sometimes be single finalizers that recursively freed too many other
objects, which would defeat the delay.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use Python 3.0 syntax for except and raise?

[Jeffrey Yasskin]

Sounds good to me. Along those lines, shall we work on fixing warnings
that -3 raises in the regression test suite?

On Feb 17, 2008 8:57 AM, Christian Heimes <[EMAIL PROTECTED]> wrote:
> Good evening everybody!
>
> I like to run the 2to3 tool with raise and except fixers over the 2.6
> sources. The raise fixer changes "raise Exception, msg" to "raise
> Exception(msg)" and the except fixer replaces "except Exception, err" by
> "except Exception as err". In my humble opinion the Python stdlib should
> give proper examples how write good code.
>
> During the migration period from the 2.x series to 3.x we have two
> obvious ways to write code. Let's stick to the new and preferred way.
>
> Oh and please use the new syntax for patches, too. It makes my job with
> svnmerge a little bit easier.
>
> Thanks!
>
> Christian
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com
>



-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding __format__ to classic classes

[Jeffrey Yasskin]

Oops, sorry for the spam. I didn't see that there were already answers
in the rest of the thread. :-(

On Feb 13, 2008 9:25 PM, Jeffrey Yasskin <[EMAIL PROTECTED]> wrote:
> On Feb 13, 2008 1:42 PM, Eric Smith <[EMAIL PROTECTED]> wrote:
> > Guido van Rossum wrote:
> > >> Much to my surprise, this already works:
> > >>
> > >>  >>> format(oldstyle(), '+^50s')
> > >> '+<__main__.oldstyle instance at 0x3d91f8>+'
> > >>  >>>
> > >>
> > >> So I guess it's a moot point.  I'm using the same code as I use in 3.0,
> > >> where I call:
> > >>meth = _PyType_Lookup(Py_TYPE(value), format_str);
> > >> where format_str is "__format__" interned.  And I'm getting a value back
> > >> for old style classes.
> > >>
> > >> Eric.
> > >
> > > But now try overriding __format__().  The 3.0 code uses
> > > _PyType_Lookup() which is not traversing the class dicts for classic
> > > classes, so it won't find __format__ overrides.
> > >
> >
> > Okay, I see and understand that issue.  But looking at len or getattr, I
> > don't see how to generalize it to __format__.  __len__ and __getattr__
> > have special support in the classes, with cl_getattr, tp_getattr, etc.
> >
> > I hate to be dense, but could you point me to some code that does
> > something similar but looks up the method by name?
>
> I'm not sure if it'll be exactly analogous, but you might look at
> __trunc__ and math.trunc for inspiration.
>
> --
> Namasté,
> Jeffrey Yasskin
> http://jeffrey.yasskin.info/
>



-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding __format__ to classic classes

[Jeffrey Yasskin]

On Feb 13, 2008 1:42 PM, Eric Smith <[EMAIL PROTECTED]> wrote:
> Guido van Rossum wrote:
> >> Much to my surprise, this already works:
> >>
> >>  >>> format(oldstyle(), '+^50s')
> >> '+<__main__.oldstyle instance at 0x3d91f8>+'
> >>  >>>
> >>
> >> So I guess it's a moot point.  I'm using the same code as I use in 3.0,
> >> where I call:
> >>meth = _PyType_Lookup(Py_TYPE(value), format_str);
> >> where format_str is "__format__" interned.  And I'm getting a value back
> >> for old style classes.
> >>
> >> Eric.
> >
> > But now try overriding __format__().  The 3.0 code uses
> > _PyType_Lookup() which is not traversing the class dicts for classic
> > classes, so it won't find __format__ overrides.
> >
>
> Okay, I see and understand that issue.  But looking at len or getattr, I
> don't see how to generalize it to __format__.  __len__ and __getattr__
> have special support in the classes, with cl_getattr, tp_getattr, etc.
>
> I hate to be dense, but could you point me to some code that does
> something similar but looks up the method by name?

I'm not sure if it'll be exactly analogous, but you might look at
__trunc__ and math.trunc for inspiration.

-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python on non IEEE-754 platforms: plea for information.

[Jeffrey Yasskin]

On Feb 1, 2008 6:31 PM, Neal Norwitz <[EMAIL PROTECTED]> wrote:
> On Feb 1, 2008 2:52 PM, Mark Dickinson <[EMAIL PROTECTED]> wrote:
> >  The IBM format is particularly troublesome because
> > it's base 16 instead of base 2 (so e.g. multiplying a float by 2 can lose
> > bits), but it appears that recent IBM machines do both IBM format and IEEE
> > format floating-point.  I assume that the S-390 buildbots are using the IEEE
> > side---is this true?
>
> I don't know and suspect the only way to figure it out would be to
> write a test that would expose which is being used.  It's using gcc,
> so we probably get whatever the compiler defaults to.  Sometimes we
> have to specify flags for certain platforms.  For example -mieee on
> the Alpha.
>
> It's fine to check in something so that you can get an answer on a buildbot.

Actually, an even better way to do this would be to craft a test case
that exposes the assumptions you've found about the floating format.
Then it'll be a valuable regression test even after someone fixes the
bug.

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tracker marks my messages as spam :-)

[Jeffrey Yasskin]

On Feb 1, 2008 6:43 AM, Nicko van Someren <[EMAIL PROTECTED]> wrote:
> Perhaps it has to do with the low signal to noise ratio of your
> messages...

That was a little uncalled for. Be polite.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] trunc()

[Jeffrey Yasskin]

On Jan 27, 2008 3:37 PM, Terry Reedy <[EMAIL PROTECTED]> wrote:
> The actual claim seems to have been that the semantics of int()
> itself is fuzzy.  This in turn seems to be based one of two ideas (I am not
> sure which)
> a. 'int' might either mean to some people 'some int associated with the
> float input' rather than the more precise 'the integer part of the float
> input', or

Just like set(sequence) is the set associated with that sequence, not
the set part of that sequence, and float('3.14') is the float
associated with '3.14', not the float part of '3.14', etc. Type names
do not normally retrieve pieces of other objects. When there's no
unique or otherwise special instance of a given type associated with
an instance of another type, the first should not take the second in
its constructor. (This does not imply the inverse.)

On the other hand, to retrieve a piece of another object, you do
normally call a member on that object, so I'd have no objections to
changing trunc() and round() (although the second isn't strictly
retrieving a piece) to methods.

> b. that some people might think the latter means the same as the former.

I don't understand this sentence. I haven't seen anyone advocating
trunc() say that "some int associated with" means the same as "the
integer part of".

> Possibility a is easy taken care of by documentation, especially when the
> behavior is what people expect.

I'll grant that the behavior is relatively easily learned. It'll take
some more evidence to convince me that it's what people expect, given
that it _looks_ like a type conversion, and many systems implement
that conversion by rounding.

> Possibility b conflicts with what I
> believe to be both the plain English meaning of 'integer part' and what I
> believe Americans, at least, learn in elementary school.

You may understand the same thing from "int" and "integer part", but
that's a learned association that I didn't have before either you or
Raymond brought it up. Among their other differences, "int" is not an
English word.

> Moreover, I challenge the notion that 'truncate' is unambiguous.  It simply
> means to cut off, without specifying how much.  '3.141' is pi truncated to
> 3 decimal places.  Indeed, it would not be unreasonable to expect trunc()
> to have a second parameter specifying where to cut.

No more unreasonable than round() having the same second parameter.
And it would be reasonable to default it to "0" as round() does, since
that's the only distinguished point in its range.

> For one data point, I asked my bright 13-year-old for the 'integer part' of
> 3.1, 3.9, -3.1, and -3.9 and got the expected 3,3,-3,-3 (as with int()).
> But asking 'truncate' the same numbers or even 'truncate toward zero' got a
> blank stare, which did not surprise me too much as it is not a common
> (American) English word.

You asked a different question than the one we're talking about. You
should have asked your 13-year-old what the "int" of 3.9 was, or, even
better, how to convert 3.9 to an integer. For the first, you'd likely
have gotten the same blank stare. For the second, I expect you'd have
gotten either 4, or an objection that it's simply not an integer and
can't be "converted" to one.

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] trunc()

[Jeffrey Yasskin]

On Jan 27, 2008 9:26 AM, Andrea Griffini <[EMAIL PROTECTED]> wrote:
> On Jan 27, 2008 5:43 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote:
>
> > - Deprecating int() is pretty radical, I think it would have to
> > happen in the distant future. OR not at all. I'm at best +0 on this,
> > more like exactly 0. I realize that in practice this kills the idea.
> > The "purist" argument for it would have worked better if it was made
> > 18 years ago.
>
> Also what happens with "%i" % 3.14 ? We incidentally found a problem
> with a script using python 2.5 because apparently the "%" formatting
> operator doesn't use "int()" for doing the conversion (to be more specific
> if the float is too large for a 32-bit integer then the format operator chokes
> while the int() operator returns a long).
> Anyway I want just to say that if "implicit" conversion from float
> to integer goes away then what happens to formatting conversion ?
> Removing that too IMO would break a lot of code and it's IMO very
> difficult to help fixing that.

Well, it seems like it would be as easy to make some formatting
conversions raise a warning on float inputs as it would be to make
int() do it. But I think you're safe here; it doesn't look like either
will be deprecated.

-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] trunc()

[Jeffrey Yasskin]

On Jan 26, 2008 11:14 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> The idea that programmers are confused by int(3.7)-->3 may not be nuts, but 
> it doesn't match any experience I've had with any
> programmer, ever.

In C, I'm pretty sure I've seen people write (long)x where they'd have
been better off with lround(x). They know that the cast truncates, and
generally that they actually wanted to round, but the type they wanted
to get to was foremost in their mind, so they didn't bother to think
about it a little and write what they really wanted. It's not that
they're confused; it's that they're accepting a default that shouldn't
be a default.

Your other points seem to have been answered already, although people
will disagree on how compelling the answers are, so I won't repeat
them here.

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] trunc()

[Jeffrey Yasskin]

On Jan 26, 2008 2:46 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> [Christian Heimes]
> > In my opinion float(complex) does do the most sensible thing. It fails
> > and points the user to abs().
>
> Right.

To elaborate the point I was trying to make: If float() does not mean
"the float part of" and should not take a complex argument (which I
completely agree with), then int() does not mean "the int part of" and
should not take a float argument.

Even assuming that's correct, I'm not certain that it's worth breaking
(well, deprecating) backwards compatibility to achieve a more
consistent set of functions in this case. I think it is, but I'm happy
to leave that point up to the rest of the list (which does seem to be
leaning against it).

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] trunc()

[Jeffrey Yasskin]

On Jan 26, 2008 12:43 AM, Georg Brandl <[EMAIL PROTECTED]> wrote:
> That, and making int(float) == int(trunc(float)) seems like reasonable
> behavior to me as a person not involved in the discussion.

That's the only position in this discussion that I don't understand.
Although int() and trunc() would still have very slightly different
behaviors, they seem too close together (identical on most common
types) to be compatible with "only one way to do it". I've been
advocating trunc() under the assumption that int(float) would be
deprecated and eliminated as soon as practical (with an error message
similar to float(complex)). Could one of the people in favor of
keeping both explain why they think that's a good idea?

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] trunc()

[Jeffrey Yasskin]

On Jan 25, 2008 11:21 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> ...  int() for making ints from the non-fractional
> portion of a float.

This interpretation implies that complex should provide __float__() to
return the non-imaginary portion of a complex number. Is that what you
intend?

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Organization of ABC modules

[Jeffrey Yasskin]

On Jan 25, 2008 9:31 PM, Nick Coghlan <[EMAIL PROTECTED]> wrote:
> Raymond Hettinger wrote:
> >> If you want ABCs to be more easily recognizable
> >> as such, perhaps we could use a naming convention,
> >
> > Essentially, that's all I was asking for.  It doesn't
> > really matter to me whether numbers.py gets called
> > abc_numbers or abc.numbers.  Either one would be an
> > improvement.
>
> I think the module level is the wrong granularity to be thinking about
> this - look at a module like collections, which contains not only ABC's
> related to containers, but also some useful concrete containers like
> deque and namedtuple.
>
> Any naming convention would have to be at the class level. For example:
>
> class NumberABC(metaclass=ABCMeta):
> ...
>
> class ComplexABC(NumberABC):
> ...
>
> class RealABC(ComplexABC):
> ...
>
> class RationalABC(NumberABC):
> ...
>
> class IntegralABC(RationalABC):
> ...
>
> I think adopting such a convention (at least for the standard library)
> would actually be useful. Normally, finding isinstance() type checks in
> code bothers me severely as it usually indicates that duck-typing has
> been broken. On the other hand, if I find a check in code that does
> something like:
>
>if not isinstance(x, NumberABC):
>   raise TypeError("Not a number!")
>
> it would clearly still be extensible, as I would know just from the
> naming convention that a third party can register their type with
> NumberABC to make it usable with this function.
>
> Without a naming convention, I would always have to go check the
> definition of the type in the isinstance() call to see if the code was
> legitimate.

That's a good point. Someone already convinced me to name the test for
numbers.py, test_abstract_numbers.py, so renaming the module makes
sense too, although I agree that collections, which contains some
concrete classes, should keep its current name. If others agree, want
to send a patch?

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] trunc()

[Jeffrey Yasskin]

On Jan 25, 2008 9:45 AM, Facundo Batista <[EMAIL PROTECTED]> wrote:
> 2008/1/25, Jeffrey Yasskin <[EMAIL PROTECTED]>:
>
> > decision comes to be that int(float) should be blessed as a correct
> > way to truncate a float, I'd agree with Raymond that trunc() is just
> > duplication and should be eliminated. I'd, of course, rather have a
> > spelling that says what it means. :)
>
> Mmm... no. int() is a builtin way to transform the builtin data type
> float into the builtin data type float.

I assume you meant "int" instead of your second "float".

There doesn't have to be a way to convert from every builtin type to
every other builtin type. Take dict(float), for example. Further, if
there is a conversion, no law says it has to be named the same as the
target type. If there are two plausible ways to get from the source to
the target, using just the target's name to pick out one of them is
really a terrible idea.

But, just because the designers of C made a bad decision that we've
inherited, doesn't mean that it's a terrible idea to keep it. As Nick
said, defining int(float) as truncation has a long history, and there
are some obvious costs to breaking away from that. I think it is time
to correct the mistake, but it's not an open and shut case.

--
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] trunc()

[Jeffrey Yasskin]

On Jan 25, 2008 5:53 AM, Paul Moore <[EMAIL PROTECTED]> wrote:
> On 24/01/2008, Jeffrey Yasskin <[EMAIL PROTECTED]> wrote:
> > int has to be a builtin because it's a fundamental type. trunc()
> > followed round() into the builtins. I have no opinion on whether ceil
> > and floor should move there; it probably depends on how often they're
> > used.
>
> Suggestion:
>
> - int() has to stay in builtins for obvious reasons.
> - put *all* of trunc, ceil, floor, round into math.
> - make int(float) an error

I'd deal with Facundo's objection that you should be able to convert
between builtin datatypes without the use of a module by leaving trunc
in the builtins, but it looks like I'm in the minority here. If the
decision comes to be that int(float) should be blessed as a correct
way to truncate a float, I'd agree with Raymond that trunc() is just
duplication and should be eliminated. I'd, of course, rather have a
spelling that says what it means. :)

> The only fly in the ointment is that 2to3 can't handle the semantic
> issues around converting int(n) to math.trunc(n) because it can't
> detect the type of n. So why not make int(float) warn in -3 mode on
> 2.6, and then let 2to3 do the conversion (on the basis that 2to3
> should only be run on code that is -3 warning free)?

I'd be happy with that too.

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] trunc()

[Jeffrey Yasskin]

On Jan 25, 2008 4:28 AM, Facundo Batista <[EMAIL PROTECTED]> wrote:
> 2008/1/24, Guido van Rossum <[EMAIL PROTECTED]>:
>
> > > So you won't be able to construct an int from a float? That sucks (and
> > > is unintuitive).
> >
> > Yes, you can, but you have to specify how you want it done by using
> > trunc() or round() or ceil() or floor(). (In 3.0, round(x) will return
> > an int, not a float.)
>
>
> 2008/1/24, Jeffrey Yasskin <[EMAIL PROTECTED]>:
>
> > That needs to be updated and implemented. I think the decision was
> > that removing float.__int__() would break too much, so it needs a
> > deprecation warning in 3.0.
>
>
> What I understand here is as int() is "ambiguous", in the future if
> you want to specify how you want to convert a float to int.
>
> But ceil and floor returns a float. And round and trunc will return an
> int. So, how I could convert a float to its upper int? Like this?:
>
> >>> trunc(math.ceil(.3))
> 1

Like this, in 3.0:

>>> math.ceil(2.2)
3

There was a previous thread in which we decided not to change that
behavior in 2.6.

> BTW, int is not giving me a deprecation warning:
>
> >>> int(.1)
> 0

Correct; that's not implemented yet.

-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Float --> Integer Ratio

[Jeffrey Yasskin]

+1. Where do people want it, and what should its name be? I can
implement it if you like.

Another plausible output would be similar to frexp but with an integer
for the mantissa instead of a float. So:
 A) math.frexp(3.2) == (0.80004, 2)
 B) integral_frexp(3.2) == (3602879701896397, -50)
 C) exact_integral_ratio(3.2) == (3602879701896397, 1125899906842624)

(B) is easier to implement in C (not that that's a deciding factor);
(C) is probably more useful. So I'd vote for (C), your suggestion, but
just wanted to make the choice explicit.

Also, to make things explicit, the implementation in rational.py
doesn't guarantee that the fraction is in lowest terms. Should the
library version?

On Jan 24, 2008 3:10 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> rational.py contains code for turning a float into an
> exact integer ratio.  I've needed something like this in
> other situations as well.  The output is more convenient
> than the mantissa/exponent pair returned by math.frexp().
>
> I propose C-coding this function and either putting it in
> the math module or making it a method for floats. That
> would simplify and speed-up rational.py while making
> the tool available for other applications.  Also, the
> tool is currently in the wrong place (while the output
> is clearly useful for rational.py, the internals of
> the function are entirely about floats and ints and
> has no internal references to the rational module).
>
> Raymond
>
>
>
> ---
>
> def _binary_float_to_ratio(x):
> """x -> (top, bot), a pair of ints s.t. x = top/bot.
>
> The conversion is done exactly, without rounding.
> bot > 0 guaranteed.
> Some form of binary fp is assumed.
> Pass NaNs or infinities at your own risk.
>
> >>> _binary_float_to_ratio(10.0)
> (10, 1)
> >>> _binary_float_to_ratio(0.0)
> (0, 1)
> >>> _binary_float_to_ratio(-.25)
> (-1, 4)
> """
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com
>



-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/

"Religion is an improper response to the Divine." — "Skinny Legs and
All", by Tom Robbins
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] trunc()

[Jeffrey Yasskin]

On Jan 24, 2008 1:11 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> [Raymond Hettinger]
> > Since something similar is happening to math.ceil and math.floor,
> > I'm curious why trunc() ended-up in builtins instead of the math
> > module.  Doesn't it make sense to collect similar functions
> > with similar signatures in the same place?
>
> [Christian Heimes]
> > Traditionally the math module is a tiny wrapper around the
> > system's libm. Functions for magic hooks like __trunc__
> > usually end up in builtins. In this particular case I don't
> > mind where the function is going to live.
>
> Traditions have gone out the window.  ceil() and floor()
> are going to have their signatures changed (Real --> Integral)
> and are going to have their own magic methods.  They cannot
> be characterized as a thin wrapper around libm.
>
> So the question stands, why is trunc() different?  Can anything
> good come from having trunc() and int() in the same namespace?

One of my goals for trunc() is to replace the from-float use of int(),
even though we can't remove it for backward-compatibility reasons. PEP
3141 says:

  "Because the int() conversion implemented by float (and by
decimal.Decimal) is equivalent to but less explicit than trunc(),
let's remove it. (Or, if that breaks too much, just add a deprecation
warning.)"

That needs to be updated and implemented. I think the decision was
that removing float.__int__() would break too much, so it needs a
deprecation warning in 3.0.

int has to be a builtin because it's a fundamental type. trunc()
followed round() into the builtins. I have no opinion on whether ceil
and floor should move there; it probably depends on how often they're
used.

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] trunc()

[Jeffrey Yasskin]

On Jan 24, 2008 1:09 PM, Daniel Stutzbach
<[EMAIL PROTECTED]> wrote:
> On Jan 24, 2008 12:46 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote:
> > trunc() has well-defined semantics -- it takes a Real instance and
> > converts it to an Integer instance using round-towards-zero semantics.
> >
> > int() has undefined semantics -- it takes any object and converts it
> > to an int (a concrete type!) using whatever rules it likes -- the
> > definition of __int__ is up to whatever the source type likes to do.
>
> What are the use-cases for when trunc() vs int() should be used, and
> for when a class should define __trunc__ vs __int__?

If you intend to convert a real number (usually float, since Decimal
has decided not to support it) to an Integral (usually int), use
whichever of trunc(), round(), math.floor(), or math.ceil() you
intend. In 2.6, that list only includes trunc(). If you absolutely
need an int (the concrete, not duck type) from an Integral or you want
to parse a string, use int().

Real numbers should define __trunc__. Integrals and, perhaps, some
string-like types (maybe an MD5 class?) should define __int__.

At least, that's my suggestion.

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rational approximation methods

[Jeffrey Yasskin]

On Jan 20, 2008 5:54 PM, Tim Peters <[EMAIL PROTECTED]> wrote:
> What would be useful is a method that generates (i.e., a generator in
> the Python sense) the (continued fraction) convergents to a rational.
> People wanting specific constraints on a rational approximation
> (including, but not limited to, the two you identified) can easily
> build them on top of such a generator.

I think that's significantly better than my two proposals. (Looking
back at the issue, I see that Mark mentioned this possibility too and
I just didn't understand him.) I'll plan on implementing that, along
with the documentation Mark suggested, unless there are objections.

Thanks!
Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Rational approximation methods

[Jeffrey Yasskin]

In the Rational class that I've recently checked into Python 2.6
(http://bugs.python.org/issue1682), it might be nice to provide a
method that, given a particular rational number, returns a nearby
number that's nicer in some way. I know of two reasonable behaviors
for this operation. Since I don't know which is likely to be more
useful, or even if either is useful enough to include in the first
version of the module, I'm coming to you guys for advice. Both were
implemented a while ago in
http://svn.python.org/view/sandbox/trunk/rational/Rational.py?rev=40988&view=markup
and are based on the continued fraction representation of the number
(http://en.wikipedia.org/wiki/Continued_fraction#Best_rational_approximations).

The first returns the closest rational whose denominator is less than
a given integer. For discussion, we might call it
.limit_denominator(), although I'm also looking for good names. This
seems useful for computations in which you want to sacrifice some
accuracy for speed. On the other hand, Decimal may fill the niche for
fast-ish computation that surprises people less than float.

The second returns the simplest rational within some distance. For
instance, it'll prefer 22/7 over 333/106 if both are close enough. We
might call it .simplest_within() for now. This seems useful for
converting from float and displaying results to users, where we prefer
readability over accuracy or have reason to believe that a simpler
fraction might be more correct.

What does the list think?

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rounding Decimals

[Jeffrey Yasskin]

On Jan 13, 2008 3:41 PM, Leif Walsh <[EMAIL PROTECTED]> wrote:
> I haven't been watching the python-dev list for very long, so maybe
> this has already been discussed ad nauseam (in which case, sorry),
> but, from the devil's advocate-ish mathematics side of things, unless
> numbers.Decimal is planned to be implemented with infinite precision
> (which I doubt, but could not confirm with an admittedly small amount
> of research), shouldn't it implement numbers.Rational instead?  Was
> this ever discussed, or was it just assumed that numbers.Real was the
> right decision?

Guido mentioned the possibility briefly at
http://mail.python.org/pipermail/python-3000/2007-April/007015.html
("One could argue that float and Decimal are <:Q, but I'm not sure if
that makes things better pragmatically") but never really discussed as
far as I can find (although I did think about it :). I prefer having
float not implement Rational (Decimal's off the table) because
developers usually think of them as approximations to arbitrary,
possibly-irrational, real numbers, rather than as rational numbers
with some odd constraints on their denominators. That is, to me, the
ABCs a type implements are more about how developers should think
about the type than the implementation of the type.

[ A new thread is probably appropriate if anyone wants to discuss the
philosophy, but I probably won't participate... ]

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rounding Decimals

[Jeffrey Yasskin]

On Jan 12, 2008 8:21 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote:
>
> On Jan 12, 2008 5:09 PM, Jeffrey Yasskin <[EMAIL PROTECTED]> wrote:
> > During the discussion about the new Rational implementation
> > (http://bugs.python.org/issue1682), Guido and Raymond decided that
> > Decimal should not implement the new Real ABC from PEP 3141. So I've
> > closed http://bugs.python.org/issue1623 and won't be pursuing any of
> > the extra rounding methods mentioned on this thread.
>
> Well, I didn't really decide anything. I suggested that if the
> developers of Decimal preferred it, it might be better if Decimal did
> not implement the Real ABC, and Raymond said he indeed preferred it.
>
> Since this reduces the usefulness of numeric.Real, I'm somewhat
> disappointed in this outcome (as I imagine you are). However, Decimal
> has very little of my own sweat in it, and a lot of Raymond, Facundo
> (and others, including Mark), and the ABC thing is still somewhat
> experimental, so it wouldn't make sense for me to impose my ideas onto
> Decimal unilaterally, and I'm sure Raymond etc. have their reasons.

Sorry for misrepresenting you. I am a little disappointed, but if
we're right that numbers.Real is a good idea, then users will ask for
Decimal to implement it, and the Decimal developers can change their
minds in 3.1. Since I'm not one of those users, I'm not the right
person to argue for this anyway. :)

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rounding Decimals

[Jeffrey Yasskin]

During the discussion about the new Rational implementation
(http://bugs.python.org/issue1682), Guido and Raymond decided that
Decimal should not implement the new Real ABC from PEP 3141. So I've
closed http://bugs.python.org/issue1623 and won't be pursuing any of
the extra rounding methods mentioned on this thread.

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rounding Decimals

[Jeffrey Yasskin]

On Jan 6, 2008 10:51 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> [Jeffrey Yasskin]
> >> > I'm not
> >> > sure exactly what you're objecting to. Could you be more precise?
> >>
> >> You note said: "I'll implement Context.round() in a separate patch. 
> >> Comment away."
> >
> > Oh, sorry for not being clear then. I don't intend to write or discuss
> > that separate patch until this one's approved and submitted. I think
> > it's worth discussing eventually, but this change is more important. I
> > mentioned this sentiment at http://bugs.python.org/msg59417 too, but
> > clearly wasn't explicit enough in either case.
>
> It's hard to keep up with all of these patches and I'm growing concerned
> that some will sneak by.

You're certainly right to be concerned. The most significant patch,
the original backport of 3141, _did_ sneak through. The only email
I've gotten about it has been in response to my question about a
detail. I don't know how I could have made it more obvious though:
nobody who suggested the backport (http://bugs.python.org/issue1623)
seemed to think it would be contentious enough to suggest emailing
this list first. Since then, I've tried to keep things transparent by
suggesting that you add yourself to the nosy list of
http://bugs.python.org/issue1623, which is where I'll post any patches
before thinking about committing them.

> In the case of complex proposals, it is often
> helpful to have a patch that we can discuss concretely, but in the case of
> simple ones like this, they just clutter to patch tracker.  There's no need
> to create the patch until the idea has been discussed.

I always like to have a patch around because abstract discussions,
even (especially?) on simple topics, have a tendency to run off into
the weeds. A patch keeps things focused and moving forward.

> In the case of Context.round(), I've already advised against it.
> So, I'm not sure why you still intend write a separate patch
> and bring it back up for discussion.  Ideally, it should die right now.

You advised against it. Tim, Mark, and glyph advised for something
like it. I think it's consistent with the fact that most operations on
Decimal instances are mirrored on Context instances, and __round__
will be an operation on Decimal instances. That doesn't sound like
enough agreement to simply drop the discussion.

> The sentiment is unchanged -- please do not build-out the API for the
> decimal module.  It is intended to be minimal and needs to stay that
> way.  Further additions will only make it harder to use, harder to
> maintain, and will complicate efforts to rewrite the module in C.

I am not building out the decimal API. I am adding enough methods to
maintain the comment that "Decimal floating point objects share many
properties with the other builtin numeric types such as float and int.
All of the usual math operations and special methods apply." in both
2.6 and 3.0. After that's done, I am interested in talking about
adding one method that 3 people on this list have been interested in.
I understand that you need to be vigilant against extraneous methods,
but I'm really not just adding these on a whim.

On Jan 6, 2008 11:28 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> [Jeffrey Yasskin]
> > Given Guido's agreement, expect another version of this patch with
> > only __trunc__.
>
> Why is __trunc__ being backported?  Is a trunc() builtin being backported?  
> What is the point for a synonym for int() and __int__ in
> Py2.6.
>
> Unless I'm missing something, this doesn't improve Py2.6 in the least.

The trunc() builtin was backported in
http://svn.python.org/view?rev=59671&view=rev and hasn't been rolled
back. All of the reasons to include trunc() in 3.0 apply to 2.6. A
couple that I like are: if trunc had been around in the beginning,
__index__ would have been unnecessary; and the name says what it's
doing, unlike int(). I don't know what Guido's reasoning was for
accepting it into the PEP.

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rounding Decimals

[Jeffrey Yasskin]

On Jan 6, 2008 7:40 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> [Jeffrey Yasskin]
> > The other 3 methods
> > specified by PEP 3141 aren't strictly necessary for 2.6, but they will
> > be needed for 3.0. I'd rather not make the two versions of Decimal
> > gratuitously different, so this patch puts them in the 2.6 version
> > too.
>
> If I understand you correctly, then the patch backports magic methods that do 
> not have corresponding invocation methods in Py2.6.
> So, they are basically useless. If that's true, then the patch is clutter -- 
> it makes 2.6 less desirable.  It is not obvious to me
> how this will help someone transition to Py3.0.  I'm curious to hear how 
> Guido makes the call on this.
>
> Also, the goal of keeping versions identical across 2.6 and 3.0 is at odds 
> with previous discussions where I believe we said that
> that is *not* the goal and will likely not even be possible in many cases.  
> Besides, if the invocation of the magic methods is
> different in 3.0, how are you going to keep the test suite code the same 
> across versions?

Given Guido's agreement, expect another version of this patch with
only __trunc__.

> There should probably be a PEP sets clearer guidelines about what should be 
> backported from Py3.0.  Perhaps something like this:
> * If there is a new feature that can be implemented in both and will make 
> both more attractive, then it should be in both.
> * If something is going away or changing in 3.0, then the 2.6 conversion tool 
> mode should warn about it if possible.
> * If neither of the above apply, then leave 2.6 alone.
>
> > I'm not
> > sure exactly what you're objecting to. Could you be more precise?
>
> You note said: "I'll implement Context.round() in a separate patch. Comment 
> away."

Oh, sorry for not being clear then. I don't intend to write or discuss
that separate patch until this one's approved and submitted. I think
it's worth discussing eventually, but this change is more important. I
mentioned this sentiment at http://bugs.python.org/msg59417 too, but
clearly wasn't explicit enough in either case.

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rounding Decimals

[Jeffrey Yasskin]

On Jan 6, 2008 5:13 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> FWIW, I don't think all of these patches are helpful.  The implementations 
> are so easy and the effects are so obvious, that it is
> simply better to decide what to do first, then do it afterwards.
>
> My own preference is to leave the decimal module alone (except for a 
> __round__ method to be called by builtin.round).

You must mean __trunc__ to support builtin.trunc(), since
builtin.round no longer calls __round__ in 2.6. The other 3 methods
specified by PEP 3141 aren't strictly necessary for 2.6, but they will
be needed for 3.0. I'd rather not make the two versions of Decimal
gratuitously different, so this patch puts them in the 2.6 version
too. That's basically all of the changes in this patch, so I'm not
sure exactly what you're objecting to. Could you be more precise?

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pkgutil, pkg_resource and Python 3.0 name space packages

[Jeffrey Yasskin]

On Jan 6, 2008 3:28 PM, Oleg Broytmann <[EMAIL PROTECTED]> wrote:
>Now I think - if we don't want a separate Python's top-level namespace
> may be we should think about a separate top-level non-Python's (3rd
> parties') namespace? With it we could have database.sqlite (Python's
> sqlite) and user.database.sqlite (a newer version); and by doing import
> database.sqlite you know exactly what version you are importing.

Bleh.

I'm +1 on allowing third-party additions to the same 'database'
namespace that the stdlib puts packages in. People will just have to
get used to the package, and not the namespace, determining who to
complain to.

In theory, it might make sense to allow libraries to "close" some
namespaces to deal with Brett's worry, but I think the consenting
adults rule says not to bother.

-- 
Namasté,
Jeffrey Yasskin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com