[Python-Dev] Ann: PyPy Sprint before PYCON 2005 in Washington

[Christian Tismer]

PyPy Sprint before PYCON 2005 in Washington
---
In the four days from 19th March till 22th March (inclusive)
the PyPy team will host a sprint on their new Python-in-Python
implementation.   The PyPy project was granted funding by the
European Union as part of its Sixth Framework Program,
and is now on track to produce a stackless Python-in-Python
Just-in-Time Compiler by December 2006.  Our Python
implementation, released under the MIT/BSD license, already
provides new levels of flexibility and extensibility at the
core interpreter and object implementation level.
Armin Rigo and Holger Krekel will also give talks about PyPy
and the separate  py.test tool (used to perform various kinds
of testing in PyPy) during the conference.
Naturally, we are eager to see how the other re-implementation
of Python, namely IronPython, is doing and to explore
collaboration possibilities.  Of course, that will depend on
the degree of openness that Microsoft wants to employ.
The Pycon2005 sprint is going to focus on reaching
compatibility with CPython (currently we target version 2.3.4)
for our PyPy version running on top of CPython. One goal of
the sprint is to pass 60% or more of the unmodified regression
tests of mainline CPython.  It will thus be a great way to get
to know CPython and PyPy better at the same time!  Other
possible work areas include:
- translation to C to get a first working lower-level representation
  of the interpreter "specified in Python"
- integrating and implementing a full parser/compiler chain
  written in Python maybe already targetting the new
  AST-branch of mainline CPython
- fixing various remaining issues that will come up while
  trying to reach the compatibility goal
- integrate or code pure python implementations of some Python modules
  currently written in C.
- whatever issues you come up with! (please tell us
  before hand so we can better plan introductions etc.pp.)
Besides core developers, Bea Düring will be present to help
improving and document our sprint and agile development
process.
We are going to give tutorials about PyPy's basic concepts and
provide help to newcomers usually by pairing them with
experienced pypythonistas. However, we kindly ask newcomers to
be present on the first day's morning (19th of March) of the
sprint to be able to get everyone a smooth start into the
sprint. So far most newcomers had few problems in getting a
good start into our codebase.  However, it is good to have the
following preparational points in mind:
- some experience with programming in the Python language and
  interest to dive deeper
- subscription to  pypy-dev and  pypy-sprint at
http://codespeak.net/pypy/index.cgi?lists
- have a subversion-client, Pygame and graphviz installed on
  the machine you bring to the sprint.
- have a look at our current  documentation, especially the
  architecture and  getting-started documents under
http://codespeak.net/pypy/index.cgi?doc
The pypy-dev and pypy-sprint lists are also the contact points
for raising questions and suggesting and discussing sprint
topics beforehand. We are on #pypy on irc.freenode.net most
of the time. Please don't hesitate to contact us or introduce
yourself and your interests!
Logistics
-
Organizational details will be posted to pypy-sprint and are
or will be available in the Pycon2005-Sprint wiki here:
http://www.python.org/moin/PyConDC2005/Sprints
Registration

send mail to [EMAIL PROTECTED], stating the days
you can be present and any specific interests if applicable.
Registered Participants
---
all days:
 Jacob Hallén
 Armin Rigo
 Holger Krekel
 Samuele Pedroni
 Anders Chrigström
 Bea Düring
 Christian Tismer
 Richard Emslie
--
Christian Tismer :^)   
tismerysoft GmbH : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A :*Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
 whom do you want to sponsor today?   http://www.stackless.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: [Zope] Windows Low Fragementation Heap yields speedup of ~15%

[Tim Peters]

[Gfeller Martin]
> I'm running a large Zope application on a 1x1GHz CPU 1GB mem
> Window XP Prof machine using Zope 2.7.3 and Py 2.3.4
> The application typically builds large lists by appending
> and extending them.

That's historically been an especially bad case for Windows systems,
although the behavior varied across specific Windows flavors.  Python
has changed lots of things over time to improve it, including yet
another twist on list-reallocation strategy new in Python 2.4.

> We regularly observed that using a given functionality a
> second time using the same process was much slower (50%)
> than when it ran the first time after startup.

Heh.  On Win98SE, the _first_ time you ran pystone after rebooting the
machine, it ran twice as fast as the second (or third, fourth, ...)
time you tried it.  The only way I ever found to get back the original
speed without a reboot was to run a different process in-between that
allocated almost all physical memory in one giant chunk.  Presumably
that convinced Win98SE to throw away its fragmented heap and start
over again.

> This behavior greatly improved with Python 2.3 (thanks
> to the improved Python object allocator, I presume).

The page you reference later describes a scheme that's (at least
superficially) a lot like pymalloc uses for "small objects".  In
effect, pymalloc takes over buckets 1-32 in the table.

> Nevertheless, I tried to convert the heap used by Python
> to a Windows Low Fragmentation Heap (available on XP
> and 2003 Server). This improved the overall run time
> of a typical CPU-intensive report by about 15%
> (overall run time is in the 5 minutes range), with the
> same memory consumption.
>
> I consider 15% significant enough to let you know about it.

Yes, and thank you.  FYI, Python doesn't call any of the Win32 heap
functions directly; the behavior it sees is inherited from whatever
Microsoft's C implementation uses to support C's
malloc()/realloc()/free().  pymalloc requests 256KB at a time from the
platform malloc, and carves it up itself, so pymalloc isn't affected
by LFH (LFH punts on requests over 16KB, much as pymalloc punts on
requests over 256 bytes).

But "large objects" (including list guts) don't go thru pymalloc to
begin with, so as long as your list guts fit in 16KB, LFH could make a
real difference to how they behave.  Well, actually, it's probably
more the case that LFH gives a boost by keeping small objects _out_ of
the general heap.  Then growing a giant list doesn't bump into
gazillions of small objects.

> For information about the Low Fragmentation Heap, see
> 
> 
> Best regards,
> Martin
>
> PS: Since I don't speak C, I used ctypes to convert all
>heaps in the process to LFH (I don't know how to determine
>which one is the C heap).

It's the one consuming all the time .
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] builtin_id() returns negative numbers

[Josiah Carlson]

James Y Knight <[EMAIL PROTECTED]> wrote:
> 
> 
> On Feb 14, 2005, at 10:41 AM, Tim Peters wrote:
> 
> >> Wouldn't it be more elegant to make builtin_id() return an unsigned
> >> long integer?
> >
> > I think so.  This is the function ZODB 3.3 uses, BTW:
> >
> > def positive_id(obj):
> > """Return id(obj) as a non-negative integer."""
> >  [...]
> 
> I think it'd be nice to change it, too. Twisted also uses a similar 
> function.
> 
> However, last time this topic came up, this Tim Peters guy argued 
> against it. ;)
> 
> Quoting 
> http://mail.python.org/pipermail/python-dev/2004-November/050049.html:
> 
> > Python doesn't promise to return a postive integer for id(), although
> > it may have been nicer if it did.  It's dangerous to change that now,
> > because some code does depend on the "32 bit-ness as a signed integer"
> > accident of CPython's id() implementation on 32-bit machines.  For
> > example, code using struct.pack(), or code using one of ZODB's
> > specialized int-key BTree types with id's as keys.

All Tim was saying is that you can't /change/ builtin_id() because of
backwards compatibiliity with Zope and struct.pack().  You are free to
create a positive_id() function, and request its inclusion into builtins
(low probability; people don't like doing that). Heck, you are even free
to drop it in your local site.py implementation.  But changing the
current function is probably a no-no.

 - Josiah

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] builtin_id() returns negative numbers

[Tim Peters]

[James Y Knight]
> I think it'd be nice to change it, too. Twisted also uses a similar
> function.
>
> However, last time this topic came up, this Tim Peters guy argued
> against it. ;)
>
> Quoting
> http://mail.python.org/pipermail/python-dev/2004-November/050049.html:
> 
>> Python doesn't promise to return a postive integer for id(), although
>> it may have been nicer if it did.  It's dangerous to change that now,
>> because some code does depend on the "32 bit-ness as a signed integer"
>> accident of CPython's id() implementation on 32-bit machines.  For
>> example, code using struct.pack(), or code using one of ZODB's
>> specialized int-key BTree types with id's as keys.

Yup, it's still a tradeoff, and it's still dangerous (as any change in
visible behavior is).  It's especially unfortunate that since

"%x" % id(obj)

does produce different output in 2.4 than in 2.3 when id(obj) < 0, we
would change that output _again_ in 2.5 if id(obj) grew a new
non-negative promise.  That is, the best time to do this would have
been for 2.4.  Maybe it's just a wart we have to live with now; OTOH,
the docs explicitly warn that id() may return a long, so any code
relying on "short int"-ness has always been relying on an
implementation quirk.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] builtin_id() returns negative numbers

[James Y Knight]

On Feb 14, 2005, at 10:41 AM, Tim Peters wrote:
Wouldn't it be more elegant to make builtin_id() return an unsigned
long integer?
I think so.  This is the function ZODB 3.3 uses, BTW:
def positive_id(obj):
"""Return id(obj) as a non-negative integer."""
 [...]
I think it'd be nice to change it, too. Twisted also uses a similar 
function.

However, last time this topic came up, this Tim Peters guy argued 
against it. ;)

Quoting 
http://mail.python.org/pipermail/python-dev/2004-November/050049.html:

Python doesn't promise to return a postive integer for id(), although
it may have been nicer if it did.  It's dangerous to change that now,
because some code does depend on the "32 bit-ness as a signed integer"
accident of CPython's id() implementation on 32-bit machines.  For
example, code using struct.pack(), or code using one of ZODB's
specialized int-key BTree types with id's as keys.
James
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] builtin_id() returns negative numbers

[Tim Peters]

[Troels Walsted Hansen]
> The Python binding in libxml2 uses the following code for __repr__():
>
> class xmlNode(xmlCore):
> def __init__(self, _obj=None):
> self._o = None
> xmlCore.__init__(self, _obj=_obj)
> 
> def __repr__(self):
> return "" % (self.name, id (self))
>
> With Python 2.3.4 I'm seeing warnings like the one below:
> :2357: FutureWarning: %u/%o/%x/%X of negative int
> will return a signed string in Python 2.4 and up
> 
> I believe this is caused by the memory address having the sign bit set,
> causing builtin_id() to return a negative integer.

Yes, that's right.

> I grepped around in the Python standard library and found a rather
> awkward work-around that seems to be slowly propagating to various
> module using the "'%x' % id(self)" idiom:

No, it's not propagating any more:  I see that none of these exist in 2.4:

> Lib/asyncore.py:
> # On some systems (RH10) id() can be a negative number.
> # work around this.
> MAX = 2L*sys.maxint+1
> return '<%s at %#x>' % (' '.join(status), id(self)&MAX)
> 
> $ grep -r 'can be a negative number' *
> Lib/asyncore.py:# On some systems (RH10) id() can be a negative
> number.
> Lib/repr.py:# On some systems (RH10) id() can be a negative
> number.
> Lib/tarfile.py:# On some systems (RH10) id() can be a negative
> number.
> Lib/test/test_repr.py:# On some systems (RH10) id() can be a
> negative number.
> Lib/xml/dom/minidom.py:# On some systems (RH10) id() can be a
> negative number.
>
> There are many modules that do not have this work-around in Python 2.3.4.

Not sure, but it looks like this stuff was ripped out in 2.4 simply
because 2.4 no longer produces a FutureWarning in these cases.  That
doesn't address that the output changed, or that the output for a
negative id() produced by %x under 2.4 is probably surprising to most.

> Wouldn't it be more elegant to make builtin_id() return an unsigned
> long integer?

I think so.  This is the function ZODB 3.3 uses, BTW:

# Addresses can "look negative" on some boxes, some of the time.  If you
# feed a "negative address" to an %x format, Python 2.3 displays it as
# unsigned, but produces a FutureWarning, because Python 2.4 will display
# it as signed.  So when you want to prodce an address, use positive_id() to
# obtain it.
def positive_id(obj):
"""Return id(obj) as a non-negative integer."""

result = id(obj)
if result < 0:
# This is a puzzle:  there's no way to know the natural width of
# addresses on this box (in particular, there's no necessary
# relation to sys.maxint).  Try 32 bits first (and on a 32-bit
# box, adding 2**32 gives a positive number with the same hex
# representation as the original result).
result += 1L << 32
if result < 0:
# Undo that, and try 64 bits.
result -= 1L << 32
result += 1L << 64
assert result >= 0 # else addresses are fatter than 64 bits
return result

The gives a non-negative result regardless of Python version and
(almost) regardless of platform (the `assert` hasn't triggered on any
ZODB 3.3 platform yet).

> Is the performance impact too great?

For some app, somewhere, maybe.  It's a tradeoff.  The very widespread
practice of embedding %x output from id() favors getting rid of the
sign issue, IMO.

> A long integer is used on platforms where SIZEOF_VOID_P > SIZEOF_LONG
> (most 64 bit platforms?),

Win64 is probably the only major (meaning likely to be popular among
Python users) platform where sizeof(void*) > sizeof(long).

> so all Python code must be prepared to handle it already...

In theory .
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] builtin_id() returns negative numbers

[Troels Walsted Hansen]

Hi all,
The Python binding in libxml2 uses the following code for __repr__():
class xmlNode(xmlCore):
def __init__(self, _obj=None):
self._o = None
xmlCore.__init__(self, _obj=_obj)
def __repr__(self):
return "" % (self.name, id (self))
With Python 2.3.4 I'm seeing warnings like the one below:
:2357: FutureWarning: %u/%o/%x/%X of negative int 
will return a signed string in Python 2.4 and up

I believe this is caused by the memory address having the sign bit set, 
causing builtin_id() to return a negative integer.

I grepped around in the Python standard library and found a rather 
awkward work-around that seems to be slowly propagating to various 
module using the "'%x' % id(self)" idiom:

Lib/asyncore.py:
# On some systems (RH10) id() can be a negative number.
# work around this.
MAX = 2L*sys.maxint+1
return '<%s at %#x>' % (' '.join(status), id(self)&MAX)
$ grep -r 'can be a negative number' *
Lib/asyncore.py:# On some systems (RH10) id() can be a negative 
number.
Lib/repr.py:# On some systems (RH10) id() can be a negative 
number.
Lib/tarfile.py:# On some systems (RH10) id() can be a negative 
number.
Lib/test/test_repr.py:# On some systems (RH10) id() can be a 
negative number.
Lib/xml/dom/minidom.py:# On some systems (RH10) id() can be a 
negative number.

There are many modules that do not have this work-around in Python 2.3.4.
Wouldn't it be more elegant to make builtin_id() return an unsigned 
long integer? Is the performance impact too great?

A long integer is used on platforms where SIZEOF_VOID_P > SIZEOF_LONG 
(most 64 bit platforms?), so all Python code must be prepared to handle 
it already...

Troels
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: [Python-checkins] python/dist/src/Doc/lib libimp.tex, 1.36, 1.36.2.1 libsite.tex, 1.26, 1.26.4.1 libtempfile.tex, 1.22, 1.22.4.1 libos.tex, 1.146.2.1, 1.146.2.2

[Just van Rossum]

[EMAIL PROTECTED] wrote:

>  \begin{datadesc}{PY_RESOURCE}
> -The module was found as a Macintosh resource.  This value can only be
> -returned on a Macintosh.
> +The module was found as a Mac OS 9 resource.  This value can only be
> +returned on a Mac OS 9 or earlier Macintosh.
>  \end{datadesc}

not entirely true: it's limited to the sa called "OS9" version of
MacPython, which happily runs natively on OSX as a Carbon app...

Just
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com