Re: [Python-Dev] int/float freelists vs pymalloc
M.-A. Lemburg wrote: > On 2008-02-07 14:09, Andrew MacIntyre wrote: >> Probably in response to the same stimulus as Christian it occurred to me >> that the freelist approach had been adopted long before PyMalloc was >> enabled as standard (in 2.3), and that much of the performance gains >> between 2.2 and 2.3 were in fact due to PyMalloc. > > One of the hopes of having a custom allocator for Python was to be > able to get rid off all free lists. For some reason that never happened. > Not sure why. People were probably too busy with adding new > features to the language at the time ;-) Very probably ;-) > Something you could try to make PyMalloc perform better for the builtin > types is to check the actual size of the allocated PyObjects and then > make sure that PyMalloc uses arenas large enough to hold a good quantity > of them, e.g. it's possible that the float types fall into the same > arena as some other type and thus don't have enough "room" to use > as free list. Like MvL, I doubt it. Uncle Timmy did a pretty thorough nose-clean on PyMalloc. However, my tests do show that something is funny with the current freelist implementation for floats on at least 2 platforms, and that doing without that sort of optimisation for float objects would likely not be a hardship with PyMalloc. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
On 2008-02-08 08:21, Martin v. Löwis wrote:
>> One of the hopes of having a custom allocator for Python was to be
>> able to get rid off all free lists. For some reason that never happened.
>> Not sure why. People were probably too busy with adding new
>> features to the language at the time ;-)
>
> Probably not. It's more that the free lists still outperformed pymalloc.
>
>> Something you could try to make PyMalloc perform better for the builtin
>> types is to check the actual size of the allocated PyObjects and then
>> make sure that PyMalloc uses arenas large enough to hold a good quantity
>> of them, e.g. it's possible that the float types fall into the same
>> arena as some other type and thus don't have enough "room" to use
>> as free list.
>
> I don't think any improvements can be gained here. PyMalloc carves
> out pools of 4096 bytes from an arena when it runs out of blocks
> for a certain size class, and then keeps a linked list of pools of
> the same size class. So when many float objects get allocated,
> you'll have a lot of pools of the float type's size class.
> IOW, PyMalloc has always enough room.
Well, yes, it doesn't run out of memory, but if pymalloc needs
to allocate lots of objects of the same size, then performance
degrades due to the management overhead involved for checking
the free pools as well as creating new arenas as needed.
To reduce this overhead, it may be a good idea to preallocate
pools for common sizes and make sure they don't drop under a
certain threshold.
Here's a list of a few object sizes in bytes for Python 2.5
on an AMD64 machine:
>>> import mx.Tools
>>> mx.Tools.sizeof(int(0))
24
>>> mx.Tools.sizeof(float(0))
24
8-bit strings are var objects:
>>> mx.Tools.sizeof(str(''))
40
>>> mx.Tools.sizeof(str('a'))
41
Unicode objects use an external buffer:
>>> mx.Tools.sizeof(unicode(''))
48
>>> mx.Tools.sizeof(unicode('a'))
48
Lists do as well:
>>> mx.Tools.sizeof(list())
40
>>> mx.Tools.sizeof(list([1,2,3]))
40
Tuples are var objects:
>>> mx.Tools.sizeof(tuple())
24
>>> mx.Tools.sizeof(tuple([1,2,3]))
48
Old style classes:
>>> class C: pass
...
>>> mx.Tools.sizeof(C)
64
New style classes are a lot heavier:
>>> class D(object): pass
...
>>> mx.Tools.sizeof(D)
848
>>>
>>> mx.Tools.sizeof(type(2))
848
As you can see, Integers and floats fall into the same pymalloc size
class. What's strange in Andrew's result is that both integers
and floats use the same free list technique and fall into the same
pymalloc size class, yet the results are different.
The only difference that's apparent is that small integers are
shared, so depending on the data set used for the test, fewer
calls to pymalloc or the free list are made.
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Feb 08 2008)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free !
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Andrew MacIntyre wrote: > However, my tests do show that something is funny with the current > freelist implementation for floats on at least 2 platforms, and that > doing without that sort of optimisation for float objects would likely > not be a hardship with PyMalloc. float objects are slightly larger than int objects. On a 32bit OS both have ob_type pointer of size 4, a ref counter of size 4. But an int object has a value of type long with 4 bytes and a float stores its value in a double with size 8. I assume that the difference in size leads to a different allocation timing. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Andrew MacIntyre wrote: > For the int case, that patch is slower than no free list at all. For the > float case its very slightly faster (55.3s vs 55.5s) than no free list at > all. Slower than the trunk code for both cases. Did you run the test > scripts on your own machine? I've used a simple timeit call to bench mark the memory allocation. The first statement saturates the free lists and the second one benchmarks memory allocation and float creation. ./python -m timeit -n10 -s "[float(x) for x in list(range(1000))]" "for i in range(100): [float(x) for x in list(range(1000))]" I've done some experiments with a fixed length *free_list[] like dict and listobject.c. A fixed length list is faster than a single linked list. I'm going to upload the patch later. > In addition to the pure performance aspect, there is the issue of memory > utilisation. The current trunk code running the int test case in my > original post peaks at 151MB according to top on my FreeBSD box, dropping > back to about 62MB after the dict is destroyed (without a compaction). > The same script running on the no-freelist build of the interpreter peaks > at 119MB, with a minima of around 57MB. I wonder why the free list has such a huge impact in memory usage. Int objects are small (4 byte pointer to type, 4 byte Py_ssize_t and 4 byte value). A thousand int object should consume less than 20kB including overhead and padding. > On the 2 platforms I have, the current trunk code for ints would appear > to be as good as it gets speed-wise. If the memory utilisation issue > were to be considered significant, dropping the int freelist would have > its attractions... > > As far as floats go, all the indications I have suggest that the current > freelist code has a performance problem. Ditching the freelist and > reverting to inlined PyObject_New()/PyObject_Del() semantics would be an > attractive course as it appears to perform to within a whisker of the > best alternatives while simplifying the object's code and taking > advantage of pymalloc, however there are a number of other > platforms/compilers that need testing before it would be prudent to do > so. I can test the code on Linux and Windows XP. My tests have shown that a linked free list doesn't give a considerable performance boost - even for code that allocates and frees a lot of ints and floats at once. But a fixed length pointer array gives a measurable performance gain. > The compaction routine you added to trunk gets the job done as far as > freelist pruning goes, but I can't say I find it anything other than an > ugly solution (though there is precedent for the concept via the gc > module). Yeah, I won't argue with that. The compaction function is a crutch. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
On Feb 8, 2008 10:54 AM, Christian Heimes <[EMAIL PROTECTED]> wrote: > Andrew MacIntyre wrote: > > However, my tests do show that something is funny with the current > > freelist implementation for floats on at least 2 platforms, and that > > doing without that sort of optimisation for float objects would likely > > not be a hardship with PyMalloc. > > float objects are slightly larger than int objects. On a 32bit OS both > have ob_type pointer of size 4, a ref counter of size 4. But an int > object has a value of type long with 4 bytes and a float stores its > value in a double with size 8. > > I assume that the difference in size leads to a different allocation timing. It's not just size. Architectures may require data aligned on 4, 8, or 16 addresses for optimal performance depending on data type. IIRC, malloc aligns by 8 (not sure if that was a particular arch or very common). I don't know if pymalloc handles alignment. n ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Tim Peters wrote: > pymalloc ensures 8-byte alignment. This is one plausible reason to > keep the current int free list: an int object struct holds 3 4-byte > members on most boxes (type pointer, refcount, and the int's value), > and the int freelist code uses exactly 12 bytes for each on most > boxes. To keep 8-byte alignment, pymalloc would have to hand out a > 16-byte chunk per int object, wasting a fourth of the space (pymalloc > always rounds up a requested size to a multiple of 8, and ensures the > address returned is 8-byte aligned). Given the background information Python's long implementation could probably optimized. In 3.0 a long has 3 4-byte members (ref count, ob_size and *ob_type) plus one to many unsigned shorts (2 bytes each) to hold the value. If pymalloc aligns the objects at 8 byte address boundaries wouldn't it be better and slightly faster to use unsigned ints instead of unsigned shorts? Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Neal Norwitz wrote: > It's not just size. Architectures may require data aligned on 4, 8, > or 16 addresses for optimal performance depending on data type. IIRC, > malloc aligns by 8 (not sure if that was a particular arch or very > common). I don't know if pymalloc handles alignment. Yes, pymalloc takes care of alignment. From Object/obmalloc.c: Small requests are grouped in size classes spaced 8 bytes apart, due to the required valid alignment of the returned address. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Initial attempt to PyCon sprint tutorial slides are up
On Feb 6, 2008 4:46 AM, Facundo Batista <[EMAIL PROTECTED]> wrote: > 2008/2/4, Brett Cannon <[EMAIL PROTECTED]>: > > > The 1 MB PDF can be found at > > http://www.cs.ubc.ca/~drifty/pycon/sprint_tutorial.pdf . If you find > > any bad info or some info that is really lacking, let me know. But > > Brett, please tell me when you have a kind of finished version of > this... I want to send it to the Python Argentina mail list. > > Thank you! The corrected version of the slides are now up at the same location. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
On 2008-02-08 12:23, M.-A. Lemburg wrote:
> On 2008-02-08 08:21, Martin v. Löwis wrote:
>>> One of the hopes of having a custom allocator for Python was to be
>>> able to get rid off all free lists. For some reason that never happened.
>>> Not sure why. People were probably too busy with adding new
>>> features to the language at the time ;-)
>> Probably not. It's more that the free lists still outperformed pymalloc.
>>
>>> Something you could try to make PyMalloc perform better for the builtin
>>> types is to check the actual size of the allocated PyObjects and then
>>> make sure that PyMalloc uses arenas large enough to hold a good quantity
>>> of them, e.g. it's possible that the float types fall into the same
>>> arena as some other type and thus don't have enough "room" to use
>>> as free list.
>> I don't think any improvements can be gained here. PyMalloc carves
>> out pools of 4096 bytes from an arena when it runs out of blocks
>> for a certain size class, and then keeps a linked list of pools of
>> the same size class. So when many float objects get allocated,
>> you'll have a lot of pools of the float type's size class.
>> IOW, PyMalloc has always enough room.
>
> Well, yes, it doesn't run out of memory, but if pymalloc needs
> to allocate lots of objects of the same size, then performance
> degrades due to the management overhead involved for checking
> the free pools as well as creating new arenas as needed.
>
> To reduce this overhead, it may be a good idea to preallocate
> pools for common sizes and make sure they don't drop under a
> certain threshold.
>
> Here's a list of a few object sizes in bytes for Python 2.5
> on an AMD64 machine:
>
import mx.Tools
mx.Tools.sizeof(int(0))
> 24
mx.Tools.sizeof(float(0))
> 24
>
> 8-bit strings are var objects:
>
mx.Tools.sizeof(str(''))
> 40
mx.Tools.sizeof(str('a'))
> 41
>
> Unicode objects use an external buffer:
>
mx.Tools.sizeof(unicode(''))
> 48
mx.Tools.sizeof(unicode('a'))
> 48
>
> Lists do as well:
>
mx.Tools.sizeof(list())
> 40
mx.Tools.sizeof(list([1,2,3]))
> 40
>
> Tuples are var objects:
>
mx.Tools.sizeof(tuple())
> 24
mx.Tools.sizeof(tuple([1,2,3]))
> 48
>
> Old style classes:
>
class C: pass
> ...
mx.Tools.sizeof(C)
> 64
>
> New style classes are a lot heavier:
>
class D(object): pass
> ...
mx.Tools.sizeof(D)
> 848
mx.Tools.sizeof(type(2))
> 848
>
>
> As you can see, Integers and floats fall into the same pymalloc size
> class. What's strange in Andrew's result is that both integers
> and floats use the same free list technique and fall into the same
> pymalloc size class, yet the results are different.
>
> The only difference that's apparent is that small integers are
> shared, so depending on the data set used for the test, fewer
> calls to pymalloc or the free list are made.
Here's the same on a 32-bit machine. Notice the differences:
>>> import mx.Tools
>>> mx.Tools.sizeof(int(0))
12
>>> mx.Tools.sizeof(float(0))
16
>>> mx.Tools.sizeof(str(''))
24
>>> mx.Tools.sizeof(str('a'))
25
>>> mx.Tools.sizeof(unicode(''))
24
>>> mx.Tools.sizeof(unicode('a'))
24
>>> mx.Tools.sizeof(list())
20
>>> mx.Tools.sizeof(list([1,2,3]))
20
>>> mx.Tools.sizeof(tuple())
12
>>> mx.Tools.sizeof(tuple([1,2,3]))
24
>>> class C: pass
>>> mx.Tools.sizeof(C)
32
>>> class D(object): pass
>>> mx.Tools.sizeof(D)
424
>>> mx.Tools.sizeof(type(2))
424
Floats use 4 bytes more than integers. However, they still fall
into the same pymalloc size class (16 bytes this time around).
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Feb 08 2008)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free !
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
On 2008-02-08 19:28, Christian Heimes wrote: >> In addition to the pure performance aspect, there is the issue of memory >> utilisation. The current trunk code running the int test case in my >> original post peaks at 151MB according to top on my FreeBSD box, dropping >> back to about 62MB after the dict is destroyed (without a compaction). >> The same script running on the no-freelist build of the interpreter peaks >> at 119MB, with a minima of around 57MB. > > I wonder why the free list has such a huge impact in memory usage. Int > objects are small (4 byte pointer to type, 4 byte Py_ssize_t and 4 byte > value). A thousand int object should consume less than 20kB including > overhead and padding. The free lists keep parts of the pymalloc pools alive. Since these are only returned to the OS if the whole pool is unused, a single object could keep 4k of memory associated with the process. I suppose that the remaining few MBs shown by the OS are not really used by the process, but simply kept associated with the process by the OS in case it quickly needs more memory. In order to be sure about the true memory usage, you'd have to force the OS to grab all available memory, e.g. by running a huge process right next to the one you're testing. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 08 2008) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Initial attempt to PyCon sprint tutorial slides are up
On Feb 8, 2008 12:53 PM, Christian Heimes <[EMAIL PROTECTED]> wrote: > Brett Cannon wrote: > > The corrected version of the slides are now up at the same location. > > I found a minor mistake > > PC/ > * Build files for compilers older than VS 7.1. > > The PC directory contains build directories for VC6.0, VC 7.1, VC8.0 and > OS2. Then a README file either in that directory or PCbuild might need updating. But then again Python's own README could use a refresh. =) -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
M.-A. Lemburg wrote: > As you can see, Integers and floats fall into the same pymalloc size > class. What's strange in Andrew's result is that both integers > and floats use the same free list technique and fall into the same > pymalloc size class, yet the results are different. My take is not that PyMalloc is so much better in the float case, but that the float freelist implementation is suboptimal (though exactly why is subject for conjecture, given it's almost identical to the int implementation). > The only difference that's apparent is that small integers are > shared, so depending on the data set used for the test, fewer > calls to pymalloc or the free list are made. My testing was done with millions of unique integers, so the small int handling should be deep in the measurement noise. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
> Given the background information Python's long implementation could > probably optimized. In 3.0 a long has 3 4-byte members (ref count, > ob_size and *ob_type) plus one to many unsigned shorts (2 bytes each) to > hold the value. If pymalloc aligns the objects at 8 byte address > boundaries wouldn't it be better and slightly faster to use unsigned > ints instead of unsigned shorts? You mean, as the digits? You would have to rewrite the entire long datatype, which is a tedious exercise. Plus, I believe you would have to make some operations 64-bit on all systems: when you multiply digits today, the product will be 32-bit. If you extend the digits, you might get 64-bit results, which will be slow on 32-bit systems (plus some 32-bit systems don't support a 64-bit integer type at all in their compilers). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Tim Peters wrote: > pymalloc ensures 8-byte alignment. This is one plausible reason to > keep the current int free list: an int object struct holds 3 4-byte > members on most boxes (type pointer, refcount, and the int's value), > and the int freelist code uses exactly 12 bytes for each on most > boxes. To keep 8-byte alignment, pymalloc would have to hand out a > 16-byte chunk per int object, wasting a fourth of the space (pymalloc > always rounds up a requested size to a multiple of 8, and ensures the > address returned is 8-byte aligned). Hmmm... the funny thing is that the freelist approach is showing higher memory consumption than the PyMalloc approach for the int case... -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Martin v. Löwis wrote: > You mean, as the digits? You would have to rewrite the entire long > datatype, which is a tedious exercise. Plus, I believe you would have > to make some operations 64-bit on all systems: when you multiply > digits today, the product will be 32-bit. If you extend the digits, > you might get 64-bit results, which will be slow on 32-bit systems > (plus some 32-bit systems don't support a 64-bit integer type at all > in their compilers). I pass! :) It may be worth a try for Python 4.0 when most OS are 64bit. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Tracker Issues
ACTIVITY SUMMARY (02/01/08 - 02/08/08)
Tracker at http://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue
number. Do NOT respond to this message.
1691 open (+30) / 12209 closed (+18) / 13900 total (+48)
Open issues with patches: 437
Average duration of open issues: 754 days.
Median duration of open issues: 1065 days.
Open Issues Breakdown
open 1667 (+30)
pending24 ( +0)
Issues Created Or Reopened (49)
___
localeconv() does not encode returned strings02/01/08
http://bugs.python.org/issue1995created pitrou
float.as_integer_ratio() needs to return fraction in lowest term 02/01/08
CLOSED http://bugs.python.org/issue1996created rhettinger
unicode and string compare should not cause an exception 02/01/08
CLOSED http://bugs.python.org/issue1997created aaron_watters
documentation grammatical error 02/02/08
CLOSED http://bugs.python.org/issue1998created mickbeaver
easy
wrong tracker02/02/08
CLOSED http://bugs.python.org/issue1999created mickbeaver
Undefined symbols: _PyOS_mystrnicmp on Mac OS X 02/02/08
CLOSED http://bugs.python.org/issue2000created belopolsky
Pydoc interactive browsing enhancement 02/02/08
http://bugs.python.org/issue2001created ron_adam
Make int() fall back to trunc() 02/02/08
CLOSED http://bugs.python.org/issue2002created jyasskin
patch
Incorrect definition of new-style class 02/03/08
CLOSED http://bugs.python.org/issue2003created thaneplummer
tarfile extractall() allows local attacker to overwrite files wh 02/03/08
CLOSED http://bugs.python.org/issue2004created mebrown
posixmodule expects sizeof(pid_t/gid_t/uid_t) <= sizeof(long)02/03/08
http://bugs.python.org/issue2005created tiran
easy
asyncore loop lacks timers and work tasks02/04/08
http://bugs.python.org/issue2006created janssen
cookielib lacks FileCookieJar class for Internet Explorer02/04/08
http://bugs.python.org/issue2007created janssen
cookielib lacks FileCookieJar class for Safari 02/04/08
http://bugs.python.org/issue2008created janssen
Grammar change to prevent shift/reduce problem with varargslist 02/05/08
http://bugs.python.org/issue2009created dalke
Link to howto section on re module documentation incorrect 02/05/08
CLOSED http://bugs.python.org/issue2010created knorby
compiler.parse("1;") adds unexpected extra Discard(Const(None)) 02/05/08
CLOSED http://bugs.python.org/issue2011created dalke
Add migration step for DictMixin -> collections.MutableMapping 02/05/08
http://bugs.python.org/issue2012created tiran
Long object free list optimization 02/05/08
http://bugs.python.org/issue2013created tiran
patch
xmlrpclib cannot send datetime objects with dates before 190002/05/0
Re: [Python-Dev] int/float freelists vs pymalloc
[Neal Norwitz] > It's not just size. Architectures may require data aligned on 4, 8, > or 16 addresses for optimal performance depending on data type. IIRC, > malloc aligns by 8 (not sure if that was a particular arch or very > common). Just very common. Because malloc has no idea what the pointer it returns will be used for, it needs to satisfy the strictest alignment requirement for any type exposed by the C language. As an extreme example, when I worked at Kendall Square Research, our hardware supported atomic locking on a "subpage" basis, and HW subpage addresses were all those divisible by 128 Subpages were exposed via C extensions, so the KSR malloc had to return 128-byte aligned pointers. > I don't know if pymalloc handles alignment. pymalloc ensures 8-byte alignment. This is one plausible reason to keep the current int free list: an int object struct holds 3 4-byte members on most boxes (type pointer, refcount, and the int's value), and the int freelist code uses exactly 12 bytes for each on most boxes. To keep 8-byte alignment, pymalloc would have to hand out a 16-byte chunk per int object, wasting a fourth of the space (pymalloc always rounds up a requested size to a multiple of 8, and ensures the address returned is 8-byte aligned). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Christian Heimes wrote: > Andrew MacIntyre wrote: >> For the int case, that patch is slower than no free list at all. For the >> float case its very slightly faster (55.3s vs 55.5s) than no free list at >> all. Slower than the trunk code for both cases. Did you run the test >> scripts on your own machine? > > I've used a simple timeit call to bench mark the memory allocation. The > first statement saturates the free lists and the second one benchmarks > memory allocation and float creation. > > ./python -m timeit -n10 -s "[float(x) for x in list(range(1000))]" "for > i in range(100): [float(x) for x in list(range(1000))]" Try testing with 100 rather than 1000. My testing was deliberately looking at the handling of large numbers of ints/floats, because small numbers (eg 1000) are not enough to make wasted memory recovery worthwhile. > I've done some experiments with a fixed length *free_list[] like dict > and listobject.c. A fixed length list is faster than a single linked > list. I'm going to upload the patch later. I tried a LIFO stack implementation (though I won't claim to have done it well), and found it slightly slower than no freelist at all. The advantage of such an approach is that the known size of the stack makes deallocating excess objects easy (and thus no need for sys.compact_free_list() ). I have been guilty of not paying attention to the performance of the small cases. >> In addition to the pure performance aspect, there is the issue of memory >> utilisation. The current trunk code running the int test case in my >> original post peaks at 151MB according to top on my FreeBSD box, dropping >> back to about 62MB after the dict is destroyed (without a compaction). >> The same script running on the no-freelist build of the interpreter peaks >> at 119MB, with a minima of around 57MB. > > I wonder why the free list has such a huge impact in memory usage. Int > objects are small (4 byte pointer to type, 4 byte Py_ssize_t and 4 byte > value). A thousand int object should consume less than 20kB including > overhead and padding. My test script allocates 400 ints - ~48MB. Much of the memory "pulse" is the creation/deletion of a dict with 200 items (25MB?). Its interesting to watch the freelist case, as first time through the loop the process size peaks at 109MB, then at the end of the 2nd pass it peaks at 151MB which is maintained on subsequent passes through the loop. (BTW, on this machine just starting the python interpreter leaves the process size at a bit over 3MB). The no-freelist build peaks at its maximum of 119MB on the first pass through the loop. >> On the 2 platforms I have, the current trunk code for ints would appear >> to be as good as it gets speed-wise. If the memory utilisation issue >> were to be considered significant, dropping the int freelist would have >> its attractions... >> >> As far as floats go, all the indications I have suggest that the current >> freelist code has a performance problem. Ditching the freelist and >> reverting to inlined PyObject_New()/PyObject_Del() semantics would be an >> attractive course as it appears to perform to within a whisker of the >> best alternatives while simplifying the object's code and taking >> advantage of pymalloc, however there are a number of other >> platforms/compilers that need testing before it would be prudent to do >> so. > > I can test the code on Linux and Windows XP. My tests have shown that a > linked free list doesn't give a considerable performance boost - even > for code that allocates and frees a lot of ints and floats at once. But > a fixed length pointer array gives a measurable performance gain. I've added my research patches to issue 2039, so I'd be interested to know how they rate in comparison. In addition to the small case testing mentioned above, I would suggest you do some large case testing (such as with the scripts I included in my original posting) as this is the justification for the effort on wasted memory recovery. Cheers, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Initial attempt to PyCon sprint tutorial slides are up
Brett Cannon wrote: > The corrected version of the slides are now up at the same location. I found a minor mistake PC/ * Build files for compilers older than VS 7.1. The PC directory contains build directories for VC6.0, VC 7.1, VC8.0 and OS2. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
