> And, of course, if the int/float freelist scheme was a real issue
> we would have probably heard of it by now in a very sound way.
As Aahz says: people run into this problem frequently, and then report it.
Regards,
Martin
___
Python-Dev mailing list
P
On Fri, Feb 22, 2008, Andrew MacIntyre wrote:
> Vladimir Marangozov wrote:
>>
>> And, of course, if the int/float freelist scheme was a real issue
>> we would have probably heard of it by now in a very sound way.
>
> Not much noise has been made here, but I've come across 2 separate
> complaints
Vladimir Marangozov wrote:
> May I chime in? :-)
Please do!
> Gents, the current implementation of Python's memory management
> is really fine and most problems it used to have in the past
> have been fixed in recent years (at least the ones I know of).
>
> Probably the only one left, indeed, is
May I chime in? :-)
Gents, the current implementation of Python's memory management
is really fine and most problems it used to have in the past
have been fixed in recent years (at least the ones I know of).
Probably the only one left, indeed, is the potential unbounded
growth of int/float objec
On Wed, Feb 20, 2008, Andrew MacIntyre wrote:
>
> I've now written up my testing and attached the write-up to issue 2039
> (http://bugs.python.org/issue2039).
Nice work! Too often we (generic we) don't try to re-simplify code.
--
Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncr
Andrew MacIntyre wrote:
> M.-A. Lemburg wrote:
>> If we're down to voting, here's my vote:
>>
>> +1 on removing the freelists from ints and floats, but not the
>>small int sharing optimization
>>
>> +1 on focusing on improving pymalloc to handle int and float
>>object allocations even bette
Christian Heimes <[EMAIL PROTECTED]> wrote:
> +1 on focusing on improving pymalloc to handle int and float
>object allocations even better
I wonder if the int and float types could use a faster internal
pymalloc interface. I can't remember the details but I seem to
recall that pymalloc must j
Andrew MacIntyre wrote:
>> By the way objects are always aligned at 8 byte address boundaries. A 12
>> or 4 bytes object occupies 16 bytes.
>
> No, with PyMalloc a 4 byte object occupies 8 bytes (see the comments at
> the top of Objects/obmalloc.c).
I know. It's a typo. It should read "12 or 14 b
Christian Heimes wrote:
> By the way objects are always aligned at 8 byte address boundaries. A 12
> or 4 bytes object occupies 16 bytes.
No, with PyMalloc a 4 byte object occupies 8 bytes (see the comments at
the top of Objects/obmalloc.c).
Cheers,
Andrew.
--
-
Christian Heimes wrote:
> By the way objects are always aligned at 8 byte address boundaries. A 12
> or 4 bytes object occupies 16 bytes.
Maybe pymalloc should be enhanced so it can cope with
certain odd-sized objects, such as 12 bytes?
--
Greg
___
Pyt
M.-A. Lemburg wrote:
> +1 on removing the freelists from ints and floats, but not the
>small int sharing optimization
>
> +1 on focusing on improving pymalloc to handle int and float
>object allocations even better
>
> -1 on changing the freelist implementations to use pymalloc for
>a
M.-A. Lemburg wrote:
> On 2008-02-13 12:56, Andrew MacIntyre wrote:
>> I'm not that interested in debating the detail of exactly how big the
>> prospective LIFO freelists are - I just want to see the situation
>> resolved with maximum utilisation of memory for minimum performance
>> penalty. To t
On 2008-02-13 12:56, Andrew MacIntyre wrote:
> I'm not that interested in debating the detail of exactly how big the
> prospective LIFO freelists are - I just want to see the situation
> resolved with maximum utilisation of memory for minimum performance
> penalty. To that end, +1 from me for acc
Christian Heimes wrote:
> Andrew MacIntyre wrote:
>> I tested the updated patch you added to issue 2039. With the int
>> freelist set to 500 and the float freelist set to 100, its about the same
>> as the no-freelist version for my tests, but PyBench shows the simple
>> float arithmetic to be abo
On 2008-02-13 08:02, Andrew MacIntyre wrote:
> Christian Heimes wrote:
>> Andrew MacIntyre wrote:
>>> I tried a LIFO stack implementation (though I won't claim to have done it
>>> well), and found it slightly slower than no freelist at all. The
>>> advantage of such an approach is that the known si
Andrew MacIntyre wrote:
> I seem to recall Tim Peters paying a lot of attention to cache effects
> when he went over the PyMalloc code before the 2.3 release, which would
> contribute to its performance.
Well, I guess we have to ask Uncle Tim on this matter and ask for his
opinion. I've no experie
I've done my own profiling with a different script. I was able to verify
Andrews results. The pymalloc approach makes int creation slightly
slower and has almost no effect on floats.
The creation of 1000 times a list of 1000 ints (range(1000)) is about
20msec slower. It almost doubles the time for
Andrew MacIntyre wrote:
> I tried a LIFO stack implementation (though I won't claim to have done it
> well), and found it slightly slower than no freelist at all. The
> advantage of such an approach is that the known size of the stack makes
> deallocating excess objects easy (and thus no need for
>
> Well, yes, it doesn't run out of memory, but if pymalloc needs
> to allocate lots of objects of the same size, then performance
> degrades due to the management overhead involved for checking
> the free pools as well as creating new arenas as needed.
>
> To reduce this overhead, it may be a good
Christian Heimes wrote:
> Andrew MacIntyre wrote:
>> For the int case, that patch is slower than no free list at all. For the
>> float case its very slightly faster (55.3s vs 55.5s) than no free list at
>> all. Slower than the trunk code for both cases. Did you run the test
>> scripts on your ow
Martin v. Löwis wrote:
> You mean, as the digits? You would have to rewrite the entire long
> datatype, which is a tedious exercise. Plus, I believe you would have
> to make some operations 64-bit on all systems: when you multiply
> digits today, the product will be 32-bit. If you extend the digits
> Given the background information Python's long implementation could
> probably optimized. In 3.0 a long has 3 4-byte members (ref count,
> ob_size and *ob_type) plus one to many unsigned shorts (2 bytes each) to
> hold the value. If pymalloc aligns the objects at 8 byte address
> boundaries would
M.-A. Lemburg wrote:
> As you can see, Integers and floats fall into the same pymalloc size
> class. What's strange in Andrew's result is that both integers
> and floats use the same free list technique and fall into the same
> pymalloc size class, yet the results are different.
My take is not th
Tim Peters wrote:
> pymalloc ensures 8-byte alignment. This is one plausible reason to
> keep the current int free list: an int object struct holds 3 4-byte
> members on most boxes (type pointer, refcount, and the int's value),
> and the int freelist code uses exactly 12 bytes for each on most
>
On 2008-02-08 19:28, Christian Heimes wrote:
>> In addition to the pure performance aspect, there is the issue of memory
>> utilisation. The current trunk code running the int test case in my
>> original post peaks at 151MB according to top on my FreeBSD box, dropping
>> back to about 62MB after t
On 2008-02-08 12:23, M.-A. Lemburg wrote:
> On 2008-02-08 08:21, Martin v. Löwis wrote:
>>> One of the hopes of having a custom allocator for Python was to be
>>> able to get rid off all free lists. For some reason that never happened.
>>> Not sure why. People were probably too busy with adding new
Tim Peters wrote:
> pymalloc ensures 8-byte alignment. This is one plausible reason to
> keep the current int free list: an int object struct holds 3 4-byte
> members on most boxes (type pointer, refcount, and the int's value),
> and the int freelist code uses exactly 12 bytes for each on most
>
Neal Norwitz wrote:
> It's not just size. Architectures may require data aligned on 4, 8,
> or 16 addresses for optimal performance depending on data type. IIRC,
> malloc aligns by 8 (not sure if that was a particular arch or very
> common). I don't know if pymalloc handles alignment.
Yes, pyma
[Neal Norwitz]
> It's not just size. Architectures may require data aligned on 4, 8,
> or 16 addresses for optimal performance depending on data type. IIRC,
> malloc aligns by 8 (not sure if that was a particular arch or very
> common).
Just very common. Because malloc has no idea what the poin
On Feb 8, 2008 10:54 AM, Christian Heimes <[EMAIL PROTECTED]> wrote:
> Andrew MacIntyre wrote:
> > However, my tests do show that something is funny with the current
> > freelist implementation for floats on at least 2 platforms, and that
> > doing without that sort of optimisation for float object
Andrew MacIntyre wrote:
> However, my tests do show that something is funny with the current
> freelist implementation for floats on at least 2 platforms, and that
> doing without that sort of optimisation for float objects would likely
> not be a hardship with PyMalloc.
float objects are slightly
Andrew MacIntyre wrote:
> For the int case, that patch is slower than no free list at all. For the
> float case its very slightly faster (55.3s vs 55.5s) than no free list at
> all. Slower than the trunk code for both cases. Did you run the test
> scripts on your own machine?
I've used a simple
M.-A. Lemburg wrote:
> On 2008-02-07 14:09, Andrew MacIntyre wrote:
>> Probably in response to the same stimulus as Christian it occurred to me
>> that the freelist approach had been adopted long before PyMalloc was
>> enabled as standard (in 2.3), and that much of the performance gains
>> between
On 2008-02-08 08:21, Martin v. Löwis wrote:
>> One of the hopes of having a custom allocator for Python was to be
>> able to get rid off all free lists. For some reason that never happened.
>> Not sure why. People were probably too busy with adding new
>> features to the language at the time ;-)
>
> One of the hopes of having a custom allocator for Python was to be
> able to get rid off all free lists. For some reason that never happened.
> Not sure why. People were probably too busy with adding new
> features to the language at the time ;-)
Probably not. It's more that the free lists still
On Feb 7, 2008 5:09 AM, Andrew MacIntyre <[EMAIL PROTECTED]> wrote:
>
> So I've been testing with the freelists ripped out and ints and floats
> reverted to fairly standard PyObject_New allocation (retaining the small
> int interning) and thus relying on PyMalloc to do any compaction.
>
> The resul
On 2008-02-07 14:09, Andrew MacIntyre wrote:
> Probably in response to the same stimulus as Christian it occurred to me
> that the freelist approach had been adopted long before PyMalloc was
> enabled as standard (in 2.3), and that much of the performance gains
> between 2.2 and 2.3 were in fact du
Andrew MacIntyre wrote:
> So I've been testing with the freelists ripped out and ints and floats
> reverted to fairly standard PyObject_New allocation (retaining the small
> int interning) and thus relying on PyMalloc to do any compaction.
Nice! What do you mean with int interning? Are you talking
I note that in r60567 Christian checked in support for compacting the
int and float freelists.
I have no problems with the implementation, but would like to note
that I have been experimenting with an alternate approach which doesn't
require the addition of a knob to initiate the compacting.
Prob
39 matches
Mail list logo
