Re: [Python-3000] Free list for small longs

2008-02-06 Thread Martin v. Löwis
Not from a real world application, no. If you run the program below with And now for the program below... Regards, Martin import gc,sys,time gc.set_debug(gc.DEBUG_STATS) t=time.time() l = [] l.append(l) for i in range(10): l.append([]) del l print >>sys.stderr,"Done looping: final gen

Re: [Python-3000] Free list for small longs

2008-02-06 Thread Martin v. Löwis
>> Both. Let (a,b,c) equal gc.get_threshold(), then a full gc happens >> after a*b*c object allocations (roughly; early object deallocations >> are deducted from that number). Currently, this product is 7. > > I see. 70k is a large number. Do you have some real world experience how > often a a

Re: [Python-3000] Free list for small longs

2008-02-06 Thread Christian Heimes
Martin v. Löwis wrote: > Both. Let (a,b,c) equal gc.get_threshold(), then a full gc happens > after a*b*c object allocations (roughly; early object deallocations > are deducted from that number). Currently, this product is 7. I see. 70k is a large number. Do you have some real world experience

Re: [Python-3000] Free list for small longs

2008-02-06 Thread Martin v. Löwis
Christian Heimes wrote: > Martin v. Löwis wrote: >> I wonder whether such a thing should run as a side effect of a full >> garbage collection, also. GC has a chance to free up arenas for good, >> and allocated-but-unused blocks on arenas prevent Python from returning >> them to the OS. > > Under

Re: [Python-3000] Free list for small longs

2008-02-06 Thread Christian Heimes
Martin v. Löwis wrote: > I wonder whether such a thing should run as a side effect of a full > garbage collection, also. GC has a chance to free up arenas for good, > and allocated-but-unused blocks on arenas prevent Python from returning > them to the OS. Under which condition does a full garbag

Re: [Python-3000] Free list for small longs

2008-02-06 Thread Martin v. Löwis
> My new > sys._compact_freelists() deallocates the float and int blocks which do > not contain a referenced object. I wonder whether such a thing should run as a side effect of a full garbage collection, also. GC has a chance to free up arenas for good, and allocated-but-unused blocks on arenas

Re: [Python-3000] Free list for small longs

2008-02-05 Thread Christian Heimes
Brett Cannon wrote: > Well, I am a little leery of another free list. I personally don't > love the fact that there are various caches and free lists laying > about in the code without a central place to turn them off or clear > them (or at least list them in a comment somewhere). And the problem >

Re: [Python-3000] Free list for small longs

2008-02-05 Thread Brett Cannon
On Feb 5, 2008 2:30 PM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > I think Christian means single digit integers. But I thought we > > already did this for positive numbers? So is the proposal to also > > cover negative numbers? > > We don't have a free list, but a cache for them, and only fo

Re: [Python-3000] Free list for small longs

2008-02-05 Thread Martin v. Löwis
> I think Christian means single digit integers. But I thought we > already did this for positive numbers? So is the proposal to also > cover negative numbers? We don't have a free list, but a cache for them, and only for numbers between -5 and +256. Christian proposes a freelist, similar to the s

Re: [Python-3000] Free list for small longs

2008-02-05 Thread Christian Heimes
Martin v. Löwis wrote: > I'm puzzled as to how you arrive at this upper bound. You can surely > have more than 2**16 integer objects whose abs values are all > below 2**15, no? Oh ... I see. You are right and I did a mistake by confusing the amount of objects with the range. I'd better add an uppe

Re: [Python-3000] Free list for small longs

2008-02-05 Thread Martin v. Löwis
> Since the free list is limited to small longs, it will consume less than > 1,5 MB on a 64bit OS and less than 900kb on a 32bit OS in a worst case > scenario (2 * (1<<15) ~ 65k objects with a size of 14 / 22 byte each). I'm puzzled as to how you arrive at this upper bound. You can surely have mor

Re: [Python-3000] Free list for small longs

2008-02-05 Thread Christian Heimes
Brett Cannon wrote: > I think Christian means single digit integers. But I thought we > already did this for positive numbers? So is the proposal to also > cover negative numbers? Ne, we don't use a free list for longs. In the 2.x series floats and ints are allocated in blocks of 1000 objects. The

Re: [Python-3000] Free list for small longs

2008-02-05 Thread Christian Heimes
[EMAIL PROTECTED] wrote: > I'm not sure what you mean by "size of 1 or -1". Do you mean you only keep > the numbers 1 and -1 on the free list? It's not obvious to me what a size > of -1 is. Do you mean positive and negative numbers which fit in one byte > or one long word? I should have explain

Re: [Python-3000] Free list for small longs

2008-02-05 Thread Brett Cannon
On Feb 5, 2008 12:13 PM, <[EMAIL PROTECTED]> wrote: > > Christian> I've implemented a free list for small long objects with a > Christian> size of 1 or -1. > > I'm not sure what you mean by "size of 1 or -1". Do you mean you only keep > the numbers 1 and -1 on the free list? It's not obv

Re: [Python-3000] Free list for small longs

2008-02-05 Thread skip
Christian> I've implemented a free list for small long objects with a Christian> size of 1 or -1. I'm not sure what you mean by "size of 1 or -1". Do you mean you only keep the numbers 1 and -1 on the free list? It's not obvious to me what a size of -1 is. Do you mean positive and nega

[Python-3000] Free list for small longs

2008-02-05 Thread Christian Heimes
I've implemented a free list for small long objects with a size of 1 or -1. I wanted to test how large the malloc overhead is. The result was astonishing. The free list quadrupled the speed of a simple test: $ ./python -m timeit "for i in range(100): list(range(1000))" Without patch: 10 loops, be