Re: [Python-Dev] Problems with the Python Memory Manager

2005-11-24 Thread Martin v. Löwis
Travis Oliphant wrote: So, I now believe that his code (plus the array scalar extension type) was actually exposing a real bug in the memory manager itself. In theory, the Python memory manager should have been able to re-use the memory for the array-scalar instances because they are

Re: [Python-Dev] Problems with the Python Memory Manager

2005-11-24 Thread Fredrik Lundh
Martin v. Löwis wrote: One way (I think the only way) this could happen if: - the objects being allocated are all smaller than 256 bytes - when allocating new objects, the requested size was different from any other size previously deallocated. So if you first allocate 1,000,000 objects

Re: [Python-Dev] urlparse brokenness

2005-11-24 Thread Donovan Baarda
On Tue, 2005-11-22 at 23:04 -0600, Paul Jimenez wrote: It is my assertion that urlparse is currently broken. Specifically, I think that urlparse breaks an abstraction boundary with ill effect. In writing a mailclient, I wished to allow my users to specify their imap server as a url, such

Re: [Python-Dev] Problems with the Python Memory Manager

2005-11-24 Thread Armin Rigo
Hi, On Thu, Nov 24, 2005 at 01:59:57AM -0800, Robert Kern wrote: You can get the version of scipy_core just before the fix that Travis applied: Now we can start debugging :-) http://projects.scipy.org/scipy/scipy_core/changeset/1490 This changeset alone fixes the small example you

Re: [Python-Dev] Problems with the Python Memory Manager

2005-11-24 Thread Armin Rigo
Hi, Ok, here is the reason for the leak... There is in scipy a type called 'int32_arrtype' which inherits from both another scipy type called 'signedinteger_arrtype', and from 'int'. Obscure! This is not 100% officially allowed: you are inheriting from two C types. You're living dangerously!

Re: [Python-Dev] (no subject)

2005-11-24 Thread Donovan Baarda
On Thu, 2005-11-24 at 14:11 +, Duncan Grisby wrote: Hi, I posted this to comp.lang.python, but got no response, so I thought I would consult the wise people here... I have encountered a problem with the re module. I have a multi-threaded program that does lots of regular expression

[Python-Dev] Re: Regular expressions

2005-11-24 Thread Duncan Grisby
On Thursday 24 November, Donovan Baarda wrote: I don't know if this will help, but in my experience compiling re's often takes longer than matching them... are you sure that it's the match and not a compile that is taking a long time? Are you using pre-compiled re's or are you dynamically

Re: [Python-Dev] (no subject)

2005-11-24 Thread Fredrik Lundh
Donovan Baarda wrote: I don't know if this will help, but in my experience compiling re's often takes longer than matching them... are you sure that it's the match and not a compile that is taking a long time? Are you using pre-compiled re's or are you dynamically generating strings and using

Re: [Python-Dev] Problems with the Python Memory Manager

2005-11-24 Thread Travis E. Oliphant
Armin Rigo wrote: Hi, Ok, here is the reason for the leak... There is in scipy a type called 'int32_arrtype' which inherits from both another scipy type called 'signedinteger_arrtype', and from 'int'. Obscure! This is not 100% officially allowed: you are inheriting from two C types.

[Python-Dev] Problems with mro for dual inheritance in C [Was: Problems with the Python Memory Manager]

2005-11-24 Thread Travis E. Oliphant
Armin Rigo wrote: Hi, Ok, here is the reason for the leak... There is in scipy a type called 'int32_arrtype' which inherits from both another scipy type called 'signedinteger_arrtype', and from 'int'. Obscure! This is not 100% officially allowed: you are inheriting from two C types.

[Python-Dev] registering unicode codecs

2005-11-24 Thread Neal Norwitz
While running regrtest with -R to find reference leaks I found a usage issue. When a codec is registered it is stored in the interpreter state and cannot be removed. Since it is stored as a list, if you repeated add the same search function, you will get duplicates in the list and they can't be

Re: [Python-Dev] registering unicode codecs

2005-11-24 Thread M.-A. Lemburg
Neal Norwitz wrote: While running regrtest with -R to find reference leaks I found a usage issue. When a codec is registered it is stored in the interpreter state and cannot be removed. Since it is stored as a list, if you repeated add the same search function, you will get duplicates in the

Re: [Python-Dev] registering unicode codecs

2005-11-24 Thread Neal Norwitz
On 11/24/05, M.-A. Lemburg [EMAIL PROTECTED] wrote: Should users have access to the search path (through a codecs.unregister())? Maybe, but why would you want to unregister a search function ? If so, should it search from the end of the list to the beginning to remove an item? That

Re: [Python-Dev] registering unicode codecs

2005-11-24 Thread M.-A. Lemburg
Neal Norwitz wrote: On 11/24/05, M.-A. Lemburg [EMAIL PROTECTED] wrote: Should users have access to the search path (through a codecs.unregister())? Maybe, but why would you want to unregister a search function ? If so, should it search from the end of the list to the beginning to remove an

Re: [Python-Dev] Problems with mro for dual inheritance in C [Was: Problems with the Python Memory Manager]

2005-11-24 Thread Armin Rigo
Hi Travis, On Thu, Nov 24, 2005 at 10:17:43AM -0700, Travis E. Oliphant wrote: Why doesn't the int32 type inherit its tp_free from the early types first? In your case I suspect that the tp_free is inherited from the tp_base which is probably 'int'. I don't see how to fix typeobject.c,

[Python-Dev] reference leaks

2005-11-24 Thread Neal Norwitz
There are still a few reference leaks I've been able to identify. I didn't see an obvious solution to these (well, I saw one obvious solution which crashed, so obviously I was wrong). When running regrtest with -R here are the ref leaks reported: test_codeccallbacks leaked [2, 2, 2, 2]

Re: [Python-Dev] Problems with the Python Memory Manager

2005-11-24 Thread Travis Oliphant
Martin v. Löwis wrote: Travis Oliphant wrote: So, I now believe that his code (plus the array scalar extension type) was actually exposing a real bug in the memory manager itself. In theory, the Python memory manager should have been able to re-use the memory for the array-scalar

Re: [Python-Dev] Problems with the Python Memory Manager

2005-11-24 Thread Travis Oliphant
Martin v. Löwis wrote: Travis Oliphant wrote: As verified by removing usage of the Python PyObject_MALLOC function, it was the Python memory manager that was performing poorly. Even though the array-scalar objects were deleted, the memory manager would not re-use their memory for later