Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-06 Thread Amaury Forgeot d'Arc
Le 6 octobre 2011 10:09, Charles-François Natali a écrit : >> But under certain circumstances (if a large block is requested), the >> allocator uses mmap(), no? > > That's right, if the block requested is bigger than mmap_threshold > (256K by default with glibc, forgetting the sliding window algor

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-06 Thread Charles-François Natali
>> > > > That's not a given. Depending on the memory allocator, a copy can be >> > > > avoided. That's why the "str += str" hack is much more efficient under >> > > > Linux than Windows, AFAIK. >> > > >> > > Even Linux will have to copy a block on realloc in certain cases, no? >> > >> > Probably so

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-05 Thread Antoine Pitrou
On Wed, 5 Oct 2011 09:38:10 -0700 Toshio Kuratomi wrote: > On Wed, Oct 05, 2011 at 06:14:08PM +0200, Antoine Pitrou wrote: > > Le mercredi 05 octobre 2011 à 18:12 +0200, "Martin v. Löwis" a écrit : > > > >> Not sure what you are using it for. If you need to extend the buffer > > > >> in case it is

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-05 Thread Toshio Kuratomi
On Wed, Oct 05, 2011 at 06:14:08PM +0200, Antoine Pitrou wrote: > Le mercredi 05 octobre 2011 à 18:12 +0200, "Martin v. Löwis" a écrit : > > >> Not sure what you are using it for. If you need to extend the buffer > > >> in case it is too small, there is absolutely no way this could work > > >> with

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-05 Thread Antoine Pitrou
Le mercredi 05 octobre 2011 à 18:12 +0200, "Martin v. Löwis" a écrit : > >> Not sure what you are using it for. If you need to extend the buffer > >> in case it is too small, there is absolutely no way this could work > >> without copies in the general case because of how computers use > >> address

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-05 Thread Martin v. Löwis
Not sure what you are using it for. If you need to extend the buffer in case it is too small, there is absolutely no way this could work without copies in the general case because of how computers use address space. Even _PyBytes_Resize will copy the data. That's not a given. Depending on the me

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Antoine Pitrou
On Tue, 04 Oct 2011 21:33:34 +0200 "Martin v. Löwis" wrote: > Am 04.10.11 21:06, schrieb Amaury Forgeot d'Arc: > > 2011/10/4 "Martin v. Löwis": > >> > >>> - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API > >>>section. > >> > >> ??? Are you proposing to add _PyBytes_Resize to t

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Nick Coghlan
On Tue, Oct 4, 2011 at 3:12 PM, Antoine Pitrou wrote: > Uh, no, it depends what you're doing. There's no reason not to allow > people to resize a bytes object which they've just allocated and is > still private to their code. That's the whole reason why > _PyBytes_Resize() exists, and the use case

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Martin v. Löwis
Am 04.10.11 21:06, schrieb Amaury Forgeot d'Arc: 2011/10/4 "Martin v. Löwis": - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API section. ??? Are you proposing to add _PyBytes_Resize to the Py_LIMITED_API set of functions? It's not even an API function in the first place (i

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Antoine Pitrou
On Tue, 4 Oct 2011 13:05:58 -0400 Nick Coghlan wrote: > > > - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API > >  section. > > No, that's not valid. Bytes are officially immutable - mutating them > when the reference count is only 1 is a private for a reason. The > correct way t

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Amaury Forgeot d'Arc
2011/10/4 "Martin v. Löwis" : > >> - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API >>   section. > > ??? Are you proposing to add _PyBytes_Resize to the Py_LIMITED_API > set of functions? It's not even an API function in the first place > (it starts with an underscore), so how can

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Martin v. Löwis
- Py_LIMITED_API is incompatible with --with-pydebug, and compilation stops. I skipped the check to continue. That seems like an odd (and undesirable) restriction. It's deliberate, though. If different Python versions are going to expose the same ABI, it seems strange of debug and release

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Martin v. Löwis
Amaury: thanks for your experiment and your report. - I replaced PyBytes_GET_SIZE() with Py_SIZE(), which is OK, and PyBytes_AS_STRING() with PyBytes_AsString(), which may have a slight performance impact. That's the whole point of the stable ABI: AS_STRING assumes that there is an ob_sval fie

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Nick Coghlan
On Tue, Oct 4, 2011 at 1:05 PM, Nick Coghlan wrote: > It's probably not a bad idea, otherwise we may compilation without > realising it. s/may/may break/ Actually testing the ABI stability would be much harder - somehow building an extension module against 3.2 with the limited API then testing i

Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Nick Coghlan
(My comments are based on the assumption Amaury started with http://hg.python.org/sandbox/nvawda/file/09d984063fca/Modules/_lzmamodule.c) On Tue, Oct 4, 2011 at 12:18 PM, Amaury Forgeot d'Arc wrote: > - Py_LIMITED_API is incompatible with --with-pydebug, and compilation stops. >  I skipped the ch

[Python-Dev] Using PEP384 Stable ABI for the lzma extension module

2011-10-04 Thread Amaury Forgeot d'Arc
Hi, Has someone already tried to *really* use Py_LIMITED_API for some "serious" extension module? I wanted to give it a try for the _lzma module (see issue 6715) because liblzma does not compile with Microsoft compilers; an alternative could be to use mingw to (pre)build _lzma.pyd, which would lin