Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
Le 6 octobre 2011 10:09, Charles-François Natali a écrit : >> But under certain circumstances (if a large block is requested), the >> allocator uses mmap(), no? > > That's right, if the block requested is bigger than mmap_threshold > (256K by default with glibc, forgetting the sliding window algorithm): > I'm not sure of what percentage of strings/buffers are concerned in a > "typical" program. Most usages of _PyBytes_Resize() are in compression libraries. 256K payloads are not rare in this area. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
>> > > > That's not a given. Depending on the memory allocator, a copy can be >> > > > avoided. That's why the "str += str" hack is much more efficient under >> > > > Linux than Windows, AFAIK. >> > > >> > > Even Linux will have to copy a block on realloc in certain cases, no? >> > >> > Probably so. How often is totally unknown to me :) >> > >> http://www.gnu.org/software/libc/manual/html_node/Changing-Block-Size.html >> >> It depends on whether there's enough free memory after the buffer you >> currently have allocated. I suppose that this becomes a question of what >> people consider "the general case" :-) > > But under certain circumstances (if a large block is requested), the > allocator uses mmap(), no? That's right, if the block requested is bigger than mmap_threshold (256K by default with glibc, forgetting the sliding window algorithm): I'm not sure of what percentage of strings/buffers are concerned in a "typical" program. > In which case mremap() should allow to resize without copying anything. Yes, there's no copying. Note however that it doesn't come for free, the kernel will still zero-fill the pages before handling them to user-space. It is still way faster than on, let's say, Solaris. cf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
On Wed, 5 Oct 2011 09:38:10 -0700 Toshio Kuratomi wrote: > On Wed, Oct 05, 2011 at 06:14:08PM +0200, Antoine Pitrou wrote: > > Le mercredi 05 octobre 2011 à 18:12 +0200, "Martin v. Löwis" a écrit : > > > >> Not sure what you are using it for. If you need to extend the buffer > > > >> in case it is too small, there is absolutely no way this could work > > > >> without copies in the general case because of how computers use > > > >> address space. Even _PyBytes_Resize will copy the data. > > > > > > > > That's not a given. Depending on the memory allocator, a copy can be > > > > avoided. That's why the "str += str" hack is much more efficient under > > > > Linux than Windows, AFAIK. > > > > > > Even Linux will have to copy a block on realloc in certain cases, no? > > > > Probably so. How often is totally unknown to me :) > > > http://www.gnu.org/software/libc/manual/html_node/Changing-Block-Size.html > > It depends on whether there's enough free memory after the buffer you > currently have allocated. I suppose that this becomes a question of what > people consider "the general case" :-) But under certain circumstances (if a large block is requested), the allocator uses mmap(), no? In which case mremap() should allow to resize without copying anything. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
On Wed, Oct 05, 2011 at 06:14:08PM +0200, Antoine Pitrou wrote: > Le mercredi 05 octobre 2011 à 18:12 +0200, "Martin v. Löwis" a écrit : > > >> Not sure what you are using it for. If you need to extend the buffer > > >> in case it is too small, there is absolutely no way this could work > > >> without copies in the general case because of how computers use > > >> address space. Even _PyBytes_Resize will copy the data. > > > > > > That's not a given. Depending on the memory allocator, a copy can be > > > avoided. That's why the "str += str" hack is much more efficient under > > > Linux than Windows, AFAIK. > > > > Even Linux will have to copy a block on realloc in certain cases, no? > > Probably so. How often is totally unknown to me :) > http://www.gnu.org/software/libc/manual/html_node/Changing-Block-Size.html It depends on whether there's enough free memory after the buffer you currently have allocated. I suppose that this becomes a question of what people consider "the general case" :-) -Toshio pgpCHlc9jDncJ.pgp Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
Le mercredi 05 octobre 2011 à 18:12 +0200, "Martin v. Löwis" a écrit : > >> Not sure what you are using it for. If you need to extend the buffer > >> in case it is too small, there is absolutely no way this could work > >> without copies in the general case because of how computers use > >> address space. Even _PyBytes_Resize will copy the data. > > > > That's not a given. Depending on the memory allocator, a copy can be > > avoided. That's why the "str += str" hack is much more efficient under > > Linux than Windows, AFAIK. > > Even Linux will have to copy a block on realloc in certain cases, no? Probably so. How often is totally unknown to me :) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
Not sure what you are using it for. If you need to extend the buffer in case it is too small, there is absolutely no way this could work without copies in the general case because of how computers use address space. Even _PyBytes_Resize will copy the data. That's not a given. Depending on the memory allocator, a copy can be avoided. That's why the "str += str" hack is much more efficient under Linux than Windows, AFAIK. Even Linux will have to copy a block on realloc in certain cases, no? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
On Tue, 04 Oct 2011 21:33:34 +0200 "Martin v. Löwis" wrote: > Am 04.10.11 21:06, schrieb Amaury Forgeot d'Arc: > > 2011/10/4 "Martin v. Löwis": > >> > >>> - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API > >>>section. > >> > >> ??? Are you proposing to add _PyBytes_Resize to the Py_LIMITED_API > >> set of functions? It's not even an API function in the first place > >> (it starts with an underscore), so how can it be a limited API function? > > > > It's not a proposal of any kind; it's just the workaround I used to compile > > and test. > > OTOH, it seems that many modules already use this function. Is there > > another method that does not need to copy data? > > Not sure what you are using it for. If you need to extend the buffer > in case it is too small, there is absolutely no way this could work > without copies in the general case because of how computers use > address space. Even _PyBytes_Resize will copy the data. That's not a given. Depending on the memory allocator, a copy can be avoided. That's why the "str += str" hack is much more efficient under Linux than Windows, AFAIK. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
On Tue, Oct 4, 2011 at 3:12 PM, Antoine Pitrou wrote: > Uh, no, it depends what you're doing. There's no reason not to allow > people to resize a bytes object which they've just allocated and is > still private to their code. That's the whole reason why > _PyBytes_Resize() exists, and the use case is not exotic. > > Telling people to "first create a bytearray and then create a bytes > object from that when you're finished" would be a shame. If developers want to use private CPython functions, then they can't use the stable API - the whole point of having private APIs is that we don't even promise *source* compatibility for those, let alone binary compatibility. If they want the stability guarantee, then they have to eschew hacks that rely on implementation details (like the ability to resize "immutable" objects). That seems pretty reasonable to me. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
Am 04.10.11 21:06, schrieb Amaury Forgeot d'Arc: 2011/10/4 "Martin v. Löwis": - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API section. ??? Are you proposing to add _PyBytes_Resize to the Py_LIMITED_API set of functions? It's not even an API function in the first place (it starts with an underscore), so how can it be a limited API function? It's not a proposal of any kind; it's just the workaround I used to compile and test. OTOH, it seems that many modules already use this function. Is there another method that does not need to copy data? Not sure what you are using it for. If you need to extend the buffer in case it is too small, there is absolutely no way this could work without copies in the general case because of how computers use address space. Even _PyBytes_Resize will copy the data. The only way to avoid copying is to run over the input twice: once to determine how large the output will have to be, and then another time to actually produce the output. Whether or not that's actually faster than copying the output depends on how much work this size computation requires. It would be nice if LZMA had "output size" information embedded in it, but it may not. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
On Tue, 4 Oct 2011 13:05:58 -0400 Nick Coghlan wrote: > > > - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API > > section. > > No, that's not valid. Bytes are officially immutable - mutating them > when the reference count is only 1 is a private for a reason. The > correct way to do this without relying on that implementation detail > is to use a byte array instead. Uh, no, it depends what you're doing. There's no reason not to allow people to resize a bytes object which they've just allocated and is still private to their code. That's the whole reason why _PyBytes_Resize() exists, and the use case is not exotic. Telling people to "first create a bytearray and then create a bytes object from that when you're finished" would be a shame. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
2011/10/4 "Martin v. Löwis" : > >> - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API >> section. > > ??? Are you proposing to add _PyBytes_Resize to the Py_LIMITED_API > set of functions? It's not even an API function in the first place > (it starts with an underscore), so how can it be a limited API function? It's not a proposal of any kind; it's just the workaround I used to compile and test. OTOH, it seems that many modules already use this function. Is there another method that does not need to copy data? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
- Py_LIMITED_API is incompatible with --with-pydebug, and compilation stops. I skipped the check to continue. That seems like an odd (and undesirable) restriction. It's deliberate, though. If different Python versions are going to expose the same ABI, it seems strange of debug and release versions can't do the same. You'll have to specify a lot of details what precisely constitutes a debug build, and what fields precisely belong to it. Nobody volunteered to specify what it should do, so I excluded it. It's also not the objective of the PEP to support loading debug-built extensions in alternative interpreter versions. I fail to see why this is undesirable, also. It's very easy to write an extension module that only uses the limited API, and still builds fine in a debug build: just don't define Py_LIMITED_API when compiling for debug mode. The stable ABI probably needs a better solution for tp_new slots invoking tp_alloc and tp_dealloc slots invoking tp_free. In fact, a systematic review of the slot documentation is probably needed, pointing out the stable ABI alternatives to all of the recommended "cross slot" invocations (and creating them if they don't already exist). Doing so would probably be better than my proposed approach of just provding a generic access function that reads a slot as a void* from a type object. What do you think about using the stable ABI even in shipped extensions? It's probably not a bad idea, otherwise we may compilation without realising it. This is especially so for extension modules that *don't* need access to any of the interpreter internals. Missing a word in the first sentence? There is the xxlimited module that is there to test that it keeps compiling under the limited API. I'll review all API additions before the next release, and will exclude a) anything that shouldn't be used by extension modules at all. There was a tradition of exposing all helper function, but I think this tradition needs to stop. Instead, adding to the API should be conservative, and only add what is positively useful to extension modules. b) anything that is not sufficiently stable from the limited API (in particular stuff that refers to new structures). The DLL .def file for Windows will make sure that nothing gets added unintentionally to the stable ABI, unfortunately, there is no easy technique for Unix achieving the same. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
Amaury: thanks for your experiment and your report. - I replaced PyBytes_GET_SIZE() with Py_SIZE(), which is OK, and PyBytes_AS_STRING() with PyBytes_AsString(), which may have a slight performance impact. That's the whole point of the stable ABI: AS_STRING assumes that there is an ob_sval field at a certain offset of the bytes object, which may not be there in a future release. PyBytes_AsString is indeed slower, but also more future-proof. - I replaced Py_TYPE(self)->tp_free((PyObject *)self); with PyObject_Del(self), I hope this is the same thing (for a non-GC object) If a subtype of self.__class__ would override tp_free, it wouldn't be the same anymore. I guess the API needs a way to read a slot from a type object. - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API section. ??? Are you proposing to add _PyBytes_Resize to the Py_LIMITED_API set of functions? It's not even an API function in the first place (it starts with an underscore), so how can it be a limited API function? I think this whole notion of resizing immutable objects in the Python C API is flawed. If you can't know how large a buffer is in advance, first allocate a regular memory block, and then copy it into an object when done. - For the "y*" argument spec, the Py_buffer structure is required (only for two fields: buf and len), as well as PyBuffer_Release() Yes, this was a debate in the API PEP. I originally had the buffer API in the stable ABI, but was then advised to remove it, as it may not be that stable at all. I'll start a thread about extending the stable ABI soon; if people now want to reconsider, it would be possible. However, taking something out of the stable ABI is not possible, so if we decide the buffer API is stable, the structure is locked until Python 4. - PyType_FromSpec() does not call PyType_Ready(), which caused crashes in __new__. Oops :-) Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
On Tue, Oct 4, 2011 at 1:05 PM, Nick Coghlan wrote: > It's probably not a bad idea, otherwise we may compilation without > realising it. s/may/may break/ Actually testing the ABI stability would be much harder - somehow building an extension module against 3.2 with the limited API then testing it against a freshly built 3.3. Perhaps we could manage something like that by building against a system installation of Python 3.2 on builders that have it available. All in all, I think PEP 384 laid the foundations, but there's still plenty of work to be done in the documentation and testing space (and perhaps a few API additions) before the majority of extensions can realistically switch to the stable ABI. A bit of "eating our own dogfood" in the extension modules we ship may be a good place to start (especially new ones that are added). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
(My comments are based on the assumption Amaury started with http://hg.python.org/sandbox/nvawda/file/09d984063fca/Modules/_lzmamodule.c) On Tue, Oct 4, 2011 at 12:18 PM, Amaury Forgeot d'Arc wrote: > - Py_LIMITED_API is incompatible with --with-pydebug, and compilation stops. > I skipped the check to continue. That seems like an odd (and undesirable) restriction. If different Python versions are going to expose the same ABI, it seems strange of debug and release versions can't do the same. > - I replaced PyBytes_GET_SIZE() with Py_SIZE(), which is OK, > and PyBytes_AS_STRING() with PyBytes_AsString(), which may > have a slight performance impact. Yes, the price of using the stable ABI is that performance tricks that depend on exact memory layouts are no longer available. > - I replaced > Py_TYPE(self)->tp_free((PyObject *)self); > with PyObject_Del(self), I hope this is the same thing > (for a non-GC object) That looks right in this particular case, but problematic in general. The stable ABI probably needs a better solution for tp_new slots invoking tp_alloc and tp_dealloc slots invoking tp_free. In fact, a systematic review of the slot documentation is probably needed, pointing out the stable ABI alternatives to all of the recommended "cross slot" invocations (and creating them if they don't already exist). > - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API > section. No, that's not valid. Bytes are officially immutable - mutating them when the reference count is only 1 is a private for a reason. The correct way to do this without relying on that implementation detail is to use a byte array instead. > - For the "y*" argument spec, the Py_buffer structure is required > (only for two fields: buf and len), as well as PyBuffer_Release() Yeah, PEP 3118 support will eventually appear in the stable ABI, but we need to fix it first (see issue 10181). > - PyType_FromSpec() does not call PyType_Ready(), which caused > crashes in __new__. That sounds like it may just be a bug. Although looking at the C API docs, PEP 384 documentation appears to be basically non-existent... > Now the module seems to work correctly and passes tests... at least on > Linux in a standard environment. I will do other tests on Windows. > > What do you think about using the stable ABI even in shipped extensions? It's probably not a bad idea, otherwise we may compilation without realising it. This is especially so for extension modules that *don't* need access to any of the interpreter internals. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Using PEP384 Stable ABI for the lzma extension module
Hi, Has someone already tried to *really* use Py_LIMITED_API for some "serious" extension module? I wanted to give it a try for the _lzma module (see issue 6715) because liblzma does not compile with Microsoft compilers; an alternative could be to use mingw to (pre)build _lzma.pyd, which would link with a static liblzma.a also compiled with mingw. Mixing compilers in a Python process is one of the reasons of PEP384, so I added #define Py_LIMITED_API on top of the module, and "fixed" the issues one by one: - Py_LIMITED_API is incompatible with --with-pydebug, and compilation stops. I skipped the check to continue. - I replaced PyBytes_GET_SIZE() with Py_SIZE(), which is OK, and PyBytes_AS_STRING() with PyBytes_AsString(), which may have a slight performance impact. - I replaced Py_TYPE(self)->tp_free((PyObject *)self); with PyObject_Del(self), I hope this is the same thing (for a non-GC object) - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API section. - For the "y*" argument spec, the Py_buffer structure is required (only for two fields: buf and len), as well as PyBuffer_Release() - PyType_FromSpec() does not call PyType_Ready(), which caused crashes in __new__. Now the module seems to work correctly and passes tests... at least on Linux in a standard environment. I will do other tests on Windows. What do you think about using the stable ABI even in shipped extensions? Have you already used it somewhere else? Cheers, -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
