[issue18227] Use Python memory allocators in external libraries like zlib or OpenSSL

[STINNER Victor]

STINNER Victor added the comment:

I modified modules when it was possible and easy to do. More modules should be 
modified, but it's more tricky. If you are interested, please open new issues.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18227] Use Python memory allocators in external libraries like zlib or OpenSSL

[Roundup Robot]

Roundup Robot added the comment:

New changeset 7f17c67b5bf6 by Christian Heimes in branch 'default':
Issue #18227: pyexpat now uses a static XML_Memory_Handling_Suite. cElementTree 
uses the same approach since at least Python 2.6
http://hg.python.org/cpython/rev/7f17c67b5bf6

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18227] Use Python memory allocators in external libraries like zlib or OpenSSL

[Roundup Robot]

Roundup Robot added the comment:

New changeset 12f26c356611 by Victor Stinner in branch 'default':
Issue #18227: "Free" function of bz2, lzma and zlib modules has no return value 
(void)
http://hg.python.org/cpython/rev/12f26c356611

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18227] Use Python memory allocators in external libraries like zlib or OpenSSL

[Roundup Robot]

Roundup Robot added the comment:

New changeset a876d9d2e4fc by Victor Stinner in branch 'default':
Issue #18227: Use PyMem_RawAlloc() in bz2, lzma and zlib modules
http://hg.python.org/cpython/rev/a876d9d2e4fc

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18227] Use Python memory allocators in external libraries like zlib or OpenSSL

[STINNER Victor]

STINNER Victor added the comment:

Here is an initial attempt: set a custom allocator for bz2, lzma and zlib 
modules. The allocator is only replaced for an instance of a compressor or 
decompress, the change does not affect the library globally.

PyMem_RawMalloc() is used instead of PyMem_Malloc() because the GIL is always 
released.

--
keywords: +patch
Added file: http://bugs.python.org/file30673/set_custom_alloc.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18227] Use Python memory allocators in external libraries like zlib or OpenSSL

[STINNER Victor]

STINNER Victor added the comment:

It looks like CRYPTO_set_mem_functions() of OpenSSL 1.0.1e-4.fc18 does not 
work: CRYPTO_set_mem_functions() calls indirectly CRYPTO_malloc() which sets 
"allow_customize = 0;" and so CRYPTO_set_mem_functions() does nothing (just 
return 0, instead of 1).

Gdb trace with a modified _ssl module:

#0  0x003803463100 in CRYPTO_malloc () from /lib64/libcrypto.so.10
#1  0x003803542fae in FIPS_drbg_new () from /lib64/libcrypto.so.10
#2  0x0038035448e1 in FIPS_drbg_health_check () from /lib64/libcrypto.so.10
#3  0x003803542e88 in FIPS_drbg_init () from /lib64/libcrypto.so.10
#4  0x0038034cf9d1 in RAND_init_fips () from /lib64/libcrypto.so.10
#5  0x003803465764 in OPENSSL_init_library () from /lib64/libcrypto.so.10
#6  0x003803462c61 in CRYPTO_set_mem_functions () from 
/lib64/libcrypto.so.10
#7  0x7135bc6c in PyInit__ssl () at 
/home/haypo/prog/python/default/Modules/_ssl.c:3180

See the code:
http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=crypto/mem.c;h=f7984fa958eb1edd6c61f6667f3f2b29753be662;hb=HEAD#l124

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18227] Use Python memory allocators in external libraries like zlib or OpenSSL

[Aaron Iles]

Changes by Aaron Iles :


--
nosy: +aliles

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18227] Use Python memory allocators in external libraries like zlib or OpenSSL

[STINNER Victor]

STINNER Victor added the comment:

Uncomplete(?) list of external libraries used by Python:

- libffi (_ctypes): has its own memory allocator (dlmalloc), is it used? see 
also issue #18178
- libmpdec (_decimal): already configured to reuse the PyMem_Malloc() family 
(see Modules/_decimal/_decimal.c: "Init libpdec")
- _sqlite (sqlite3: see http://www.sqlite.org/malloc.html
- expat (expact): ?
- zlib (zlib): http://www.zlib.net/manual.html#Usage
- OpenSSL (_ssl, hashlib): CRYPTO_set_mem_functions, 
http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=crypto/mem.c;h=f7984fa958eb1edd6c61f6667f3f2b29753be662;hb=HEAD#l124
- Tcl/Tk (_tkinter): http://tmml.sourceforge.net/doc/tcl/Alloc.html
- bz2: http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html
- libncurses (curses): ?
- dbm libraries: ndbm, gdbm, db, ... (dbm): ?
- lzma: http://www.asawicki.info/news_1368_lzma_sdk_-_how_to_use.html
- Windows API (_winapi, nt): ?
- readline (readline): ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18227] Use Python memory allocators in external libraries like zlib or OpenSSL

[STINNER Victor]

STINNER Victor added the comment:

See also the issue #18203: "Replace calls to malloc() with PyMem_Malloc() or 
PyMem_RawMalloc()".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18227] Use Python memory allocators in external libraries like zlib or OpenSSL

[STINNER Victor]

New submission from STINNER Victor:

With the PEP 445 and the issue #3329, Python will get an API to setup custom 
memory allocators. To be able to configure how memory is handled in external 
libraries, some libraries allow to setup a custom allocator too. New functions 
PyMem_RawMalloc(), PyMem_GetRawAllocators() PyMem_SetRawAllocators() can be 
used for that.

The safest option is to only reuse custom allocators if a library allows to 
setup them for a specfic function call or a specific object, and not replace 
the memory allocators globally. For example, the lzma library allows to set 
memory allocators only for one compressor object: 
LzmaEnc_Create(&SzAllocForLzma);

We might change the global allocators of a library if Python is not embedded, 
but Python *is* the application (the standard "python" program).

I don't know yet if it is safe to reuse custom memory allocators.

Windows has for example a special behaviour: each DLL (dynamic library) has its 
own heap, memory allocator in a DLL cannot be released from another DLL. Would 
this issue introduce such bug?

--
components: Interpreter Core
messages: 191248
nosy: christian.heimes, haypo
priority: normal
severity: normal
status: open
title: Use Python memory allocators in external libraries like zlib or OpenSSL
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com