[issue18178] Redefinition of malloc(3) family of functions at build time
Christian Heimes added the comment: Thx for the fix! -- nosy: +christian.heimes resolution: - fixed stage: commit review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18178 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18178] Redefinition of malloc(3) family of functions at build time
Roundup Robot added the comment: New changeset f09ca52747a6 by Christian Heimes in branch '3.3': Issue #18178: Fix ctypes on BSD. dlmalloc.c was compiled twice which broke malloc weak symbols. http://hg.python.org/cpython/rev/f09ca52747a6 New changeset bea2f12e899e by Christian Heimes in branch 'default': Issue #18178: Fix ctypes on BSD. dlmalloc.c was compiled twice which broke malloc weak symbols. http://hg.python.org/cpython/rev/bea2f12e899e New changeset d4ac6eee7061 by Christian Heimes in branch '2.7': Issue #18178: Fix ctypes on BSD. dlmalloc.c was compiled twice which broke malloc weak symbols. http://hg.python.org/cpython/rev/d4ac6eee7061 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18178 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18178] Redefinition of malloc(3) family of functions at build time
koobs added the comment: Commit looks good, confirming test suite passing for 3.x, 3.3 and 2.7.on http://buildbot.python.org/all/buildslaves/koobs-freebsd10 Thank you for picking this up and finishing it off Christian. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18178 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18178] Redefinition of malloc(3) family of functions at build time
koobs added the comment: I've added a new FreeBSD 10.0-CURRENT buildbot to the pool (thanks antoine) that reproduces the issue and should provide sufficient coverage for testing the proposed patch: http://buildbot.python.org/all/buildslaves/koobs-freebsd10 I'll upgrade the FreeBSD 9-STABLE buildbot once this is resolved -- nosy: +koobs ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18178 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18178] Redefinition of malloc(3) family of functions at build time
New submission from rm: Hello. Marcel Moolenaar (mar...@freebsd.org) pointed this out after committing FreeBSD revision 250991 [1], that makes the malloc(3) family of functions weak symbols. I'm citing him, because (silly me) I don't understand all of this completely: After my commit to head that makes the malloc(3) family of functions weak symbols, a bug in what appears to be the Python build was exposed. The net effect of the bug is that _ctypes.so contains (strong) definitions of the malloc(3) family of functions and resulting in unintended symbol resolution. _ctypes.so incorporates the libffi functionality for what I presume is the basis for Python bindings. libffi includes dlmalloc.c, an open source allocator. dlmalloc.c is incuded by closures.c and closures.c defines USE_DL_PREFIX. On top of that closures.c makes all allocator functions static. This, therefore adds a memory allocator to libffi without exposure of standard symbols In short: dlmalloc.c never gets compiler separately or independently for this reason. The python build however compiles dlmalloc.c separately/independently. As such, dlmalloc.c now defines and exports the malloc(3) family of functions and it also get linked into _ctypes.so. Thus when libffi is built as part of the Python build, the set of symbols exported is different from when libffi is compiled as a port. The simplest test case is this (on a -current machines): 1. Build install lang/python27 (unmodified; if needed) 2. try and build databases/py-sqlite3 The build of databases/py-sqlite3 fails because this: running config *** Signal 11 On amd64 I observed an assertion failure of FreeBSD's malloc, with the same effect. Manually triggering this: fbsdvm% pwd /usr/ports/databases/py-sqlite3/work/Python-2.7.5/Modules fbsdvm% /usr/local/bin/python2.7 setup.py config running config Segmentation fault But if symbols are resolved non-lazily, things change: fbsdvm% setenv LD_BIND_NOW yes fbsdvm% /usr/local/bin/python2.7 setup.py config running config This demonstrates how, after loading _ctypes.so, malloc(3) and friends get resolved differently and thus inconsistently from before. The publicly visible symbols in _ctypes.so, also show the problem: fbsdvm% nm /usr/local/lib/python2.7/lib-dynload/_ctypes.so | grep ' T ' | egrep '(alloc)|(free)' c3f0 T _ctypes_alloc_callback 5250 T _ctypes_alloc_format_string 00016d10 T calloc 000121e0 T ffi_closure_alloc 00013760 T ffi_closure_free 00016050 T free 000170c0 T independent_calloc 000172d0 T independent_comalloc 000148e0 T malloc 00017460 T malloc_footprint 00017480 T malloc_max_footprint 000175c0 T malloc_stats 00017440 T malloc_trim 000176e0 T malloc_usable_size 000173a0 T pvalloc 00016d90 T realloc 00017310 T valloc There are definitions of malloc(3) and friends that shouldn't be there. We have similar reports in our bug-tracker [2] and [3]. And the patch attached fixes this for post-r250991 versions of FreeBSD, and noop for earlier versions. The same patch applied to python2.7, python3.2, python3.3 and patched in FreeBSD ports tree locally. [1] http://svnweb.freebsd.org/changeset/base/250991 [2] http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/179102 [3] http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/179413 -- components: ctypes files: patch-Modules-_ctypes-libffi_fficonfig.py.in messages: 190892 nosy: cvs-src priority: normal severity: normal status: open title: Redefinition of malloc(3) family of functions at build time type: crash versions: Python 2.7 Added file: http://bugs.python.org/file30522/patch-Modules-_ctypes-libffi_fficonfig.py.in ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18178 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18178] Redefinition of malloc(3) family of functions at build time
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +amaury.forgeotdarc, belopolsky, meador.inge ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18178 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18178] Redefinition of malloc(3) family of functions at build time
Amaury Forgeot d'Arc added the comment: Said simpler: dlmalloc.c code is indeed compiled twice: - once as part of closures.c, which #include dlmalloc.c; this is done carefully (with 'static' and 'USE_DL_PREFIX') to not clash with standard malloc functions. - once as a separate target; without USE_DL_PREFIX it will define functions named malloc() and free()... The second one is not necessary, of course. The patch looks good to me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18178 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18178] Redefinition of malloc(3) family of functions at build time
Alexander Belopolsky added the comment: I don't have a system affected by the change, but the explanation and the patch look right to me. FWIW, the patch applies cleanly to 3.4 head and passes 'make test'. -- stage: - commit review versions: +Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18178 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
