[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Okay, it looks like all issues have been fixed. The optimization for compact keys has been implemented. I didn't see any crash on buildbots recently, test_tracemalloc pass on all platforms. Nathaniel asked how domains will be "allocated" or "registered", b

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset 06552275fa30 by Victor Stinner in branch 'default': _tracemalloc: use compact key for traces https://hg.python.org/cpython/rev/06552275fa30 -- ___ Python tracker _

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset 922b632808ac by Victor Stinner in branch 'default': Cleanup hashtable.h https://hg.python.org/cpython/rev/922b632808ac -- ___ Python tracker __

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset 0e0a464faffb by Victor Stinner in branch 'default': Issue #26588: https://hg.python.org/cpython/rev/0e0a464faffb New changeset 14a552645c30 by Victor Stinner in branch 'default': Issue #26588: https://hg.python.org/cpython/rev/14a552645c30

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset dd887518b569 by Victor Stinner in branch 'default': Issue #26588: Optimize tracemalloc_realloc() https://hg.python.org/cpython/rev/dd887518b569 -- ___ Python tracker _

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: (Ooops, I closed the bug by mistake.) -- resolution: fixed -> status: closed -> open ___ Python tracker ___ ___

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: TODO for this issue: smart storage in _tracemalloc for trace keys. Use compact void* key until the first domain != 0 is used. -- resolution: -> fixed status: open -> closed ___ Python tracker

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Ooook, I think that I found and fixed the random failures in the change af1c1149784a. -- ___ Python tracker ___ __

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset c6f30e2731af by Victor Stinner in branch 'default': Issue #26588: remove debug traces from _tracemalloc. https://hg.python.org/cpython/rev/c6f30e2731af New changeset af1c1149784a by Victor Stinner in branch '3.5': Fix _tracemalloc start/stop https:/

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: The bug was seen on 5 buildbots yet: * http://buildbot.python.org/all/builders/PPC64%20Fedora%203.x/ * http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/ * http://buildbot.python.org/all/builders/s390x%20RHEL%203.x/ * http://buildbot.python.org

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset 708beeb65026 by Victor Stinner in branch 'default': Issue #26588: skip test_warnings.test_tracemalloc() https://hg.python.org/cpython/rev/708beeb65026 -- ___ Python tracker

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset f2d64f91d992 by Victor Stinner in branch 'default': Issue #26588: Don't call tracemalloc_init() at module initilization https://hg.python.org/cpython/rev/f2d64f91d992 New changeset d0b2f70731fb by Victor Stinner in branch 'default': Issue #26588: mo

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Failure on "x86 Ubuntu Shared 3.x" at revision af388201c9976aebc4a8a433c95bfc4a1abe014b. I added an assertion in tracemalloc_init() to ensure that the reeentrant flag is set at the end of the function. The reentrant flag was no more set at tracemalloc_start()

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset af388201c997 by Victor Stinner in branch 'default': Issue #26588: one more assertion https://hg.python.org/cpython/rev/af388201c997 -- ___ Python tracker _

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Failure at revision 636fa01842f597bedff7054616c65a4784c92b4a on AMD64 Windows10 3.x (regrtest runs without the -j option, so all tests are run in the same thread). http://buildbot.python.org/all/builders/AMD64%20Windows10%203.x/builds/763/steps/test/logs/stdio

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Oh, "s390x RHEL 3.x" and "AMD64 Debian root 3.x" buildbots use "-j 1" command line option. In practice, tests are not run in subprocesses, -j1 option is ignored (issue #25285). -- ___ Python tracker

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset 636fa01842f5 by Victor Stinner in branch 'default': Add assertions on tracemalloc_reentrant_key https://hg.python.org/cpython/rev/636fa01842f5 -- ___ Python tracker __

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset bbdb24611294 by Victor Stinner in branch 'default': Issue #26588: more assertions https://hg.python.org/cpython/rev/bbdb24611294 -- ___ Python tracker

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset 340ed3ff2656 by Victor Stinner in branch 'default': Issue #26588: fix compilation on Windows https://hg.python.org/cpython/rev/340ed3ff2656 -- ___ Python tracker _

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset 4ec81b497a84 by Victor Stinner in branch 'default': Issue #26588: add debug traces https://hg.python.org/cpython/rev/4ec81b497a84 -- ___ Python tracker ___

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Crap. I expected the set_reentrant() bug to be fixed, but I saw it one more time on the latest build (revision 0da4532a893389bd1ee1ddfe8227750d968023ba): http://buildbot.python.org/all/builders/s390x%20RHEL%203.x/builds/817/steps/test/logs/stdio 0:02:54 [103/4

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: > New changeset b86cdebe0e97 by Victor Stinner in branch 'default': > tracemalloc now supports domains > https://hg.python.org/cpython/rev/b86cdebe0e97 I hate working on huge patches, it's a pain to rebase them. I chose to push a first implementation to unblock

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset b86cdebe0e97 by Victor Stinner in branch 'default': tracemalloc now supports domains https://hg.python.org/cpython/rev/b86cdebe0e97 -- ___ Python tracker _

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset 7c894911eb59 by Victor Stinner in branch 'default': Simplify implementation of hashtable.c https://hg.python.org/cpython/rev/7c894911eb59 New changeset cd9a40a5ea90 by Victor Stinner in branch 'default': Remove _Py_hashtable_delete() https://hg.pyth

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Hi, > - How do you anticipate the integers naming domains will be allocated? Is it > like port numbers, and you'll maintain a registry somewhere ("domains 0-100 > are reserved for the interpreter, pycuda has reserved 100-110, ...")? I simply have no idea at a

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Nathaniel Smith added the comment: High-level questions: - How do you anticipate the integers naming domains will be allocated? Is it like port numbers, and you'll maintain a registry somewhere ("domains 0-100 are reserved for the interpreter, pycuda has reserved 100-110, ...")? - You have it

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: About memory alignment, _Py_HASHTABLE_ENTRY_DATA_AS_VOID_P() macro uses pointer dereference. It may be replaced with _Py_HASHTABLE_ENTRY_READ_DATA() to use memcpy() and avoid memory alignment issue. -- ___ Python tr

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Hum, an assertion fails in set_reentrant(), helper function to set a TLS (Thread Local Storage), but I'm unable to reproduce the bug on my Fedora 23 (AMD64). http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/3316/steps/test/logs/stdio

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset 98a1c8cb882f by Victor Stinner in branch 'default': Issue #26588: Fix compilation warning on Windows https://hg.python.org/cpython/rev/98a1c8cb882f -- ___ Python tracker

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Roundup Robot added the comment: New changeset aca4e9af1ca6 by Victor Stinner in branch 'default': hashtable.h now supports keys of any size https://hg.python.org/cpython/rev/aca4e9af1ca6 -- nosy: +python-dev ___ Python tracker

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Hum, the patch is really too hard to review. So I extract to the "key_size" part: hashtable_key_size.patch. I added new macros to get the key, wrapper to memcpy() with an assert to check the key size in debug mode. -- Added file: http://bugs.python.or

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Patch version 5: * add hashtable_compare_pointer_t() which compares pointer_t fields (ptr and domain) rather using _Py_hashtable_compare_direct() which causes issues with padding * hashtable_hash_pointer_t() now uses memcpy() to get the pointer_t value rather

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Antoine Pitrou added the comment: What about compilers that don't support "__attribute__((packed))"? Instead you could use a compare func that compares struct fields... -- ___ Python tracker __

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: > What about alignment issues on strict platforms? (SPARC?) Which kind of alignment issue do you expect? I added "__attribute__((packed))" to frame_t. I expected SIGBUS because of that (it's even documented in a comment ;-)), but I practice nobdy complained, n

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Antoine Pitrou added the comment: > Instead of filling pointer_t padding with zeros, I changed the size of the > key (key_size) in the hashtable to exclude padding. What about alignment issues on strict platforms? (SPARC?) -- ___ Python tracker

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Updated patch taking Antoine's comments in account. Instead of filling pointer_t padding with zeros, I changed the size of the key (key_size) in the hashtable to exclude padding. -- Added file: http://bugs.python.org/file42232/tracemalloc-4.patch

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Crap. I pushed tracemalloc-3.patch by mistake. I just reverted my mistake. At least, it helped to find first bugs using buildbots: http://buildbot.python.org/all/builders/PPC64%20Fedora%203.x/builds/676/steps/test/logs/stdio [230/400/3] test_tracemalloc python

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

Antoine Pitrou added the comment: Le 18/03/2016 22:39, STINNER Victor a écrit : > > In the issue #26530, it was proposed (required?) to support multiple addresses spaces to be able to use tracemalloc in numpy. Only proposed :-) Numpy itself only works on the CPU, however Nathaniel may have othe

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Patch version 2: - modify _tracemalloc._get_traces() to include domain in traces - add tracemalloc.DomainFilter(domain) - add domain optional parameter to tracemalloc.Filter constructor Currently, Snapshot._group_by() ignores domain information. I don't know ho

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: > Also, Numba allows use of the Numpy API (part thereof, anyway) on GPUs. Do you mean that Numba requires supporting multiple address spaces to be able to use tracemalloc? Or that supporting multiple address spaces would also to *also* track GPU memory allocat

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

New submission from STINNER Victor: The tracemalloc module uses a hashtable: pointer (void*) => trace. Some embedded devices use multiple address spaces. Each GPU also has its own address space. CUDA, OpenCL and OpenGL should also be seen as different address spaces. In the issue #26530, it wa

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Patch 3: - avoid the memory overhead of storing domain if domains are not used - add optional use_domain parameter to tracemalloc.start() - by default, no change: use a C void* (pointer) as key for the trace hashtable - if start() is called with use_domain=True,

[issue26588] _tracemalloc: add support for multiple address spaces (domains)

STINNER Victor added the comment: Hum, on OpenStack it's common for me to work on a patch serie. Since our development workflow doesn't support that, I used a single patch. It's hard to review it, all changes are merged into one big patch :-/ -- ___