[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: I reread the issue. I hope that I now addressed all issues. The remaining issue, bytearray(int) is now tracked by the new issue #21644. -- resolution: -> fixed status: open -> closed ___ Python tracker

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: "2) I'm not happy with the refactoring in bytearray_init(). (...) 3) Somewhat similarly, I wonder if it was necessary to refactor PyBytes_FromStringAndSize(). (...)" Ok, I reverted the change on bytearray(int) and opened the issue #21644 to discuss these tw

[issue21233] Add *Calloc functions to CPython memory allocation API

Roundup Robot added the comment: New changeset dff6b4b61cac by Victor Stinner in branch 'default': Issue #21233: Revert bytearray(int) optimization using calloc() http://hg.python.org/cpython/rev/dff6b4b61cac -- ___ Python tracker

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: "Okay, then let's please call it: _PyObject_Calloc(void *ctx, size_t nobjs, size_t objsize) _PyObject_Alloc(int use_calloc, void *ctx, size_t nobjs, size_t objsize)" "void * PyMem_RawCalloc(size_t nelem, size_t elsize);" prototype comes from the POSIX standad:

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: > I'm not sure: The usual case with ABI changes is that extensions may segfault > if they are *not* recompiled [1]. Ok, I renamed the structure PyMemAllocator to PyMemAllocatorEx, so the compilation fails because PyMemAllocator name is not defined. Modules com

[issue21233] Add *Calloc functions to CPython memory allocation API

Roundup Robot added the comment: New changeset 6374c2d957a9 by Victor Stinner in branch 'default': Issue #21233: Rename the C structure "PyMemAllocator" to "PyMemAllocatorEx" to http://hg.python.org/cpython/rev/6374c2d957a9 -- ___ Python tracker

[issue21233] Add *Calloc functions to CPython memory allocation API

Roundup Robot added the comment: New changeset 358a12f4d4bc by Victor Stinner in branch 'default': Issue #21233: Fix _PyObject_Alloc() when compiled with WITH_VALGRIND defined http://hg.python.org/cpython/rev/358a12f4d4bc -- ___ Python tracker

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: STINNER Victor wrote: > My final commit includes an addition to What's New in Python 3.5 doc, > including a notice in the porting section. It is not enough? I'm not sure: The usual case with ABI changes is that extensions may segfault if they are *not* recompiled

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: STINNER Victor wrote: > PyObject_Malloc(100) asks to allocate one object of 100 bytes. Okay, then let's please call it: _PyObject_Calloc(void *ctx, size_t nobjs, size_t objsize) _PyObject_Alloc(int use_calloc, void *ctx, size_t nobjs, size_t objsize) --

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: >"allocate nbytes elements of size 1" PyObject_Malloc(100) asks to allocate one object of 100 bytes. For PyMem_Malloc() and PyMem_RawMalloc(), it's more difficult to guess, but IMO it's sane to bet that a single memory block of size bytes is requested. I c

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: > > 5) If WITH_VALGRIND is defined, nbytes is uninitialized in > _PyObject_Alloc(). > > Did you see my second commit? It's nlt already fixed? I don't think so, I have revision 5d076506b3f5 here. -- ___ Python tracker

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: > 5) If WITH_VALGRIND is defined, nbytes is uninitialized in _PyObject_Alloc(). Did you see my second commit? It's nlt already fixed? -- ___ Python tracker ___

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: > 6) We need some kind of prominent documentation that existing >programs need to be changed: My final commit includes an addition to What's New in Python 3.5 doc, including a notice in the porting section. It is not enough? Even if the API is public, the P

[issue21233] Add *Calloc functions to CPython memory allocation API

Nathaniel Smith added the comment: A simple solution would be to change the name of the struct, so that non-updated libraries will get a compile error instead of a runtime crash. -- ___ Python tracker

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: Another thing: 6) We need some kind of prominent documentation that existing programs need to be changed: Python 3.5.0a0 (default:62438d1b11c7+, May 3 2014, 23:35:03) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information.

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: I forgot one thing: 5) If WITH_VALGRIND is defined, nbytes is uninitialized in _PyObject_Alloc(). -- ___ Python tracker ___ __

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: I did a post-commit review. A couple of things: 1) I think Victor and I have a different view of the calloc() parameters. calloc(size_t nmemb, size_t size) If a memory region of bytes is allocated, IMO 'nbytes' should be in the place of 'nmemb' and

[issue21233] Add *Calloc functions to CPython memory allocation API

Roundup Robot added the comment: New changeset 62438d1b11c7 by Victor Stinner in branch 'default': Issue #21233: Oops, Fix _PyObject_Alloc(): initialize nbytes before going to http://hg.python.org/cpython/rev/62438d1b11c7 -- ___ Python tracker

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: Antoine Pitrou wrote: > The real use case I envision is with huge powers of two. If I write: > x = 2 ** 100 I created the issue #21419 for this idea. -- ___ Python tracker __

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: > There is no need to hurry. I changed my mind :-p It should be easier for numpy to test the development version of Python. Let's wait for buildbots. -- ___ Python tracker _

[issue21233] Add *Calloc functions to CPython memory allocation API

Roundup Robot added the comment: New changeset 5b0fda8f5718 by Victor Stinner in branch 'default': Issue #21233: Add new C functions: PyMem_RawCalloc(), PyMem_Calloc(), http://hg.python.org/cpython/rev/5b0fda8f5718 -- nosy: +python-dev ___ Python trac

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: > If you prefer to commit very soon, > I promise to do a post commit review. There is no need to hurry. -- ___ Python tracker ___

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: Victor, sure, maybe not right away. If you prefer to commit very soon, I promise to do a post commit review. -- ___ Python tracker ___ ___

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: @Stefan: Can you please review calloc-6.patch? Charles-François wrote that the patch looks good, but for such critical operation (memory allocation), I would prefer a second review ;) -- ___ Python tracker

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: LGTM! -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: Patch version 6: - I renamed "int zero" parameter to "int use_calloc" and move the new parameter at the first position to avoid confusion with nelem. For example, _PyObject_Alloc(ctx, 1, nbytes, 0) becomes _PyObject_Alloc(0, ctx, 1, nbytes). It also more logi

[issue21233] Add *Calloc functions to CPython memory allocation API

Nathaniel Smith added the comment: > It would be interesting to see some NumPy benchmarks (Nathaniel?). What is it you want to see? NumPy already uses calloc; we benchmarked it when we added it and it made a huge difference to various realistic workloads :-). What NumPy gets out of this isn't

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: The order of the nelem/elsize matters for readability. Otherwise it is not intuitive what happens after the jump to redirect in _PyObject_Alloc(). Why would you assert that 'nelem' is one? -- ___ Python tracker

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: > Hmm, obmalloc.c changed as well, so already the gcc optimizer can take > different paths and produce different results. If decimal depends on allocator performances, you should maybe try to implement a freelist. > Also I did set mpd_callocfunc to PyMem_Calloc

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: > Also I did set mpd_callocfunc to PyMem_Calloc(). 2% slowdown is far > from being a tragic result, so I guess we can ignore that. Agreed. > The bytes() speedup is very nice. Allocations that took one second > are practically instant now. Indeed. Vict

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: Hmm, obmalloc.c changed as well, so already the gcc optimizer can take different paths and produce different results. Also I did set mpd_callocfunc to PyMem_Calloc(). 2% slowdown is far from being a tragic result, so I guess we can ignore that. The bytes() speedup

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: > With the latest patch the decimal benchmark with a lot of small > allocations is consistently 2% slower. Does your benchmark use bytes(int) or bytearray(int)? If not, I guess that your benchmark is not reliable because only these two functions are changed by

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: With the latest patch the decimal benchmark with a lot of small allocations is consistently 2% slower. Large factorials (where the operands are initialized to zero for the number-theoretic transform) have the same performance with and without the patch. It would be

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: Demo of calloc-5.patch on Linux. Thanks to calloc(), bytes(50 * 1024 * 1024) doesn't allocate memory for null bytes and so the RSS memory is unchanged (+148 kB, not +50 MB), but tracemalloc says that 50 MB were allocated. $ ./python -X tracemalloc Python 3.5.0

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: Patch version 5. This patch is ready for a review. Summary of calloc-5.patch: - add the following functions: * void* PyMem_RawCalloc(size_t nelem, size_t elsize) * void* PyMem_Calloc(size_t nelem, size_t elsize) * void* PyObject_Calloc(size_t nelem, size

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: Changes on the pickle module don't look like an interesting optimization. It even looks slower. $ python perf.py -b fastpickle,fastunpickle,pickle,pickle_dict,pickle_list,slowpickle,slowunpickle,unpickle ../default/python.orig ../default/python.calloc ... Re

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: It looks like Windows supports also lazy initialization of memory pages initialized to zero. According to my microbenchmark on Linux and Windows, only bytes(n) and bytearray(n) are really faster with use_calloc.patch. Most changes of use_calloc.patch are mayb

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: > The real use case I envision is with huge powers of two. I'm not sure that it's a common use case, but it can be nice to optimize this case if it doesn't make longobject.c more complex. It looks like calloc() becomes interesting for objects larger than 1 MB.

[issue21233] Add *Calloc functions to CPython memory allocation API

Antoine Pitrou added the comment: > Ok, now the real use case where it becomes faster: I implemented the > same optimization for bytearray. The real use case I envision is with huge powers of two. If I write: x = 2 ** 100 then all of x's bytes except the highest one will be zeros. If we

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: > Don't hesitate to rerun my benchmark on more different platforms? Oops, I wanted to write ";-)" not "?". -- ___ Python tracker ___ __

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: > Are you sure this is a good platform for performance reports? :) Don't hesitate to rerun my benchmark on more different platforms? -- ___ Python tracker

[issue21233] Add *Calloc functions to CPython memory allocation API

Antoine Pitrou added the comment: > Common platform: > Timer: time.perf_counter > Timer info: namespace(adjustable=False, > implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, > resolution=1e-09) > Platform: Linux-3.13.9-200.fc20.x86_64-x86_64-with-fedora-20-Heisenbug

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: bench_alloc2.py: updated benchmark script. I added bytes(n) and bytearray(n) tests and removed the test decoding from ASCII. Common platform: Timer: time.perf_counter Timer info: namespace(adjustable=False, implementation='clock_gettime(CLOCK_MONOTONIC)', mono

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: I splitted my patch into two parts: - calloc-4.patch: add new "Calloc" functions including _PyObject_GC_Calloc() - use_calloc.patch: patch types (bytes, dict, list, set, tuple, etc.) and various modules to use calloc I reverted my changes on _PyObject_GC_Mallo

[issue21233] Add *Calloc functions to CPython memory allocation API

Changes by STINNER Victor : Added file: http://bugs.python.org/file35064/use_calloc.patch ___ Python tracker ___ ___ Python-bugs-list mailing

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: >> Hm... >> What's /proc/sys/vm/overcommit_memory ? >> If it's set to 0, then the kernel will always overcommit. > > Ah, indeed. See above, I mistyped: 0 is the default (which is already quite optimistic), 1 is always. >> If you set it to 2, normally y

[issue21233] Add *Calloc functions to CPython memory allocation API

Antoine Pitrou added the comment: > Hm... > What's /proc/sys/vm/overcommit_memory ? > If it's set to 0, then the kernel will always overcommit. Ah, indeed. > If you set it to 2, normally you'd definitely get ENOMEM You're right, but with weird results: $ gcc -o /tmp/test test.c; /tmp/test mal

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: > Hm... > What's /proc/sys/vm/overcommit_memory ? > If it's set to 0, then the kernel will always overcommit. I meant 1 (damn, I need sleep). -- ___ Python tracker __

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: > Both OOM here (3.11.0-20-generic, 64-bit, Ubuntu). Hm... What's /proc/sys/vm/overcommit_memory ? If it's set to 0, then the kernel will always overcommit. If you set it to 2, normally you'd definitely get ENOMEM (which is IMO much nicer than getting

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: This is probably offtopic, but I think people who want reliable MemoryErrors can use limits, e.g. via djb's softlimit (daemontools): $ softlimit -m 1 ./python Python 3.5.0a0 (default:462470859e57+, Apr 27 2014, 19:34:06) [GCC 4.7.2] on linux Type "help", "c

[issue21233] Add *Calloc functions to CPython memory allocation API

Antoine Pitrou added the comment: > $ gcc -o /tmp/test /tmp/test.c; /tmp/test > malloc() returned NULL after 3050MB > $ gcc -DDO_MEMSET -o /tmp/test /tmp/test.c; /tmp/test > malloc() returned NULL after 2130MB > > Without memset, the kernel happily allocates until we reach the 3GB > user address

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: > So yeah, touching pages can affect whether a later malloc returns ENOMEM. > > I'm not sure any of this actually matters in the Python case though :-). > There's still no reason to go touching pages pre-emptively just in case we > might write to them

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: Alright, it bothered me so I wrote a small C testcase (attached), which calls malloc in a loop, and can call memset upon the allocated block right after allocation: $ gcc -o /tmp/test /tmp/test.c; /tmp/test malloc() returned NULL after 3050MB $ gcc -DDO

[issue21233] Add *Calloc functions to CPython memory allocation API

Nathaniel Smith added the comment: Right, python3 -c 'b"x" * (2 ** 48)' does give an instant MemoryError for me. So I was wrong about it being the VM limit indeed. The documentation on this is terrible! But, if I'm reading this right: http://lxr.free-electrons.com/source/mm/util.c#L434 the a

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: Dammit, read: python -c 'b"x" * (2**48)' -- ___ Python tracker ___ ___ Python-bugs-list ma

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: > And your test.py produces the same result. Are you sure you don't have a > ulimit set on address space? Yep, I'm sure: $ ulimit -v unlimited It's probably due to the exponential over-allocation used by the array (to guarantee amortized constant cos

[issue21233] Add *Calloc functions to CPython memory allocation API

Nathaniel Smith added the comment: On my laptop (x86-64, Linux 3.13, 12 GB RAM): $ python3 -c "[i for i in range(9)]" zsh: killed python3 -c "[i for i in range(9)]" $ dmesg | tail -n 2 [404714.401901] Out of memory: Kill process 10752 (python3) score 687 or sacrifice child

[issue21233] Add *Calloc functions to CPython memory allocation API

Antoine Pitrou added the comment: > Just try python -c "[i for i in > range()]" on a 64-bit machine, I'll bet you'll get a > MemoryError (ENOMEM). Hmm, I get an OOM kill here. -- ___ Python tracker ___

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: > @Charles-François: I think your worries about calloc and overcommit are > unjustified. First, calloc and malloc+memset actually behave the same way > here -- with a large allocation and overcommit enabled, malloc and calloc > will both go ahead and

[issue21233] Add *Calloc functions to CPython memory allocation API

Nathaniel Smith added the comment: @Charles-François: I think your worries about calloc and overcommit are unjustified. First, calloc and malloc+memset actually behave the same way here -- with a large allocation and overcommit enabled, malloc and calloc will both go ahead and return the large

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: "Because if a code creates many such objects which basically just do calloc(), on operating systems with memory overommitting (such as Linux), the calloc() allocations will pretty much always succeed, but will segfault when the page is first written to in case of

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: list: items are allocated in a second memory block. PyList_New() uses memset(0) to set all items to NULL. tuple: header and items are stored in a single structure (PyTupleObject), in a single memory block. PyTuple_New() fills the items will NULL (so write agai

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: > __libc_calloc() starts with a check on integer overflow. Yes, see my previous message: """ AFAICT, the two arguments are purely historical (it was used when malloc() didn't guarantee suitable alignment, and has the advantage of performing overflow che

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: "And http://www.eglibc.org/cgi-bin/viewvc.cgi/trunk/libc/malloc/malloc.c?view=markup to check that calloc(nelem, elsize) is implemented as calloc(nelem * elsize)" __libc_calloc() starts with a check on integer overflow. --

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: > It looks like calloc-3.patch is wrong: it modify _PyObject_GC_Malloc() to > fill the newly allocated buffer with zeros, but _PyObject_GC_Malloc() is not > only called by PyType_GenericAlloc(): it is also used by _PyObject_GC_New() > and _PyObject_GC

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: Actually, I think we have to match the C-API: For instance, in Modules/_decimal/_decimal.c:5527 the libmpdec allocators are set to the Python allocators. So I'd need to do: mpd_callocfunc = PyMem_Calloc; I suppose that's a common use case. --

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: It looks like calloc-3.patch is wrong: it modify _PyObject_GC_Malloc() to fill the newly allocated buffer with zeros, but _PyObject_GC_Malloc() is not only called by PyType_GenericAlloc(): it is also used by _PyObject_GC_New() and _PyObject_GC_NewVar(). The pa

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: Just to add another data point, I don't find the calloc() API cumbersome. -- ___ Python tracker ___ ___

[issue21233] Add *Calloc functions to CPython memory allocation API

Antoine Pitrou added the comment: > Regarding the *Calloc functions: how about we provide a sane API > instead of reproducing the cumbersome C API? Isn't the point of reproducing the C API to allow quickly switching from calloc() to PyObject_Calloc()? (besides, it seems the OpenBSD guys like th

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: I wrote a short microbenchmark allocating objects using my benchmark.py script. It looks like the operation "(None,) * N" is slower with calloc-3.patch, but it's unclear how much times slower it is. I don't understand why only this operation has different spee

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: Note to numpy devs: it would be great if some of you followed the python-dev mailing list (I know it can be quite volume intensive, but maybe simple filters could help keep the noise down): you guys have definitely both expertise and real-life applicatio

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: > This issue was opened to be able to use tracemalloc on numpy. I would > like to make sure that calloc is enough for numpy. I would prefer to > change the malloc API only once. Then please at least rename the issue. Also, I don't see why everything sho

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: 2014-04-27 10:30 GMT+02:00 Charles-François Natali : >> I read again some remarks about alignement, it was suggested to provide >> allocators providing an address aligned to a requested alignement. This >> topic was already discussed in #18835. > > The aligneme

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: > I read again some remarks about alignement, it was suggested to provide > allocators providing an address aligned to a requested alignement. This topic > was already discussed in #18835. The alignement issue is really orthogonal to the calloc one, s

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: I read again some remarks about alignement, it was suggested to provide allocators providing an address aligned to a requested alignement. This topic was already discussed in #18835. If Python doesn't provide such memory allocators, it was suggested to provide

[issue21233] Add *Calloc functions to CPython memory allocation API

Julian Taylor added the comment: I just tested it, PyObject_NewVar seems to use RawMalloc not the GC malloc so its probably fine. -- ___ Python tracker ___ _

[issue21233] Add *Calloc functions to CPython memory allocation API

Josh Rosenberg added the comment: Well, to be more specific, PyType_GenericAlloc was originally calling one of two methods that didn't zero the memory (one of which was GC_Malloc), then memset-ing. Just realized you're talking about something else; not sure if you're correct about this now, bu

[issue21233] Add *Calloc functions to CPython memory allocation API

Josh Rosenberg added the comment: Julian: No. See the diff: http://bugs.python.org/review/21233/diff/11644/Objects/typeobject.c The original GC_Malloc was explicitly memset-ing after confirming that it received a non-NULL pointer from the underlying malloc call; that memset is removed in favo

[issue21233] Add *Calloc functions to CPython memory allocation API

Julian Taylor added the comment: won't replacing _PyObject_GC_Malloc with a calloc cause Var objects (PyObject_NewVar) to be completely zeroed which I think they didn't before? Some numeric programs stuff a lot of data into var objects and could care about python suddenly setting them to zero w

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: Do you have benchmarks? (I'm not looking for an improvement, just no regression.) -- ___ Python tracker ___ __

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: Patch version 3: remove _PyObject_GC_Calloc(), modify _PyObject_GC_Malloc() instead of use calloc() instead of malloc()+memset(0). -- Added file: http://bugs.python.org/file34924/calloc-3.patch ___ Python tracker

[issue21233] Add *Calloc functions to CPython memory allocation API

Antoine Pitrou added the comment: On mer., 2014-04-16 at 08:06 +, STINNER Victor wrote: > I didn't check which objects use (indirectly) _PyObject_GC_Calloc(). I've checked: lists, tuples, dicts and sets at least seem to use it. Obviously, objects which are not tracked by the GC (such as str

[issue21233] Add *Calloc functions to CPython memory allocation API

Stefan Krah added the comment: I left a Rietveld comment, which probably did not get mailed. -- ___ Python tracker ___ ___ Python-bugs

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: 2014-04-16 3:18 GMT-04:00 Charles-François Natali : >> It calls calloc(size) instead of malloc(size), calloc() which can be faster >> than malloc()+memset(), see: >> https://mail.python.org/pipermail/python-dev/2014-April/133985.html > > It will only make a diff

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: >>> So what is the point of _PyObject_GC_Calloc ? >> >> It calls calloc(size) instead of malloc(size) > > No, the question is why you didn't simply change _PyObject_GC_Malloc > (which is a private function). Oh ok, I didn't understand. I don't like changing the

[issue21233] Add *Calloc functions to CPython memory allocation API

Charles-François Natali added the comment: >> So what is the point of _PyObject_GC_Calloc ? > > It calls calloc(size) instead of malloc(size), calloc() which can be faster > than malloc()+memset(), see: > https://mail.python.org/pipermail/python-dev/2014-April/133985.html It will only make a di

[issue21233] Add *Calloc functions to CPython memory allocation API

Antoine Pitrou added the comment: Le 16/04/2014 04:40, STINNER Victor a écrit : > > STINNER Victor added the comment: > >> So what is the point of _PyObject_GC_Calloc ? > > It calls calloc(size) instead of malloc(size) No, the question is why you didn't simply change _PyObject_GC_Malloc (which

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: New patch: - replace "size_t size" with "size_t nelem, size_t elsize" in the prototype of calloc functions (the parameter names come from the POSIX standard) - replace "int calloc" with "int zero" in helper functions -- Added file: http://bugs.python.o

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: In numpy, I found the two following functions: /*NUMPY_API * Allocates memory for array data. */ void* PyDataMem_NEW(size_t size); /*NUMPY_API * Allocates zeroed memory for array data. */ void* PyDataMem_NEW_ZEROED(size_t size, size_t elsize); So it looks

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: > So what is the point of _PyObject_GC_Calloc ? It calls calloc(size) instead of malloc(size), calloc() which can be faster than malloc()+memset(), see: https://mail.python.org/pipermail/python-dev/2014-April/133985.html _PyObject_GC_Calloc() is used by PyType

[issue21233] Add *Calloc functions to CPython memory allocation API

Josh Rosenberg added the comment: Sorry for breaking it up, but the same comment on consistent prototypes mirroring the C standard lib calloc would apply to all the API functions as well, e.g. PyMem_RawCalloc, PyMem_Calloc, PyObject_Calloc and _PyObject_GC_Calloc, not just the structure functi

[issue21233] Add *Calloc functions to CPython memory allocation API

Josh Rosenberg added the comment: Additional comment on clarity: Might it make sense to make the calloc structure member take both the num and size arguments that the underlying calloc takes? That is, instead of: void* (*calloc) (void *ctx, size_t size); Declare it as: void* (*calloc) (void

[issue21233] Add *Calloc functions to CPython memory allocation API

Josh Rosenberg added the comment: General comment on patch: For the flag value that toggles zero-ing, perhaps use a different name, e.g. setzero, clearmem, initzero or somesuch instead of calloc? calloc already gets used to refer to both the C standard function and the function pointer structu

[issue21233] Add *Calloc functions to CPython memory allocation API

Antoine Pitrou added the comment: So what is the point of _PyObject_GC_Calloc ? -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue21233] Add *Calloc functions to CPython memory allocation API

Changes by Josh Rosenberg : -- nosy: +josh.rosenberg ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue21233] Add *Calloc functions to CPython memory allocation API

Changes by STINNER Victor : -- nosy: +neologix, pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://

[issue21233] Add *Calloc functions to CPython memory allocation API

STINNER Victor added the comment: Here is a first patch adding the following functions: void* PyMem_RawCalloc(size_t n); void* PyMem_Calloc(size_t n); void* PyObject_Calloc(size_t n); PyObject* _PyObject_GC_Calloc(size_t); It adds the following field after malloc field to PyMemAllocator

[issue21233] Add *Calloc functions to CPython memory allocation API

Changes by Éric Araujo : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.or

[issue21233] Add *Calloc functions to CPython memory allocation API

Changes by Stefan Krah : -- nosy: +skrah ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.or

[issue21233] Add *Calloc functions to CPython memory allocation API

New submission from Nathaniel Smith: Numpy would like to switch to using the CPython allocator interface in order to take advantage of the new tracemalloc infrastructure in 3.4. But, numpy relies on the availability of calloc(), and the CPython allocator API does not expose calloc(). https:/

  1   2   >