"""
So, Victor, I'm interested to hear how the new API you're proposing
will affect this solution I've come up with for pyparallel; I'm
going to be absolutely dependent upon the ability to lock main
thread pages as read-only in one fell-swoop -- am I still going to
be able to do that with your new API in place?
"""
2013/6/19 Trent Nelson <tr...@snakebite.org>:
> On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:
>> > 1. All memory allocated in a parallel context is localized to a
>> > private heap.
>>
>> How do you allocate memory in this "private" heap? Did you add new
>> functions to allocate memory?
>
> Yup:
> _PyHeap_Malloc():
> http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365.
>
> All memory operations (PyObject_New/Malloc etc) get intercepted
> during parallel thread execution and redirected to _PyHeap_Malloc(),
> which is a very simple slab allocator. (No need for convoluted
> buckets because we never free individual objects during parallel
> execution; instead, we just blow everything away at the end.)
Ok, so I don't think that the PEP 445 would change anything for you.
The following change might have an impact: If _PyHeap_Malloc is not
thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the
GIL is not held would avoid bugs in your code.
If you want to choose dynamically the allocator at runtime, you can
replace PyObject_Malloc allocator using:
-------------------------- 8< -----------------
static void *
_PxMem_AllocMalloc(void *ctx, size_t size)
{
PyMemBlockAllocator *ctx;
if (Py_PXCTX)
return _PxMem_Malloc(size))
else
return alloc->malloc(alloc->ctx, size);
}
...
PyMemBlockAllocator pyparallel_pyobject;
static void *
setup_pyparallel_allocator(void)
{
PyMemBlockAllocator alloc;
PyObject_GetAllocator(&pyparallel_pyobject);
alloc.ctx = &pyparallel_pyobject;
alloc.malloc = _PxMem_AllocMalloc;
...
PyObject_SetAllocator(&alloc);
}
-------------------------- 8< -----------------
But I don't know if you want pyparallel to be an "optional" feature
chosen at runtime...
Victor
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
