Re: [Python-Dev] If aligned_alloc() is missing on your platform, please let us know.

[Victor Stinner]

2017-10-27 10:17 GMT+02:00 Stefan Krah :
> Victor wrote a patch and would like to avoid adding a (probably unnecessary)
> emulation function. I agree with that.
> (...)
> So if any platform does not have some form of aligned_alloc(), please
> speak up.

I'm not really opposed to implement an aligned allocator on top of an
existing allocator. I only propose to discuss that in a separated
issue. I wrote "PEP 445 -- Add new APIs to customize Python memory
allocators" to implement tracemalloc, but also because I was working
on a Python version patched to use custom memory allocators, different
than malloc()/free():
https://www.python.org/dev/peps/pep-0445/#rationale

IMHO only users of PyMem_SetAllocators() would need such "fallback". I
expect all modern platforms to provide a "aligned" memory allocator.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] If aligned_alloc() is missing on your platform, please let us know.

[Stefan Krah]

On Sat, Oct 28, 2017 at 03:43:47PM +1000, Nick Coghlan wrote:
> 1. CPython's own support for platforms where we don't have a native aligned
> memory allocation API to call is covered by PEP 11, so if all current
> buildbots still work, then it will be up to the folks interested in a
> platform that *doesn't* offer aligned allocations to provide both a
> suitable emulation and a buildbot to test it on.

Indeed, the feature is backed up by PEP 11.


> 2.While all of the CPython-provided memory allocators will implement the
> new slots, the folks implementing their own custom allocators will need a
> defined upgrade path in the "Porting" section of the What's New guide. For
> that, an explicit error on 3.7+ that says "Configured custom allocator
> doesn't implement aligned memory allocations" is likely going to be easier
> to debug than subtle issues with the way an implicit emulation layer
> interacts with the other memory allocator slots.
> 
> To appropriately address 2, we need more info not about which platforms
> natively support aligned memory allocations, but rather from folks that
> actually are implementing their own custom allocators. It may be that
> making the low level cross-platform raw alligned allocators available as a
> public API (independently of the switchable allocator machinery) will be a
> more appropriate way of handling that case than providing an emulation
> layer in terms of the old slots.

I don't have an opinion whether new slots should be available.  For my use
case I just need PyMem_AlignedAlloc(), PyMem_AlignedFree() that automatically
use the faster allocator for 'sizeof(void *) <= align <= ALIGNMENT' and
'size <= SMALL_REQUEST_THRESHOLD'.


So yes, it would be nice to hear from people who implement custom allocators.



Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] If aligned_alloc() is missing on your platform, please let us know.

[Nick Coghlan]

On 27 October 2017 at 18:17, Stefan Krah  wrote:

> Victor wrote a patch and would like to avoid adding a (probably
> unnecessary)
> emulation function. I agree with that.
>
> So if any platform does not have some form of aligned_alloc(), please
> speak up.
>

I think Victor's suggested strategy in
https://bugs.python.org/issue18835#msg305122 of deferring emulation support
to its own follow-up issue is a good one, since there are two distinct
cases to consider:

1. CPython's own support for platforms where we don't have a native aligned
memory allocation API to call is covered by PEP 11, so if all current
buildbots still work, then it will be up to the folks interested in a
platform that *doesn't* offer aligned allocations to provide both a
suitable emulation and a buildbot to test it on.

2.While all of the CPython-provided memory allocators will implement the
new slots, the folks implementing their own custom allocators will need a
defined upgrade path in the "Porting" section of the What's New guide. For
that, an explicit error on 3.7+ that says "Configured custom allocator
doesn't implement aligned memory allocations" is likely going to be easier
to debug than subtle issues with the way an implicit emulation layer
interacts with the other memory allocator slots.

To appropriately address 2, we need more info not about which platforms
natively support aligned memory allocations, but rather from folks that
actually are implementing their own custom allocators. It may be that
making the low level cross-platform raw alligned allocators available as a
public API (independently of the switchable allocator machinery) will be a
more appropriate way of handling that case than providing an emulation
layer in terms of the old slots.

That is, the suggested approach for custom allocator developers would be:

1. Test against 3.7, get an error complaining the new slots aren't defined
2. Set the new slots to the CPython provided cross-platform abstraction (if
appropriate), or else wrap that abstraction layer as needed, or else
implement your own aligned allocation routines

If the only reason for the custom allocator was to allow tracemalloc to
track additional memory that wouldn't otherwise be handled through
CPython's memory allocators, then consider switching to using
PyTraceMalloc_Track()/PyTraceMalloc_Untrack() instead.

Either way, that transition strategy discussion shouldn't block making the
feature itself available - it will be a lot easier to discuss transition
enablement if potentially affected folks can download 3.7.0a3 and tell us
what actually breaks, rather than asking them to speculate about what they
think *might* break.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] If aligned_alloc() is missing on your platform, please let us know.

[Stefan Krah]

Hello,

we want to add aligned versions of allocation functions to 3.7:

   https://bugs.python.org/issue18835


C11 has aligned_alloc().

Linux, BSD, OSX, MSVC, Android all have either posix_memalign() or
_aligned_malloc().

Cygwin apparently has posix_memalign().

MinGW has:

   
https://github.com/Alexpux/mingw-w64/blob/master/mingw-w64-crt/misc/mingw-aligned-malloc.c



Victor wrote a patch and would like to avoid adding a (probably unnecessary)
emulation function. I agree with that.

So if any platform does not have some form of aligned_alloc(), please
speak up.



Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com