Re: [Numpy-discussion] deprecate updateifcopy in nditer operand, flags?

[Matti Picus]

Date: Wed, 8 Nov 2017 18:41:03 +0200 From: Matti Picus <matti.pi...@gmail.com> To: numpy-discussion@python.org Subject: [Numpy-discussion] deprecate updateifcopy in nditer operand flags? Message-ID: <c46bfba8-bad8-5166-e580-456527042...@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed I filed issue 9714 https://github.com/numpy/numpy/issues/9714 and wrote a mail in September trying to get some feedback on what to do with updateifcopy semantics and user-exposed nditer. It garnered no response, so I am trying again. For those who are unfamiliar with the issue see below for a short summary and issue 7054 for a lengthy discussion. Note that pull request 9639 which should be merged very soon changes the magical UPDATEIFCOPY into WRITEBACKIFCOPY, and hopefully will appear in NumPy 1.14. As I mention in the issue, there is a magical update done in this snippet in the next-to-the-last line: |a = np.arange(24, dtype='f8').reshape(2, 3, 4).T i = np.nditer(a, [], [['readwrite', 'updateifcopy']], casting='same_kind', op_dtypes=[np.dtype('f4')]) # Check that UPDATEIFCOPY is activated i.operands[0][2, 1, 1] = -12.5 assert a[2, 1, 1] != -12.5 i = None # magic!!! assert a[2, 1, 1] == -12.5| Formatting a = np.arange(24, dtype='f8').reshape(2, 3, 4).T i = np.nditer(a, [], [['readwrite', 'updateifcopy']], casting='same_kind',                op_dtypes=[np.dtype('f4')]) # Check that WRITEBACKIFCOPY is activated i.operands[0][2, 1, 1] = -12.5 assert a[2, 1, 1] != -12.5 i=None                       # magic assert a[2, 1, 1] == -12.5 Not only is this magic very implicit, it relies on refcount semantics and thus does not work on PyPy. Possible solutions: 1. nditer is rarely used, just deprecate updateifcopy use on operands 2. make nditer into a context manager, so the code would become explicit |a = np.arange(24, dtype='f8').reshape(2, 3, 4).T with np.nditer(a, [], [['readwrite', 'updateifcopy']], casting='same_kind', op_dtypes=[np.dtype('f4')]) as i: # Check that WRITEBACKIFCOPY is activated i.operands[0][2, 1, 1] = -12.5 assert a[2, 1, 1] != -12.5 assert a[2, 1, 1] == -12.5 # a is modified in i.__exit__| Formatting a = np.arange(24, dtype='f8').reshape(2, 3, 4).T with np.nditer(a, [], [['readwrite', 'updateifcopy']], casting='same_kind',                op_dtypes=[np.dtype('f4')]) as i:     # Check that WRITEBACKIFCOPY is activated     i.operands[0][2, 1, 1] = -12.5     assert a[2, 1, 1] != -12.5 assert a[2, 1, 1] == -12.5                   # a is modified in i.__exit__ 3. something else? Any opinions? Does anyone use nditer in production code? Matti ------------------------- what are updateifcopy semantics? When a temporary copy or work buffer is required, NumPy can (ab)use the base attribute of an ndarray by ?? - creating a copy of the data from the base array ?? - mark the base array read-only Then when the temporary buffer is "no longer needed" ?? - the data is copied back ?? - the original base array is marked read-write The trigger for the "no longer needed" decision before pull request 9639 is in the dealloc function. That is not generally a place to do useful work, especially on PyPy which can call dealloc much later. Pull request 9639 adds an explicit PyArray_ResolveWritebackIfCopy api function, and recommends calling it explicitly before dealloc. The only place this change is visible to the python-level user is in nditer. C-API users will need to adapt their code to use the new API function, with a deprecation cycle that is backwardly compatible on CPython.

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion