Re: [Numpy-discussion] New iterator APIs (nditer / MapIter): Overlap detection in NumPy
On Mo, 2016-09-12 at 20:22 +, Pauli Virtanen wrote: > Mon, 12 Sep 2016 11:31:07 +0200, Sebastian Berg kirjoitti: > > > > > > > > * NPY_ITER_COPY_IF_OVERLAP, NPY_ITER_OVERLAP_NOT_SAME > > > flags for NpyIter_New. > > > > > > * New API function PyArray_MapIterArrayCopyIfOverlap, > > > as ufunc.at needs to check overlaps for index arrays before > > > constructing iterators, and the parsing is done in multiarray. > > I think here Nathaniels point might be right. It could be we can > > assume > > that copying is always fine, there is probably only one or two > > downstream projects using this function, plus it seems harder to > > create > > abusing structures that actually do something useful. > > It was only exposed for usage in `ufunc.at` if I remember right. I > > know > > theano uses it though, but not sure about anyone else, maybe numba. > > On > > the other hand It is not the worst API clutter in history. > Do you suggest that I break the PyArray_MapIterArray API? > > One issue here is that the function doesn't make distinction between > read- > only access and read-write access, so copying may give unnecessary > slowdown. The second thing is that it will result to a bit uglier > code, as > I need to manage the overlap with the second operation in ufunc_at. > Yeah, was only wondering about MapIterArray, because I might get away with the API break in the case that it works everywhere for our internal usage. But if its not quite straight forward, there is no point in thinking about it. > For NpyIter, I'd still be wary about copying by default, because it's > not > needed everywhere (the may_share_memory checks are better done > earlier), > and since the semantic change can break things inside Numpy. > Yes, I tend to agree here about it. You can always wonder whether its still the most convenient place to do the checks (at least for a few places), but from glancing at the code, it still seems elegant to me. If we are concerned about making the iterator more and more complex, maybe we can really do something else about it as well. I am not sure whether I will manage to look at it very closely soon, so would invite anyone to take a look; this is definitely a highlight! - Sebastian > Pauli > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New iterator APIs (nditer / MapIter): Overlap detection in NumPy
Mon, 12 Sep 2016 11:31:07 +0200, Sebastian Berg kirjoitti: >> * NPY_ITER_COPY_IF_OVERLAP, NPY_ITER_OVERLAP_NOT_SAME >> flags for NpyIter_New. >> >> * New API function PyArray_MapIterArrayCopyIfOverlap, >> as ufunc.at needs to check overlaps for index arrays before >> constructing iterators, and the parsing is done in multiarray. > > I think here Nathaniels point might be right. It could be we can assume > that copying is always fine, there is probably only one or two > downstream projects using this function, plus it seems harder to create > abusing structures that actually do something useful. > It was only exposed for usage in `ufunc.at` if I remember right. I know > theano uses it though, but not sure about anyone else, maybe numba. On > the other hand It is not the worst API clutter in history. Do you suggest that I break the PyArray_MapIterArray API? One issue here is that the function doesn't make distinction between read- only access and read-write access, so copying may give unnecessary slowdown. The second thing is that it will result to a bit uglier code, as I need to manage the overlap with the second operation in ufunc_at. For NpyIter, I'd still be wary about copying by default, because it's not needed everywhere (the may_share_memory checks are better done earlier), and since the semantic change can break things inside Numpy. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New iterator APIs (nditer / MapIter): Overlap detection in NumPy
On So, 2016-09-11 at 21:21 +, Pauli Virtanen wrote: > Hi, > > In the end some further API additions turn out to appear needed: > Very nice :). > * NPY_ITER_COPY_IF_OVERLAP, NPY_ITER_OVERLAP_NOT_SAME > flags for NpyIter_New. > > * New API function PyArray_MapIterArrayCopyIfOverlap, > as ufunc.at needs to check overlaps for index arrays > before constructing iterators, and the parsing is done > in multiarray. I think here Nathaniels point might be right. It could be we can assume that copying is always fine, there is probably only one or two downstream projects using this function, plus it seems harder to create abusing structures that actually do something useful. It was only exposed for usage in `ufunc.at` if I remember right. I know theano uses it though, but not sure about anyone else, maybe numba. On the other hand It is not the worst API clutter in history. > > Continuation here: https://github.com/numpy/numpy/pull/8043 > > > > Wed, 07 Sep 2016 18:02:59 +0200, Sebastian Berg kirjoitti: > > > > > Hi all, > > > > Pauli just opened a nice pull request [1] to add overlap detection > > to > > the new iterator, this means adding a new iterator flag: > > > > `NPY_ITER_COPY_IF_OVERLAP` > > > > If passed to the iterator (also exposed in python), the iterator > > will > > copy the operands such that reading and writing should only occur > > for > > identical operands. For now this is implemented by always copying > > the > > output/writable operand (this could be improved though, so I would > > not > > say its fixed API). > > > > Since adding this flag is new API, please feel free to suggest > > other > > names/approaches or even decline the change ;). > > > > > > This is basically a first step, which should be easily followed by > > adding overlap detection to ufuncs, removing traps such as the well > > (or > > not so well known) `a += a.T`. Other parts of numpy may follow one > > by > > one. > > > > The work is based on his older awesome new memory overlap detection > > implementation. > > > > If there are no comments, I will probably merge it very soon, so we > > can > > look at the follow up things. > > > > - Sebastian > > > > > > [1] https://github.com/numpy/numpy/ > pull/8026___ > > > > NumPy-Discussion mailing list NumPy-Discussion@scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] New iterator APIs (nditer / MapIter): Overlap detection in NumPy
Hi, In the end some further API additions turn out to appear needed: * NPY_ITER_COPY_IF_OVERLAP, NPY_ITER_OVERLAP_NOT_SAME flags for NpyIter_New. * New API function PyArray_MapIterArrayCopyIfOverlap, as ufunc.at needs to check overlaps for index arrays before constructing iterators, and the parsing is done in multiarray. Continuation here: https://github.com/numpy/numpy/pull/8043 Wed, 07 Sep 2016 18:02:59 +0200, Sebastian Berg kirjoitti: > Hi all, > > Pauli just opened a nice pull request [1] to add overlap detection to > the new iterator, this means adding a new iterator flag: > > `NPY_ITER_COPY_IF_OVERLAP` > > If passed to the iterator (also exposed in python), the iterator will > copy the operands such that reading and writing should only occur for > identical operands. For now this is implemented by always copying the > output/writable operand (this could be improved though, so I would not > say its fixed API). > > Since adding this flag is new API, please feel free to suggest other > names/approaches or even decline the change ;). > > > This is basically a first step, which should be easily followed by > adding overlap detection to ufuncs, removing traps such as the well (or > not so well known) `a += a.T`. Other parts of numpy may follow one by > one. > > The work is based on his older awesome new memory overlap detection > implementation. > > If there are no comments, I will probably merge it very soon, so we can > look at the follow up things. > > - Sebastian > > > [1] https://github.com/numpy/numpy/ pull/8026___ > NumPy-Discussion mailing list NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
