Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray
I agree with Number 2 and 4. On Sat, Oct 27, 2018 at 12:38 AM Eric Wieser wrote: > in order to be used prior to calling C or Fortran code that expected at > least a 1-d array > > I’d argue that the behavior for these functions should have just been to > raise an error saying “this function does not support 0d arrays”, rather > than silently inserting extra dimensions. As a bonus, that would push the > function developers to add support for 0d. Obviously we can’t make it do > that now, but what we can do is have it emit a warning in those cases. > > I think our options are: > >1. Deprecate the entire function >2. Deprecate and eventually(?) throw an error upon calling the >function on 0d arrays, with a message like *“in future using >ascontiguousarray to promote 0d arrays to 1d arrays will not be supported. >If promotion is intentional, use ascontiguousarray(atleast1d(x)) to silence >this warning and keep the old behavior, and if not use asarray(x, >order='C') to preserve 0d arrays”* >3. Deprecate (future-warning) when passed 0d arrays, and eventually >skip the upcast to 1d. >If the calling code really needed a 1d array, then it will probably >fail, which is not really different to 2, but has the advantage that the >names are less surprising. >4. Only improve the documentation > > My preference would be 3 > > Eric > > On Fri, 26 Oct 2018 at 17:35 Travis Oliphant wrote: > > On Fri, Oct 26, 2018 at 7:14 PM Alex Rogozhnikov < >> alex.rogozhni...@yandex.ru> wrote: >> >>> > If the desire is to shrink the API of NumPy, I could see that. >>> >>> Very good desire, but my goal was different. >>> >> >>> > For some users this is exactly what is wanted. >>> >>> Maybe so, but I didn't face such example (and nobody mentioned those so >>> far in the discussion). >>> The opposite (according to the issue) happened. Mxnet example is >>> sufficient in my opinion. >>> >> >> I agree that the old motivation of APIs that would make it easy to create >> SciPy is no longer a major motivation for most users and even developers >> and so these reasons would not be very present (as well as why it wasn't >> even mentioned in the documentation). >> >> >>> Simple example: >>> x = np.zeros([]) >>> assert(x.flags.c_contiguous) >>> assert(np.ascontiguousarray(x).shape == x.shape) >>> >>> Behavior contradicts to documentation (shape is changed) and to name >>> (flags are saying - it is already c_contiguous) >>> >>> If you insist, that keeping ndmin=1 is important (I am not yet >>> convinced, but I am ready to believe your autority), >>> we can add ndmin=1 to functions' signatures, this way explicitly >>> notifying users about expected dimension. >>> >> >> I understand the lack of being convinced. This is ultimately a problem >> of 0-d arrays not being fully embraced and accepted by the Numeric >> community originally (which NumPy inherited during the early days). Is >> there a way to document functions that will be removed on a major version >> increase which don't print warnings on use? I would support this. >> >> I'm a big supporter of making a NumPy 2.0 and have been for several >> years. Now that Python 3 transition has happened, I think we could >> seriously discuss this. I'm trying to raise funding for maintenance and >> progress for NumPy and SciPy right now via Quansight Labs >> http://www.quansight.com/labs and I hope to be able to help find grants >> to support the wonderful efforts that have been happening for some time. >> >> While I'm thrilled and impressed by the number of amazing devs who have >> kept NumPy and SciPy going in mostly their spare time, it has created >> challenges that we have not had continuous maintenance funding to allow >> continuous paid development so that several people who know about the early >> decisions could not be retained to spend time on helping the transition. >> >> Your bringing the problem of mxnet devs is most appreciated. I will make >> a documentation PR. >> >> -Travis >> >> >> >> >>> Alex. >>> >>> >>> 27.10.2018, 02:27, "Travis Oliphant" : >>> >>> What is the justification for deprecation exactly? These functions have >>> been well documented and have had the intended behavior of producing arrays >>> with dimension at least 1 for some time. Why is it unexpected to produce >>> arrays of at least 1 dimension? For some users this is exactly what is >>> wanted. I don't understand the statement that behavior with 0-d arrays is >>> unexpected. >>> >>> If the desire is to shrink the API of NumPy, I could see that. But, it >>> seems odd to me to remove a much-used function with an established behavior >>> except as part of a wider API-shrinkage effort. >>> >>> 0-d arrays in NumPy are a separate conversation. At this point, I think >>> it was a mistake not to embrace 0-d arrays in NumPy from day one. In some >>> sense 0-d arrays *are* scalars at least conceptually and for JIT-producing >>> systems that exist now and will be
Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray
I would also argue against deprecating these functions that we are using increasingly in several projects that I am involved in. On Sat, Oct 27, 2018, 01:28 Travis Oliphant wrote: > What is the justification for deprecation exactly? These functions have > been well documented and have had the intended behavior of producing arrays > with dimension at least 1 for some time. Why is it unexpected to produce > arrays of at least 1 dimension? For some users this is exactly what is > wanted. I don't understand the statement that behavior with 0-d arrays is > unexpected. > > If the desire is to shrink the API of NumPy, I could see that. But, it > seems odd to me to remove a much-used function with an established behavior > except as part of a wider API-shrinkage effort. > > 0-d arrays in NumPy are a separate conversation. At this point, I think > it was a mistake not to embrace 0-d arrays in NumPy from day one. In some > sense 0-d arrays *are* scalars at least conceptually and for JIT-producing > systems that exist now and will be growing in the future, they can be > equivalent to scalars. > > The array scalars should become how you define what is *in* a NumPy array > making them true Python types, rather than Python 1-style "instances" of a > single "Dtype" object. You would then have 0-d arrays and these Python > "memory" types describing what is *in* the array. > > There is a clear way to do this, some of which has been outlined by > Nathaniel, and the rest I have an outline for how to implement. I can > advise someone on how to do this. > > -Travis > > > > > On Thu, Oct 25, 2018 at 3:17 PM Alex Rogozhnikov < > alex.rogozhni...@yandex.ru> wrote: > >> Dear numpy community, >> >> I'm planning to depreciate np.asfortranarray and np.ascontiguousarray >> functions due to their misbehavior on scalar (0-D tensors) with PR #12244 >> . >> >> Current behavior (converting scalars to 1-d array with single element) >> - is unexpected and contradicts to documentation >> - probably, can't be changed without breaking external code >> - I believe, this was a cause for poor support of 0-d arrays in mxnet. >> - both functions are easily replaced with asarray(..., order='...'), >> which has expected behavior >> >> There is no timeline for removal - we just need to discourage from using >> this functions in new code. >> >> Function naming may be related to how numpy treats 0-d tensors specially, >> >> and those probably should not be called arrays. >> https://www.numpy.org/neps/nep-0027-zero-rank-arrarys.html >> However, as a user I never thought about 0-d arrays being special and >> being "not arrays". >> >> >> Please see original discussion at github for more details >> https://github.com/numpy/numpy/issues/5300 >> >> Your comments welcome, >> Alex Rogozhnikov >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Reminder: weekly status meeting
> On Saturday, Oct 27, 2018 at 6:11 AM, Ralf Gommers (mailto:ralf.gomm...@gmail.com)> wrote: > > > On Sat, Oct 27, 2018 at 11:10 AM Stefan van der Walt (mailto:stef...@berkeley.edu)> wrote: > > On Sat, 27 Oct 2018 10:27:49 +1300, Ralf Gommers wrote: > > > Just to make sure we're talking about the same things here: Stefan, I > > > think > > > with "sparray" you mean "an n-D sparse array implementation that lives in > > > SciPy", nothing more specific? In that case pydata/sparse is the one > > > implementation, and including it in scipy.sparse would make it "sparray". > > > I'm currently indeed leaning towards depending on pydata/sparse rather > > > than > > > including it in scipy. > > > > I want to double check: when we last spoke, it seemed as though certain > > refactorings inside of SciPy (specifically, sparray was mentioned) would > > simplify the life of pydata/sparse devs. That no longer seems to be the > > case? > > There's no such thing as `sparray` anywhere in SciPy. There's two inactive > projects to create an n-D sparse array implementation, one of which is called > sparray (https://github.com/perimosocordiae/sparray). And there's one very > active project to do that same thing which is https://github.com/pydata/sparse > > > > > If our recommended route is to tell users to use pydata/sparse instead > > of SciPy (for the sparse array object), we probably want to get rid of > > our own internal implementation, and deprecate spmatrix > > Doc-deprecate I think; the sparse matrix classes in SciPy are very heavily > used, so it doesn't make sense to start emitting deprecation warnings for > them. But at some point we'll want to point users to pydata/sparse for new > code. > > > (or, build > > spmatrix on top of pydata/sparse)? > > It's the matrix vs. array semantics that are the issue, so not sure that > building one on top of the other would be useful. > > > > > Once we can define a clear API for sparse arrays, we can include some > > algorithms that ingest those objects in SciPy. But, I'm not sure we > > have an API in place that will allow handover of such objects to the > > existing C/FORTRAN-level code. > > I don't think the constructors for sparse matrix/array care about C/F order. > pydata/sparse is pure Python (and uses Numba). For reusing > scipy.sparse.linalg and scipy.sparse.csgraph you're right I think that that > will need some careful design work. Not sure anyone has thought about that in > a lot of detail yet. > They don’t yet. That is a planned feature, allowing an arbitrary permutation of input coordinates. > > There are interesting API questions probably, such as how to treat explicit > zeros (that debate still isn't settled for the matrix classes IIRC). > Explicit zeros are easier now, just use a fill_value of NaN and work with zeros as usual. Best Regards, Hameer Abbasi > > And there's an interesting transition puzzle to figure out (which also > includes np.matrix). At the moment the discussion on that is spread out over > many mailing list threads and Github issues, at some point we'll need to > summarize that. Probably around the time that the CSR/CSC replacement that > Hameer mentioned is finished. > > Cheers, > Ralf > > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Reminder: weekly status meeting
> On Saturday, Oct 27, 2018 at 12:10 AM, Stefan van der Walt > mailto:stef...@berkeley.edu)> wrote: > On Sat, 27 Oct 2018 10:27:49 +1300, Ralf Gommers wrote: > > Just to make sure we're talking about the same things here: Stefan, I think > > with "sparray" you mean "an n-D sparse array implementation that lives in > > SciPy", nothing more specific? In that case pydata/sparse is the one > > implementation, and including it in scipy.sparse would make it "sparray". > > I'm currently indeed leaning towards depending on pydata/sparse rather than > > including it in scipy. > > I want to double check: when we last spoke, it seemed as though certain > refactorings inside of SciPy (specifically, sparray was mentioned) would > simplify the life of pydata/sparse devs. That no longer seems to be the > case? Hi! I can’t recall having said this, perhaps you inferred it from the docs (it’s on the front page, so that isn’t unreasonable). We should update that sometime. That said, we use very little of scipy.sparse in PyData/Sparse. When Matt Rocklin was maintaining the project, that was the case, but even in the later days he shifted much of his code to pure NumPy. I followed that path further, not out of unwillingness to depend on it, but out of desire for generality. In its current state, the only things in PyData/Sparse that depend on scipy.sparse are: Conversion to/from scipy.sparse spmatrix classes A bit of linear algebra i.e. dot, tensordot, matmul. Best Regards, Hameer Abbasi > > If our recommended route is to tell users to use pydata/sparse instead > of SciPy (for the sparse array object), we probably want to get rid of > our own internal implementation, and deprecate spmatrix (or, build > spmatrix on top of pydata/sparse)? > > Once we can define a clear API for sparse arrays, we can include some > algorithms that ingest those objects in SciPy. But, I'm not sure we > have an API in place that will allow handover of such objects to the > existing C/FORTRAN-level code. > > Stéfan > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray
On Sat, Oct 27, 2018 at 6:37 PM Eric Wieser wrote: > in order to be used prior to calling C or Fortran code that expected at > least a 1-d array > > I’d argue that the behavior for these functions should have just been to > raise an error saying “this function does not support 0d arrays”, rather > than silently inserting extra dimensions. As a bonus, that would push the > function developers to add support for 0d. Obviously we can’t make it do > that now, but what we can do is have it emit a warning in those cases. > > I think our options are: > >1. Deprecate the entire function >2. Deprecate and eventually(?) throw an error upon calling the >function on 0d arrays, with a message like *“in future using >ascontiguousarray to promote 0d arrays to 1d arrays will not be supported. >If promotion is intentional, use ascontiguousarray(atleast1d(x)) to silence >this warning and keep the old behavior, and if not use asarray(x, >order='C') to preserve 0d arrays”* >3. Deprecate (future-warning) when passed 0d arrays, and eventually >skip the upcast to 1d. >If the calling code really needed a 1d array, then it will probably >fail, which is not really different to 2, but has the advantage that the >names are less surprising. >4. Only improve the documentation > > My preference would be 3 > I'd go for 4, or alternatively for the warning in 2 (which can be left in place indefinitely). 1 is unwarranted, and 3 will change behavior which is worse than just warning or stopping to support existing behavior (= 2). Eric > > On Fri, 26 Oct 2018 at 17:35 Travis Oliphant wrote: > > On Fri, Oct 26, 2018 at 7:14 PM Alex Rogozhnikov < >> alex.rogozhni...@yandex.ru> wrote: >> >>> > If the desire is to shrink the API of NumPy, I could see that. >>> >>> Very good desire, but my goal was different. >>> >> >>> > For some users this is exactly what is wanted. >>> >>> Maybe so, but I didn't face such example (and nobody mentioned those so >>> far in the discussion). >>> The opposite (according to the issue) happened. Mxnet example is >>> sufficient in my opinion. >>> >> >> I agree that the old motivation of APIs that would make it easy to create >> SciPy is no longer a major motivation for most users and even developers >> and so these reasons would not be very present (as well as why it wasn't >> even mentioned in the documentation). >> >> >>> Simple example: >>> x = np.zeros([]) >>> assert(x.flags.c_contiguous) >>> assert(np.ascontiguousarray(x).shape == x.shape) >>> >>> Behavior contradicts to documentation (shape is changed) and to name >>> (flags are saying - it is already c_contiguous) >>> >>> If you insist, that keeping ndmin=1 is important (I am not yet >>> convinced, but I am ready to believe your autority), >>> we can add ndmin=1 to functions' signatures, this way explicitly >>> notifying users about expected dimension. >>> >> >> I understand the lack of being convinced. This is ultimately a problem >> of 0-d arrays not being fully embraced and accepted by the Numeric >> community originally (which NumPy inherited during the early days). Is >> there a way to document functions that will be removed on a major version >> increase which don't print warnings on use? I would support this. >> > No, there's no such thing at the moment - the closest thing is https://github.com/numpy/numpy/wiki/Backwards-incompatible-ideas-for-a-major-release. I doubt we want such a thing anyway - removing functions without deprecation warnings first doesn't seem quite right. > >> I'm a big supporter of making a NumPy 2.0 and have been for several >> years. Now that Python 3 transition has happened, I think we could >> seriously discuss this. >> > I think it's more helpful to discuss goals and concrete plans for those, rather than a "NumPy 2.0" label. The latter never worked in the past, and not just because of lack of time/funding - it just means different things to different people. We now have a good start on what our major goals are ( http://www.numpy.org/neps/#roadmap), let's build on that. I'm trying to raise funding for maintenance and progress for NumPy and >> SciPy right now via Quansight Labs http://www.quansight.com/labs and I >> hope to be able to help find grants to support the wonderful efforts that >> have been happening for some time. >> > The NumPy grant and having Tyler/Matti/Stefan at BIDS is a great start to funded development; more and more diverse funding sources would be awesome. Cheers, Ralf >> While I'm thrilled and impressed by the number of amazing devs who have >> kept NumPy and SciPy going in mostly their spare time, it has created >> challenges that we have not had continuous maintenance funding to allow >> continuous paid development so that several people who know about the early >> decisions could not be retained to spend time on helping the transition. >> >> Your bringing the problem of mxnet devs is most appreciated. I will make >> a documentation PR.
