Re: [Numpy-discussion] Reminder: weekly status meeting

[Hameer Abbasi]

Hi everyone,

Like I said, we just use those to coerce SciPy arrays to native ones for 
compatibility. You could remove all those and the package would work fine, as 
long as you were using native PyData/Sparse arrays.

The only core functionality dependent on scipy.sparse is matrix multiplication 
and the like. Everything else is for inter-operability.

Best Regards,
Hameer Abbasi

> On Friday, Oct 26, 2018 at 1:19 AM, Tyler Reddy  (mailto:tyler.je.re...@gmail.com)> wrote:
> What exactly would you like Cython wrappers for? Some of the C++ code in 
> scipy/sparse/sparsetools?
>
> I see you have COO.from_scipy_sparse(x) in some pydata/sparse code paths, 
> which presumably you'd like to avoid or improve?
> On Thu, 25 Oct 2018 at 03:41, Hameer Abbasi  (mailto:einstein.edi...@gmail.com)> wrote:
> > Hi!
> >
> > Sorry to miss this week’s meeting.
> >
> > If I may point out an inaccuracy in the notes: in PyData/Sparse most things 
> > are implemented from the ground up without relying on scipy.sparse. The 
> > only part that does rely on it is `sparse.matmul`, `sparse.dot` and 
> > `sparse.tensordot`, as well as a few conversions to/from SciPy, if these 
> > could depend on Cython wrappers instead that’d be nice.
> >
> > I should probably update the docs on that. If anyone is willing to discuss 
> > pydata/sparse with me, I’ll be available for a meeting anytime.
> >
> > Best Regards,
> > Hameer Abbasi
> >
> >
> > > On Thursday, Oct 25, 2018 at 12:08 AM, Stefan van der Walt 
> > > mailto:stef...@berkeley.edu)> wrote:
> > > Hi all,
> > >
> > > On Mon, 22 Oct 2018 09:56:37 +0300, Matti Picus wrote:
> > > > We therefore invite you to join us for our weekly calls,
> > > > each **Wednesday from 12:00 to 13:00 Pacific Time**.
> > > >
> > > > Detail of the next meeting (2018-10-24) is given in the agenda
> > >
> > > This week's meeting notes are at:
> > >
> > > https://github.com/BIDS-numpy/docs/blob/master/status_meetings/status-2018-10-24.md
> > >
> > > Stéfan
> > > ___
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion@python.org (mailto:NumPy-Discussion@python.org)
> > > https://mail.python.org/mailman/listinfo/numpy-discussion
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@python.org (mailto:NumPy-Discussion@python.org)
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reminder: weekly status meeting

[Stefan van der Walt]

Hi Hameer,

On Fri, 26 Oct 2018 10:47:09 +0200, Hameer Abbasi wrote:
> The only core functionality dependent on scipy.sparse is matrix
> multiplication and the like. Everything else is for inter-operability.

Thank you for commenting here.

As you know, I am enthusiastic about seeing an `sparray` equivalent to
`spmatrix`.  When we last spoke, my recollection was that it would be
beneficial to `pydata/sparse`.  Is this still correct?

If not, are we now in a situation where it would be more helpful to
build `sparray` based on `pydata/sparse`.

If we can have a good sparse array API in place in SciPy, it may
significantly simplify code in various other libraries (I'm thinking of
scikit-learn, e.g.).

Best regards,
Stéfan
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reminder: weekly status meeting

[Hameer Abbasi]

Hi Stefan!

PyData/Sparse is pretty far along, by January or so we should have a CSR/CSC 
replacement that is ND. It needs optimisation in a lot of cases but the API is 
compatible with NumPy and works pretty well already IMO.

PyData/Sparse is pretty much independent of any changes to scipy.sparse at this 
point. We build on top of NumPy, not scipy.sparse.

Feel free to use any or all of my code for sparray, although I think Ralf 
Gommers, Matthew Rocklin and others were of the opinion that the data structure 
should stay in PyData/Sparse and linear algebra and csgraph etc should go into 
SciPy.

Best Regards,
Hameer Abbasi

> On Friday, Oct 26, 2018 at 7:03 PM, Stefan van der Walt  (mailto:stef...@berkeley.edu)> wrote:
> Hi Hameer,
>
> On Fri, 26 Oct 2018 10:47:09 +0200, Hameer Abbasi wrote:
> > The only core functionality dependent on scipy.sparse is matrix
> > multiplication and the like. Everything else is for inter-operability.
>
> Thank you for commenting here.
>
> As you know, I am enthusiastic about seeing an `sparray` equivalent to
> `spmatrix`. When we last spoke, my recollection was that it would be
> beneficial to `pydata/sparse`. Is this still correct?
>
> If not, are we now in a situation where it would be more helpful to
> build `sparray` based on `pydata/sparse`.
>
> If we can have a good sparse array API in place in SciPy, it may
> significantly simplify code in various other libraries (I'm thinking of
> scikit-learn, e.g.).
>
> Best regards,
> Stéfan
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray

[Alex Rogozhnikov]

 The conservative way to handle this would be to do a deprecation cycle, specifically by issuing FutureWarning when scalars or 0d arrays are encountered as inputs.Sounds good to me. Behavior should be scheduled for numpy 1.18?   26.10.2018, 05:02, "Stephan Hoyer" :On Thu, Oct 25, 2018 at 3:10 PM Andras Deak  wrote:On Thu, Oct 25, 2018 at 11:48 PM Joseph Fox-Rabinovitz wrote:>> In that vein, would it be advisable to re-implement them as aliases for the correctly behaving functions instead?>> - JoeWouldn't "probably, can't be changed without breaking external code"still apply? As I understand the suggestion for _deprecation_ is onlybecause there's (a lot of) code relying on the current behaviour (orat least there's risk). I would also advocate for fixing these functions if possible (removing ndim=1). ascontiguousarray(...) is certainly more readable than asarray(... order='C'). The conservative way to handle this would be to do a deprecation cycle, specifically by issuing FutureWarning when scalars or 0d arrays are encountered as inputs. Cheers,Stephan,___NumPy-Discussion mailing listNumPy-Discussion@python.orghttps://mail.python.org/mailman/listinfo/numpy-discussion___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray

[Stefan van der Walt]

On Thu, 25 Oct 2018 19:02:20 -0700, Stephan Hoyer wrote:
> I would also advocate for fixing these functions if possible (removing
> ndim=1). ascontiguousarray(...) is certainly more readable than asarray(...
> order='C').

I agree; these are widely used, and makes intuitive sense as part of the
API.

Stéfan
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray

[Stephan Hoyer]

On Fri, Oct 26, 2018 at 12:55 PM Alex Rogozhnikov <
alex.rogozhni...@yandex.ru> wrote:

>
> The conservative way to handle this would be to do a deprecation cycle,
> specifically by issuing FutureWarning when scalars or 0d arrays are
> encountered as inputs.
> Sounds good to me. Behavior should be scheduled for numpy 1.18?
>

Yes, that sounds about right to me.
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reminder: weekly status meeting

[Ralf Gommers]

On Sat, Oct 27, 2018 at 6:10 AM Hameer Abbasi 
wrote:

> Hi Stefan!
>
> PyData/Sparse is pretty far along, by January or so we should have a
> CSR/CSC replacement that is ND. It needs optimisation in a lot of cases but
> the API is compatible with NumPy and works pretty well already IMO.
>
> PyData/Sparse is pretty much independent of any changes to scipy.sparse at
> this point. We build on top of NumPy, not scipy.sparse.
>
> Feel free to use any or all of my code for sparray, although I think Ralf
> Gommers, Matthew Rocklin and others were of the opinion that the data
> structure should stay in PyData/Sparse and linear algebra and csgraph etc
> should go into SciPy.
>

Just to make sure we're talking about the same things here: Stefan, I think
with "sparray" you mean "an n-D sparse array implementation that lives in
SciPy", nothing more specific? In that case pydata/sparse is the one
implementation, and including it in scipy.sparse would make it "sparray".
I'm currently indeed leaning towards depending on pydata/sparse rather than
including it in scipy.

Cheers,
Ralf



> Best Regards,
> Hameer Abbasi
>
> On Friday, Oct 26, 2018 at 7:03 PM, Stefan van der Walt <
> stef...@berkeley.edu> wrote:
> Hi Hameer,
>
> On Fri, 26 Oct 2018 10:47:09 +0200, Hameer Abbasi wrote:
>
> The only core functionality dependent on scipy.sparse is matrix
> multiplication and the like. Everything else is for inter-operability.
>
>
> Thank you for commenting here.
>
> As you know, I am enthusiastic about seeing an `sparray` equivalent to
> `spmatrix`. When we last spoke, my recollection was that it would be
> beneficial to `pydata/sparse`. Is this still correct?
>
> If not, are we now in a situation where it would be more helpful to
> build `sparray` based on `pydata/sparse`.
>
> If we can have a good sparse array API in place in SciPy, it may
> significantly simplify code in various other libraries (I'm thinking of
> scikit-learn, e.g.).
>
> Best regards,
> Stéfan
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reminder: weekly status meeting

[Stefan van der Walt]

On Sat, 27 Oct 2018 10:27:49 +1300, Ralf Gommers wrote:
> Just to make sure we're talking about the same things here: Stefan, I think
> with "sparray" you mean "an n-D sparse array implementation that lives in
> SciPy", nothing more specific? In that case pydata/sparse is the one
> implementation, and including it in scipy.sparse would make it "sparray".
> I'm currently indeed leaning towards depending on pydata/sparse rather than
> including it in scipy.

I want to double check: when we last spoke, it seemed as though certain
refactorings inside of SciPy (specifically, sparray was mentioned) would
simplify the life of pydata/sparse devs.  That no longer seems to be the
case?

If our recommended route is to tell users to use pydata/sparse instead
of SciPy (for the sparse array object), we probably want to get rid of
our own internal implementation, and deprecate spmatrix (or, build
spmatrix on top of pydata/sparse)?

Once we can define a clear API for sparse arrays, we can include some
algorithms that ingest those objects in SciPy.  But, I'm not sure we
have an API in place that will allow handover of such objects to the
existing C/FORTRAN-level code.

Stéfan
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray

[Sebastian Berg]

On Fri, 2018-10-26 at 13:25 -0700, Stephan Hoyer wrote:
> On Fri, Oct 26, 2018 at 12:55 PM Alex Rogozhnikov <
> alex.rogozhni...@yandex.ru> wrote:
> >  
> > The conservative way to handle this would be to do a deprecation
> > cycle, specifically by issuing FutureWarning when scalars or 0d
> > arrays are encountered as inputs.
> > Sounds good to me. Behavior should be scheduled for numpy 1.18?
> > 
> 
> Yes, that sounds about right to me.
> 

Is there a way to avoid the future warning? An unavoidable warning in a
widely used function seems really annoying to me. Unless, the 0d thing
happens rarely, but then it might be the downstream users that get the
warning for no reason.

- Sebastian


>  
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion


signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray

[Stephan Hoyer]

On Fri, Oct 26, 2018 at 3:48 PM Sebastian Berg 
wrote:

> On Fri, 2018-10-26 at 13:25 -0700, Stephan Hoyer wrote:
> > On Fri, Oct 26, 2018 at 12:55 PM Alex Rogozhnikov <
> > alex.rogozhni...@yandex.ru> wrote:
> > >
> > > The conservative way to handle this would be to do a deprecation
> > > cycle, specifically by issuing FutureWarning when scalars or 0d
> > > arrays are encountered as inputs.
> > > Sounds good to me. Behavior should be scheduled for numpy 1.18?
> > >
> >
> > Yes, that sounds about right to me.
> >
>
> Is there a way to avoid the future warning? An unavoidable warning in a
> widely used function seems really annoying to me. Unless, the 0d thing
> happens rarely, but then it might be the downstream users that get the
> warning for no reason.
>
> - Sebastian
>

My suspicion is that 0d arrays are rarely used as arguments to
ascontiguousarray / asfortranarray. But it's hard to say for sure...
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray

[Travis Oliphant]

What is the justification for deprecation exactly?  These functions have
been well documented and have had the intended behavior of producing arrays
with dimension at least 1 for some time.  Why is it unexpected to produce
arrays of at least 1 dimension?  For some users this is exactly what is
wanted.  I don't understand the statement that behavior with 0-d arrays is
unexpected.

If the desire is to shrink the API of NumPy, I could see that.   But, it
seems odd to me to remove a much-used function with an established behavior
except as part of a wider API-shrinkage effort.

0-d arrays in NumPy are a separate conversation.  At this point, I think it
was a mistake not to embrace 0-d arrays in NumPy from day one.  In some
sense 0-d arrays *are* scalars at least conceptually and for JIT-producing
systems that exist now and will be growing in the future, they can be
equivalent to scalars.

The array scalars should become how you define what is *in* a NumPy array
making them true Python types, rather than Python 1-style "instances" of a
single "Dtype" object.  You would then have 0-d arrays and these Python
"memory" types describing what is *in* the array.

There is a clear way to do this, some of which has been outlined by
Nathaniel, and the rest I have an outline for how to implement.  I can
advise someone on how to do this.

-Travis




On Thu, Oct 25, 2018 at 3:17 PM Alex Rogozhnikov 
wrote:

> Dear numpy community,
>
> I'm planning to depreciate np.asfortranarray and np.ascontiguousarray
> functions due to their misbehavior on scalar (0-D tensors) with PR #12244.
>
> Current behavior (converting scalars to 1-d array with single element)
> - is unexpected and contradicts to documentation
> - probably, can't be changed without breaking external code
> - I believe, this was a cause for poor support of 0-d arrays in mxnet.
> - both functions are easily replaced with asarray(..., order='...'), which
> has expected behavior
>
> There is no timeline for removal - we just need to discourage from using
> this functions in new code.
>
> Function naming may be related to how numpy treats 0-d tensors specially,
> and those probably should not be called arrays.
> https://www.numpy.org/neps/nep-0027-zero-rank-arrarys.html
> However, as a user I never thought about 0-d arrays being special and
> being "not arrays".
>
>
> Please see original discussion at github for more details
> https://github.com/numpy/numpy/issues/5300
>
> Your comments welcome,
> Alex Rogozhnikov
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray

[Travis Oliphant]

I see now the original motivation as the unfortunate situation that mxnet
authors did not understand that np.ascontiguousarray returned an array of
at least one dimension and perhaps used that one API to assume that NumPy
did not support 0-d arrays --- which NumPy does indeed support.

Certainly that situation would motivate a documentation change to help
steer other future users from making the same incorrect assumption, but
deprecation is a separate question entirely.  I do not agree at all with
the trend to remove functions from NumPy API prior to a dedicated NumPy 2.0
effort.  This breaks the idea of semantic versioning for NumPy.

These functions do, in fact, have a use and were very much intended to
produce one-dimensional arrays --- in order to be used prior to calling C
or Fortran code that expected at least a 1-d array.A lot of the SciPy
wrapping code needed this behavior.  It is a misinterpretation to assume
this is buggy or unintended.

Improving the documentation to warn about the behavior for 0-d arrays could
indeed be useful.

-Travis


On Fri, Oct 26, 2018 at 6:27 PM Stephan Hoyer  wrote:

> On Fri, Oct 26, 2018 at 3:48 PM Sebastian Berg 
> wrote:
>
>> On Fri, 2018-10-26 at 13:25 -0700, Stephan Hoyer wrote:
>> > On Fri, Oct 26, 2018 at 12:55 PM Alex Rogozhnikov <
>> > alex.rogozhni...@yandex.ru> wrote:
>> > >
>> > > The conservative way to handle this would be to do a deprecation
>> > > cycle, specifically by issuing FutureWarning when scalars or 0d
>> > > arrays are encountered as inputs.
>> > > Sounds good to me. Behavior should be scheduled for numpy 1.18?
>> > >
>> >
>> > Yes, that sounds about right to me.
>> >
>>
>> Is there a way to avoid the future warning? An unavoidable warning in a
>> widely used function seems really annoying to me. Unless, the 0d thing
>> happens rarely, but then it might be the downstream users that get the
>> warning for no reason.
>>
>> - Sebastian
>>
>
> My suspicion is that 0d arrays are rarely used as arguments to
> ascontiguousarray / asfortranarray. But it's hard to say for sure...
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray

[Alex Rogozhnikov]

> If the desire is to shrink the API of NumPy, I could see that. Very good desire, but my goal was different. > For some users this is exactly what is wanted. Maybe so, but I didn't face such example (and nobody mentioned those so far in the discussion).The opposite (according to the issue) happened. Mxnet example is sufficient in my opinion. Simple example:x = np.zeros([])assert(x.flags.c_contiguous)assert(np.ascontiguousarray(x).shape == x.shape) Behavior contradicts to documentation (shape is changed) and to name (flags are saying - it is already c_contiguous) If you insist, that keeping ndmin=1 is important (I am not yet convinced, but I am ready to believe your autority),we can add ndmin=1 to functions' signatures, this way explicitly notifying users about expected dimension. Alex.  27.10.2018, 02:27, "Travis Oliphant" :What is the justification for deprecation exactly?  These functions have been well documented and have had the intended behavior of producing arrays with dimension at least 1 for some time.  Why is it unexpected to produce arrays of at least 1 dimension?  For some users this is exactly what is wanted.  I don't understand the statement that behavior with 0-d arrays is unexpected. If the desire is to shrink the API of NumPy, I could see that.   But, it seems odd to me to remove a much-used function with an established behavior except as part of a wider API-shrinkage effort.  0-d arrays in NumPy are a separate conversation.  At this point, I think it was a mistake not to embrace 0-d arrays in NumPy from day one.  In some sense 0-d arrays *are* scalars at least conceptually and for JIT-producing systems that exist now and will be growing in the future, they can be equivalent to scalars.   The array scalars should become how you define what is *in* a NumPy array making them true Python types, rather than Python 1-style "instances" of a single "Dtype" object.  You would then have 0-d arrays and these Python "memory" types describing what is *in* the array.   There is a clear way to do this, some of which has been outlined by Nathaniel, and the rest I have an outline for how to implement.  I can advise someone on how to do this.   -TravisOn Thu, Oct 25, 2018 at 3:17 PM Alex Rogozhnikov  wrote:Dear numpy community, I'm planning to depreciate np.asfortranarray and np.ascontiguousarrayfunctions due to their misbehavior on scalar (0-D tensors) with PR #12244. Current behavior (converting scalars to 1-d array with single element)- is unexpected and contradicts to documentation- probably, can't be changed without breaking external code- I believe, this was a cause for poor support of 0-d arrays in mxnet.- both functions are easily replaced with asarray(..., order='...'), which has expected behavior There is no timeline for removal - we just need to discourage from using this functions in new code. Function naming may be related to how numpy treats 0-d tensors specially,  and those probably should not be called arrays.https://www.numpy.org/neps/nep-0027-zero-rank-arrarys.htmlHowever, as a user I never thought about 0-d arrays being special and being "not arrays".  Please see original discussion at github for more detailshttps://github.com/numpy/numpy/issues/5300 Your comments welcome,Alex Rogozhnikov ___NumPy-Discussion mailing listNumPy-Discussion@python.orghttps://mail.python.org/mailman/listinfo/numpy-discussion,___NumPy-Discussion mailing listNumPy-Discussion@python.orghttps://mail.python.org/mailman/listinfo/numpy-discussion___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray

[Travis Oliphant]

On Fri, Oct 26, 2018 at 7:14 PM Alex Rogozhnikov 
wrote:

> > If the desire is to shrink the API of NumPy, I could see that.
>
> Very good desire, but my goal was different.
>

> > For some users this is exactly what is wanted.
>
> Maybe so, but I didn't face such example (and nobody mentioned those so
> far in the discussion).
> The opposite (according to the issue) happened. Mxnet example is
> sufficient in my opinion.
>

I agree that the old motivation of APIs that would make it easy to create
SciPy is no longer a major motivation for most users and even developers
and so these reasons would not be very present (as well as why it wasn't
even mentioned in the documentation).


> Simple example:
> x = np.zeros([])
> assert(x.flags.c_contiguous)
> assert(np.ascontiguousarray(x).shape == x.shape)
>
> Behavior contradicts to documentation (shape is changed) and to name
> (flags are saying - it is already c_contiguous)
>
> If you insist, that keeping ndmin=1 is important (I am not yet convinced,
> but I am ready to believe your autority),
> we can add ndmin=1 to functions' signatures, this way explicitly notifying
> users about expected dimension.
>

I understand the lack of being convinced.  This is ultimately a problem of
0-d arrays not being fully embraced and accepted by the Numeric community
originally (which NumPy inherited during the early days).   Is there a way
to document functions that will be removed on a major version increase
which don't print warnings on use? I would support this.

I'm a big supporter of making a NumPy 2.0 and have been for several years.
Now that Python 3 transition has happened, I think we could seriously
discuss this.  I'm trying to raise funding for maintenance and progress for
NumPy and SciPy right now via Quansight Labs http://www.quansight.com/labs
and I hope to be able to help find grants to support the wonderful efforts
that have been happening for some time.

While I'm thrilled and impressed by the number of amazing devs who have
kept NumPy and SciPy going in mostly their spare time, it has created
challenges that we have not had continuous maintenance funding to allow
continuous paid development so that several people who know about the early
decisions could not be retained to spend time on helping the transition.

Your bringing the problem of mxnet devs is most appreciated.  I will make a
documentation PR.

-Travis




> Alex.
>
>
> 27.10.2018, 02:27, "Travis Oliphant" :
>
> What is the justification for deprecation exactly?  These functions have
> been well documented and have had the intended behavior of producing arrays
> with dimension at least 1 for some time.  Why is it unexpected to produce
> arrays of at least 1 dimension?  For some users this is exactly what is
> wanted.  I don't understand the statement that behavior with 0-d arrays is
> unexpected.
>
> If the desire is to shrink the API of NumPy, I could see that.   But, it
> seems odd to me to remove a much-used function with an established behavior
> except as part of a wider API-shrinkage effort.
>
> 0-d arrays in NumPy are a separate conversation.  At this point, I think
> it was a mistake not to embrace 0-d arrays in NumPy from day one.  In some
> sense 0-d arrays *are* scalars at least conceptually and for JIT-producing
> systems that exist now and will be growing in the future, they can be
> equivalent to scalars.
>
> The array scalars should become how you define what is *in* a NumPy array
> making them true Python types, rather than Python 1-style "instances" of a
> single "Dtype" object.  You would then have 0-d arrays and these Python
> "memory" types describing what is *in* the array.
>
> There is a clear way to do this, some of which has been outlined by
> Nathaniel, and the rest I have an outline for how to implement.  I can
> advise someone on how to do this.
>
> -Travis
>
>
>
>
> On Thu, Oct 25, 2018 at 3:17 PM Alex Rogozhnikov <
> alex.rogozhni...@yandex.ru> wrote:
>
> Dear numpy community,
>
> I'm planning to depreciate np.asfortranarray and np.ascontiguousarray
> functions due to their misbehavior on scalar (0-D tensors) with PR #12244.
>
> Current behavior (converting scalars to 1-d array with single element)
> - is unexpected and contradicts to documentation
> - probably, can't be changed without breaking external code
> - I believe, this was a cause for poor support of 0-d arrays in mxnet.
> - both functions are easily replaced with asarray(..., order='...'), which
> has expected behavior
>
> There is no timeline for removal - we just need to discourage from using
> this functions in new code.
>
> Function naming may be related to how numpy treats 0-d tensors specially,
> and those probably should not be called arrays.
> https://www.numpy.org/neps/nep-0027-zero-rank-arrarys.html
> However, as a user I never thought about 0-d arrays being special and
> being "not arrays".
>
>
> Please see original discussion at github for more details
> https://github.com/numpy/numpy/issues/5300
>
> Y

Re: [Numpy-discussion] asanyarray vs. asarray

[Travis Oliphant]

On Fri, Oct 19, 2018 at 8:24 PM Marten van Kerkwijk <
m.h.vankerkw...@gmail.com> wrote:

> Hi All,
>
> It seems there are two extreme possibilities for general functions:
> 1. Put `asarray` everywhere. The main benefit that I can see is that even
> if people put in list instead of arrays, one is guaranteed to have shape,
> dtype, etc. But it seems a bit like calling `int` on everything that might
> get used as an index, instead of letting the actual indexing do the proper
> thing and call `__index__`.
>

Yes, actually getting a proper "array protocol" into Python would be a
fantastic approach.   We have been working with Lenore Mullin who is a
researcher on the mathematics of arrays on what it means to be an array and
believe we can come up with an actual array protocol that perhaps could be
put into Python itself (though that isn't our immediate goal right now).



> 2. Do not coerce at all, but rather write code assuming something is an
> array already. This will often, but not always, just work for array mimics,
> with coercion done only where necessary (e.g., in lower-lying C code such
> as that of the ufuncs which has a smaller API surface and can be overridden
> more easily).
>
> The current __array_function__ work may well provide us with a way to
> combine both, if we (over time) move the coercion inside
> `ndarray.__array_function__` so that the actual implementation *can* assume
> it deals with pure ndarray - then, when relevant, calling that
> implementation will be what subclasses/duck arrays can happily do (and it
> is up to them to ensure this works).
>

Also, we could get rid of asarray entirely by changing expectations.  This
automatic conversion code throughout NumPy and SciPy is an example of the
confusion in both of these libraries between "user-oriented interfaces" and
"developer-oriented interfaces".   A developer just wants the library to
use duck-typing and then raise errors if you don't provide the right type
(i.e. a list instead of an array).  The user-interface could happen in
Jupyter, or be isolated to a high-level library or meta-code approach (of
which there are several possibilities for Python).



>
> Of course, the above does not really answer what to do in the meantime.
> But perhaps it helps in thinking of what we are actually aiming for.
>

> One last thing: could we please stop bashing subclasses? One can subclass
> essentially everything in python, often to great advantage. Subclasses such
> as MaskedArray and, yes, Quantity, are widely used, and if they cause
> problems perhaps that should be seen as a sign that ndarray subclassing
> should be made easier and clearer.
>
>
I agree that we can stop bashing subclasses in general.   The problem with
numpy subclasses is that they were made without adherence to SOLID:
https://en.wikipedia.org/wiki/SOLID.  In particular the Liskov substitution
principle:  https://en.wikipedia.org/wiki/Liskov_substitution_principle .
Much of this is my fault.  Being a scientist/engineer more than a computer
scientist, I had no idea what these principles were and did not properly
apply them in creating np.matrix which clearly violates the substitution
principle.

We can clean all this and more up.

But, we really need to start talking about NumPy 2.0 to do it.   Now that
Python 3.x is really here, we can raise the money for it and get it done.
We don't have to just rely on volunteer time.

The world will thank us for actually pushing NumPy 2.0.  I know not
everyone agrees, but for whatever its worth, I feel very, very strongly
about this, and despite not being very active on this list for the past
years, I do have a lot of understanding about how the current code actually
works (and where and why its warts are).

-Travis






> All the best,
>
> Marten
>
>
> On Fri, Oct 19, 2018 at 7:02 PM Ralf Gommers 
> wrote:
>
>>
>>
>> On Fri, Oct 19, 2018 at 10:28 PM Ralf Gommers 
>> wrote:
>>
>>>
>>>
>>> On Fri, Oct 19, 2018 at 4:15 PM Hameer Abbasi 
>>> wrote:
>>>
 Hi!

 On Friday, Oct 19, 2018 at 6:09 PM, Stephan Hoyer 
 wrote:
 I don't think it makes much sense to change NumPy's existing usage of
 asarray() to asanyarray() unless we add subok=True arguments (which default
 to False). But this ends up cluttering NumPy's public API, which is also
 undesirable.

 Agreed so far.

>>>
>>> I'm not sure I agree. "subok" is very unpythonic; the average numpy
>>> library function should work fine for a well-behaved subclass (i.e. most
>>> things out there except np.matrix).
>>>

 The preferred way to override NumPy functions going forward should be
 __array_function__.


 I think we should “soft support” i.e. allow but consider unsupported,
 the case where one of NumPy’s functions is implemented in terms of others
 and “passing through” an array results in the correct behaviour for that
 array.

>>>
>>> I don't think we have or want such a concept as "soft support". We
>>> inte

Re: [Numpy-discussion] Reminder: weekly status meeting

[Ralf Gommers]

On Sat, Oct 27, 2018 at 11:10 AM Stefan van der Walt 
wrote:

> On Sat, 27 Oct 2018 10:27:49 +1300, Ralf Gommers wrote:
> > Just to make sure we're talking about the same things here: Stefan, I
> think
> > with "sparray" you mean "an n-D sparse array implementation that lives in
> > SciPy", nothing more specific? In that case pydata/sparse is the one
> > implementation, and including it in scipy.sparse would make it "sparray".
> > I'm currently indeed leaning towards depending on pydata/sparse rather
> than
> > including it in scipy.
>
> I want to double check: when we last spoke, it seemed as though certain
> refactorings inside of SciPy (specifically, sparray was mentioned) would
> simplify the life of pydata/sparse devs.  That no longer seems to be the
> case?
>

There's no such thing as `sparray` anywhere in SciPy. There's two inactive
projects to create an n-D sparse array implementation, one of which is
called sparray (https://github.com/perimosocordiae/sparray). And there's
one very active project to do that same thing which is
https://github.com/pydata/sparse


> If our recommended route is to tell users to use pydata/sparse instead
> of SciPy (for the sparse array object), we probably want to get rid of
> our own internal implementation, and deprecate spmatrix


Doc-deprecate I think; the sparse matrix classes in SciPy are very heavily
used, so it doesn't make sense to start emitting deprecation warnings for
them. But at some point we'll want to point users to pydata/sparse for new
code.


> (or, build
> spmatrix on top of pydata/sparse)?
>

It's the matrix vs. array semantics that are the issue, so not sure that
building one on top of the other would be useful.


> Once we can define a clear API for sparse arrays, we can include some
> algorithms that ingest those objects in SciPy.  But, I'm not sure we
> have an API in place that will allow handover of such objects to the
> existing C/FORTRAN-level code.
>

I don't think the constructors for sparse matrix/array care about C/F
order. pydata/sparse is pure Python (and uses Numba). For reusing
scipy.sparse.linalg and scipy.sparse.csgraph you're right I think that that
will need some careful design work. Not sure anyone has thought about that
in a lot of detail yet.

There are interesting API questions probably, such as how to treat explicit
zeros (that debate still isn't settled for the matrix classes IIRC). And
there's an interesting transition puzzle to figure out (which also includes
np.matrix). At the moment the discussion on that is spread out over many
mailing list threads and Github issues, at some point we'll need to
summarize that. Probably around the time that the CSR/CSC replacement that
Hameer mentioned is finished.

Cheers,
Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray

[Eric Wieser]

in order to be used prior to calling C or Fortran code that expected at
least a 1-d array

I’d argue that the behavior for these functions should have just been to
raise an error saying “this function does not support 0d arrays”, rather
than silently inserting extra dimensions. As a bonus, that would push the
function developers to add support for 0d. Obviously we can’t make it do
that now, but what we can do is have it emit a warning in those cases.

I think our options are:

   1. Deprecate the entire function
   2. Deprecate and eventually(?) throw an error upon calling the function
   on 0d arrays, with a message like *“in future using ascontiguousarray to
   promote 0d arrays to 1d arrays will not be supported. If promotion is
   intentional, use ascontiguousarray(atleast1d(x)) to silence this warning
   and keep the old behavior, and if not use asarray(x, order='C') to preserve
   0d arrays”*
   3. Deprecate (future-warning) when passed 0d arrays, and eventually skip
   the upcast to 1d.
   If the calling code really needed a 1d array, then it will probably
   fail, which is not really different to 2, but has the advantage that the
   names are less surprising.
   4. Only improve the documentation

My preference would be 3

Eric

On Fri, 26 Oct 2018 at 17:35 Travis Oliphant  wrote:

On Fri, Oct 26, 2018 at 7:14 PM Alex Rogozhnikov 
> wrote:
>
>> > If the desire is to shrink the API of NumPy, I could see that.
>>
>> Very good desire, but my goal was different.
>>
>
>> > For some users this is exactly what is wanted.
>>
>> Maybe so, but I didn't face such example (and nobody mentioned those so
>> far in the discussion).
>> The opposite (according to the issue) happened. Mxnet example is
>> sufficient in my opinion.
>>
>
> I agree that the old motivation of APIs that would make it easy to create
> SciPy is no longer a major motivation for most users and even developers
> and so these reasons would not be very present (as well as why it wasn't
> even mentioned in the documentation).
>
>
>> Simple example:
>> x = np.zeros([])
>> assert(x.flags.c_contiguous)
>> assert(np.ascontiguousarray(x).shape == x.shape)
>>
>> Behavior contradicts to documentation (shape is changed) and to name
>> (flags are saying - it is already c_contiguous)
>>
>> If you insist, that keeping ndmin=1 is important (I am not yet convinced,
>> but I am ready to believe your autority),
>> we can add ndmin=1 to functions' signatures, this way explicitly
>> notifying users about expected dimension.
>>
>
> I understand the lack of being convinced.  This is ultimately a problem of
> 0-d arrays not being fully embraced and accepted by the Numeric community
> originally (which NumPy inherited during the early days).   Is there a way
> to document functions that will be removed on a major version increase
> which don't print warnings on use? I would support this.
>
> I'm a big supporter of making a NumPy 2.0 and have been for several years.
> Now that Python 3 transition has happened, I think we could seriously
> discuss this.  I'm trying to raise funding for maintenance and progress for
> NumPy and SciPy right now via Quansight Labs http://www.quansight.com/labs
> and I hope to be able to help find grants to support the wonderful efforts
> that have been happening for some time.
>
> While I'm thrilled and impressed by the number of amazing devs who have
> kept NumPy and SciPy going in mostly their spare time, it has created
> challenges that we have not had continuous maintenance funding to allow
> continuous paid development so that several people who know about the early
> decisions could not be retained to spend time on helping the transition.
>
> Your bringing the problem of mxnet devs is most appreciated.  I will make
> a documentation PR.
>
> -Travis
>
>
>
>
>> Alex.
>>
>>
>> 27.10.2018, 02:27, "Travis Oliphant" :
>>
>> What is the justification for deprecation exactly?  These functions have
>> been well documented and have had the intended behavior of producing arrays
>> with dimension at least 1 for some time.  Why is it unexpected to produce
>> arrays of at least 1 dimension?  For some users this is exactly what is
>> wanted.  I don't understand the statement that behavior with 0-d arrays is
>> unexpected.
>>
>> If the desire is to shrink the API of NumPy, I could see that.   But, it
>> seems odd to me to remove a much-used function with an established behavior
>> except as part of a wider API-shrinkage effort.
>>
>> 0-d arrays in NumPy are a separate conversation.  At this point, I think
>> it was a mistake not to embrace 0-d arrays in NumPy from day one.  In some
>> sense 0-d arrays *are* scalars at least conceptually and for JIT-producing
>> systems that exist now and will be growing in the future, they can be
>> equivalent to scalars.
>>
>> The array scalars should become how you define what is *in* a NumPy array
>> making them true Python types, rather than Python 1-style "instances" of a
>> single "Dtype" object.  You would the

Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray

[Ralf Gommers]

On Sat, Oct 27, 2018 at 6:37 PM Eric Wieser 
wrote:

> in order to be used prior to calling C or Fortran code that expected at
> least a 1-d array
>
> I’d argue that the behavior for these functions should have just been to
> raise an error saying “this function does not support 0d arrays”, rather
> than silently inserting extra dimensions. As a bonus, that would push the
> function developers to add support for 0d. Obviously we can’t make it do
> that now, but what we can do is have it emit a warning in those cases.
>
> I think our options are:
>
>1. Deprecate the entire function
>2. Deprecate and eventually(?) throw an error upon calling the
>function on 0d arrays, with a message like *“in future using
>ascontiguousarray to promote 0d arrays to 1d arrays will not be supported.
>If promotion is intentional, use ascontiguousarray(atleast1d(x)) to silence
>this warning and keep the old behavior, and if not use asarray(x,
>order='C') to preserve 0d arrays”*
>3. Deprecate (future-warning) when passed 0d arrays, and eventually
>skip the upcast to 1d.
>If the calling code really needed a 1d array, then it will probably
>fail, which is not really different to 2, but has the advantage that the
>names are less surprising.
>4. Only improve the documentation
>
> My preference would be 3
>
I'd go for 4, or alternatively for the warning in 2 (which can be left in
place indefinitely). 1 is unwarranted, and 3 will change behavior which is
worse than just warning or stopping to support existing behavior (= 2).

Eric
>
> On Fri, 26 Oct 2018 at 17:35 Travis Oliphant  wrote:
>
> On Fri, Oct 26, 2018 at 7:14 PM Alex Rogozhnikov <
>> alex.rogozhni...@yandex.ru> wrote:
>>
>>> > If the desire is to shrink the API of NumPy, I could see that.
>>>
>>> Very good desire, but my goal was different.
>>>
>>
>>> > For some users this is exactly what is wanted.
>>>
>>> Maybe so, but I didn't face such example (and nobody mentioned those so
>>> far in the discussion).
>>> The opposite (according to the issue) happened. Mxnet example is
>>> sufficient in my opinion.
>>>
>>
>> I agree that the old motivation of APIs that would make it easy to create
>> SciPy is no longer a major motivation for most users and even developers
>> and so these reasons would not be very present (as well as why it wasn't
>> even mentioned in the documentation).
>>
>>
>>> Simple example:
>>> x = np.zeros([])
>>> assert(x.flags.c_contiguous)
>>> assert(np.ascontiguousarray(x).shape == x.shape)
>>>
>>> Behavior contradicts to documentation (shape is changed) and to name
>>> (flags are saying - it is already c_contiguous)
>>>
>>> If you insist, that keeping ndmin=1 is important (I am not yet
>>> convinced, but I am ready to believe your autority),
>>> we can add ndmin=1 to functions' signatures, this way explicitly
>>> notifying users about expected dimension.
>>>
>>
>> I understand the lack of being convinced.  This is ultimately a problem
>> of 0-d arrays not being fully embraced and accepted by the Numeric
>> community originally (which NumPy inherited during the early days).   Is
>> there a way to document functions that will be removed on a major version
>> increase which don't print warnings on use? I would support this.
>>
>
No, there's no such thing at the moment - the closest thing is
https://github.com/numpy/numpy/wiki/Backwards-incompatible-ideas-for-a-major-release.
I doubt we want such a thing anyway - removing functions without
deprecation warnings first doesn't seem quite right.



>
>> I'm a big supporter of making a NumPy 2.0 and have been for several
>> years. Now that Python 3 transition has happened, I think we could
>> seriously discuss this.
>>
>
I think it's more helpful to discuss goals and concrete plans for those,
rather than a "NumPy 2.0" label. The latter never worked in the past, and
not just because of lack of time/funding - it just means different things
to different people. We now have a good start on what our major goals are (
http://www.numpy.org/neps/#roadmap), let's build on that.

  I'm trying to raise funding for maintenance and progress for NumPy and
>> SciPy right now via Quansight Labs http://www.quansight.com/labs and I
>> hope to be able to help find grants to support the wonderful efforts that
>> have been happening for some time.
>>
>
The NumPy grant and having Tyler/Matti/Stefan at BIDS is a great start to
funded development; more and more diverse funding sources would be awesome.

Cheers,
Ralf


>> While I'm thrilled and impressed by the number of amazing devs who have
>> kept NumPy and SciPy going in mostly their spare time, it has created
>> challenges that we have not had continuous maintenance funding to allow
>> continuous paid development so that several people who know about the early
>> decisions could not be retained to spend time on helping the transition.
>>
>> Your bringing the problem of mxnet devs is most appreciated.  I will make
>> a documentation PR.
>>