Re: [Numpy-discussion] Depreciating asfortranarray and ascontiguousarray

[Sylvain Corlay]

I would also argue against deprecating these functions that we are using
increasingly in several projects that I am involved in.

On Sat, Oct 27, 2018, 01:28 Travis Oliphant  wrote:

> What is the justification for deprecation exactly?  These functions have
> been well documented and have had the intended behavior of producing arrays
> with dimension at least 1 for some time.  Why is it unexpected to produce
> arrays of at least 1 dimension?  For some users this is exactly what is
> wanted.  I don't understand the statement that behavior with 0-d arrays is
> unexpected.
>
> If the desire is to shrink the API of NumPy, I could see that.   But, it
> seems odd to me to remove a much-used function with an established behavior
> except as part of a wider API-shrinkage effort.
>
> 0-d arrays in NumPy are a separate conversation.  At this point, I think
> it was a mistake not to embrace 0-d arrays in NumPy from day one.  In some
> sense 0-d arrays *are* scalars at least conceptually and for JIT-producing
> systems that exist now and will be growing in the future, they can be
> equivalent to scalars.
>
> The array scalars should become how you define what is *in* a NumPy array
> making them true Python types, rather than Python 1-style "instances" of a
> single "Dtype" object.  You would then have 0-d arrays and these Python
> "memory" types describing what is *in* the array.
>
> There is a clear way to do this, some of which has been outlined by
> Nathaniel, and the rest I have an outline for how to implement.  I can
> advise someone on how to do this.
>
> -Travis
>
>
>
>
> On Thu, Oct 25, 2018 at 3:17 PM Alex Rogozhnikov <
> alex.rogozhni...@yandex.ru> wrote:
>
>> Dear numpy community,
>>
>> I'm planning to depreciate np.asfortranarray and np.ascontiguousarray
>> functions due to their misbehavior on scalar (0-D tensors) with PR #12244
>> .
>>
>> Current behavior (converting scalars to 1-d array with single element)
>> - is unexpected and contradicts to documentation
>> - probably, can't be changed without breaking external code
>> - I believe, this was a cause for poor support of 0-d arrays in mxnet.
>> - both functions are easily replaced with asarray(..., order='...'),
>> which has expected behavior
>>
>> There is no timeline for removal - we just need to discourage from using
>> this functions in new code.
>>
>> Function naming may be related to how numpy treats 0-d tensors specially,
>>
>> and those probably should not be called arrays.
>> https://www.numpy.org/neps/nep-0027-zero-rank-arrarys.html
>> However, as a user I never thought about 0-d arrays being special and
>> being "not arrays".
>>
>>
>> Please see original discussion at github for more details
>> https://github.com/numpy/numpy/issues/5300
>>
>> Your comments welcome,
>> Alex Rogozhnikov
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Add pybind11 to docs about writing binding code

[Sylvain Corlay]

Hi Hans,


On Thu, Aug 16, 2018 at 10:51 AM, Hans Dembinski 
wrote:

> Hi Sylvain,
>
> On 15. Aug 2018, at 19:38, Sylvain Corlay 
> wrote:
>
> If `pybind11` is included, it could be interesting to also include
> `xtensor` and `xtensor-python`.
>
>  - Xtensor is a C++ dynamic N-d array library that offers numpy-like
> features including broadcasting and universal functions. It is also lazy
> evaluated and continuously benchmarked against numpy, eigen, pythran and
> numba. You can check out the numpy to xtensor cheat sheet:
> https://xtensor.readthedocs.io/en/latest/numpy.html.
>
>  - Xtensor-python makes it possible to operate on numpy arrays inplace
> using the xtensor API. So that e.g. an xtensor reshape will result in a
> reshape on the python side (using the numpy C API under the hood).
>
> Xtensor-python is built upon pybind11, but brings it much closer to
> feature parity with NumPy. There is a vibrant community of users and
> developers, actively working to make xtensor faster and cover more of numpy
> APIs.
>
> I would argue that xtensor-python is one of the easiest ways to make use
> of numpy arrays from a C++ program, given the similar high level API, and
> tools to make ufuncs and bindings with one-liners.
>
> Resources:
>
> - xtensor: https://github.com/QuantStack/xtensor (documentation:
> https://xtensor.readthedocs.io/)
> - xtensor-python: https://github.com/QuantStack/xtensor-python
> (documentation: https://xtensor-python.readthedocs.io/)
> - xtensor-blas:  https://github.com/QuantStack/xtensor-blas (documentation:
> https://xtensor-blas.readthedocs.io)
> - xtensor-io: https://github.com/QuantStack/xtensor-io (documentation:
> https://xtensor-io.readthedocs.io) for reading and writing various file
> formats
>
> Other language bindings:
>
> - xtensor-julia: https://github.com/QuantStack/xtensor-julia
> (documentation: https://xtensor-julia.readthedocs.io/en/latest/)
> - xtensor-r: https://github.com/QuantStack/xtensor-r (documentation:
> https://xtensor-r.readthedocs.io/en/latest/)
>
>
> sounds good, I think it should be mentioned in the pybind11 part. I just
> stumbled over xtensor yesterday. Based on your post I read a bit more about
> it. I like the expression engine and lazy evaluation, the concept is
> similar to Eigen. xtensor itself has nothing to do with binding, but makes
> working with numpy arrays on the C++ side easier - especially when you are
> familiar with the numpy API.
>


Actually, xtensor-python does a lot more in terms of numpy bindings, as it
uses the C APIs of numpy directly for a number of things.

Plus, the integration into the xtensor expression system lets you do things
such as view / broadcasting / newaxis / ufuncs directly from the C++ side
(and all that is in the cheat sheets).


> The docs say:
> "Xtensor operations are continuously benchmarked, and are significantly
> improved at each new version. Current performances on statically
> dimensioned tensors match those of the Eigen library. Dynamically dimension
> tensors for which the shape is heap allocated come at a small additional
> cost."
>
> I couldn't find these benchmark results online, though, could you point me
> to the right page? Google only produced an outdated SO post where numpy
> performed better than xtensor.
>
>
That is because we run the benchmarks on our own hardware. Since xtensor is
explicitly SIMD accelerated for a variety of architectures including e.g.
avx512, it is hard to have a consistent environment to run the benchmarks.
We have a I9 machine that runs the benchmarks with various options, and
manually run them on raspberry pis for the neon acceleration benchmarks
(continuous testing of neon instruction sets are tested with an emulator on
travisci in the xsimd project).

Cheers,

Sylvain



> Best regards,
> Hans
>
> PS: A bit of nitpicking: you use the term "tensor" for an n-dimensional
> block of numbers - a generalisation of "matrix",  but the term "tensor" in
> mathematics and physics is more specific. A tensor has well-defined
> transformation properties when you change the basis of your vector space,
> just like a "vector" (a vector is a one-dimensional tensor), while a
> general block of numbers does not.
>
> https://en.wikipedia.org/wiki/Tensor
>
>
it is clearly a very overloaded term.
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Add pybind11 to docs about writing binding code

[Sylvain Corlay]

Hi Hans, Pauli,

If `pybind11` is included, it could be interesting to also include
`xtensor` and `xtensor-python`.

 - Xtensor is a C++ dynamic N-d array library that offers numpy-like
features including broadcasting and universal functions. It is also lazy
evaluated and continuously benchmarked against numpy, eigen, pythran and
numba. You can check out the numpy to xtensor cheat sheet:
https://xtensor.readthedocs.io/en/latest/numpy.html.

 - Xtensor-python makes it possible to operate on numpy arrays inplace
using the xtensor API. So that e.g. an xtensor reshape will result in a
reshape on the python side (using the numpy C API under the hood).

Xtensor-python is built upon pybind11, but brings it much closer to feature
parity with NumPy. There is a vibrant community of users and developers,
actively working to make xtensor faster and cover more of numpy APIs.

I would argue that xtensor-python is one of the easiest ways to make use of
numpy arrays from a C++ program, given the similar high level API, and
tools to make ufuncs and bindings with one-liners.

Resources:

- xtensor: https://github.com/QuantStack/xtensor (documentation:
https://xtensor.readthedocs.io/)
- xtensor-python: https://github.com/QuantStack/xtensor-python
(documentation: https://xtensor-python.readthedocs.io/)
- xtensor-blas:  https://github.com/QuantStack/xtensor-blas (documentation:
https://xtensor-blas.readthedocs.io)
- xtensor-io: https://github.com/QuantStack/xtensor-io (documentation:
https://xtensor-io.readthedocs.io) for reading and writing various file
formats

Other language bindings:

- xtensor-julia: https://github.com/QuantStack/xtensor-julia
(documentation: https://xtensor-julia.readthedocs.io/en/latest/)
- xtensor-r: https://github.com/QuantStack/xtensor-r (documentation:
https://xtensor-r.readthedocs.io/en/latest/)

Best,

Sylvain


On Wed, Aug 15, 2018 at 12:50 PM, Pauli Virtanen  wrote:

> Hi,
>
> ke, 2018-08-15 kello 11:40 +0200, Hans Dembinski kirjoitti:
> [clip: pybind11]
> > So far all these reasons and more, it should be mentioned and even
> > highlighted here:
> >
> > https://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html <
> > https://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html>
> >
> > I am happy to write a section about it. Disclaimer: I am not at all
> > affiliated with the pybind11 developers, just a thankful user.
>
> Please go ahead --- the relevant source file is here:
>
> https://github.com/numpy/numpy/blob/master/doc/source/
> user/c-info.python-as-glue.rst
>
> It also does no mention CFFI either, mostly because the original text
> predates the project.
>
> --
> Pauli Virtanen
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Adoption of a Code of Conduct

[Sylvain Corlay]

The "political belief" was recently removed from the Jupyter CoC.

One reason for this decision is that *Racism and Sexism are increasingly
considered as mainstream "political beliefs"*, and we wanted to make it
clear that people can still be sanctioned for e.g. sexist or racist
behavior when engaging with the project (at events, on the mailing list or
GitHub...) even if their racism and sexism is corresponds to a "political
belief".

It is still not OK for people to be excluded or discriminated against
because of their political affiliation. The CoC statement reads "*This
includes, but is not limited to...*". Also we* don't wish to prioritize or
elevate any members of a particular political belief to the same level as
any members of the examples remaining in the document*. Ultimately, the CoC
committee uses their own judgement to assess reports and the appropriate
response.

Best,

Sylvain


On Thu, Aug 2, 2018 at 7:04 AM, Robert Kern  wrote:

> On Wed, Aug 1, 2018 at 8:35 PM Ryan May  wrote:
>
>> When experts say that something is a bad idea, and when the people who
>>> a CoC is supposed to protect says it makes them feel unsafe, I feel
>>> like we should listen to that.
>>>
>>> I also thought that the points made in the Jupyter discussion thread
>>> made a lot of sense: of course it's possible for people to start
>>> harassing each other over any excuse, and a CoC can, should, and does
>>> make clear that that's not OK. But if you specifically *call out*
>>> political affiliation as a protected class, at a time when lots of the
>>> people who the CoC is trying to protect are facing governmental
>>> harassment justified as "mere political disagreement", then it really
>>> sends the wrong message.
>>>
>>> Besides, uh... isn't the whole definition of politics that it's topics
>>> where there is active debate? Not really sure why it's even in that
>>> list to start with.
>>>
>>
>> So I hear all the arguments about people feeling unsafe due to some truly
>> despicable, discriminatory behavior, and I want absolutely no parts of
>> protecting that. However, I also recognize that we in the U.S. are in a
>> particularly divisive atmosphere, and people of varied political
>> persuasions want absolutely nothing to do with those who share differing
>> views. So, as a concrete example, if someone were to show up at a NumPy
>> developer summit with a MAGA ("Make America Great Again") hat, or talks
>> about their support for the president in non-numpy channels, WITHOUT
>> expressing anything discriminatory or support for such views, if "political
>> beliefs" is not in the CoC, is this person welcome? I'm not worried about
>> my own views, but I have friends of widely varying views, and I truly
>> wonder if they would be welcome. With differing "political beliefs" listed
>> as something welcomed, I feel ok for them; if this language is removed, I'm
>> much less certain.
>>
>> IMO, "political beliefs" encompasses so much more things than a handful
>> of very specific, hateful views. People can disagree about a wide array of
>> "political beliefs" and it is important that we as a community welcome a
>> wide array of such views. If the CoC needs to protect against the wide
>> array of discriminatory views and behavior that make up U.S. politics right
>> now, how about specifically calling those behaviors out as not-welcome,
>> rather than completely ignoring the fact that 99% of "political beliefs"
>> are perfectly welcome within the community?
>>
>> The CoC is about spelling out the community norms--how about just
>> spelling out that we welcome everyone, but, in the words of Will Wheaton,
>> "Don't be a dick"?
>>
>
> I agree that it's worth clarifying in the text what this clause is
> intended to do. I think it has been misinterpreted as defining a legalistic
> set of protected classes along the lines of anti-discrimination laws and
> can be interpreted by itself outside of the context of the CoC as a whole.
> But it's not that. It's an aspirational statement, and a high one, at that,
> if we interpret it at its broadest. We will fail to meet it, in its
> entirety, and that's *okay* if the spirit of the CoC is being defended. I
> am perfectly happy to keep "political beliefs" explicit in the CoC and
> still boot the neo-feudalist for making the project's/conference's
> environment unwelcoming for a more vulnerable group of people, even if just
> by their presence. I *am* sensitive to how nominally well-intentioned
> "viewpoint diversity" efforts get hijacked by regressives looking to
> (re)assert their traditional power. But that problem is mostly confined to
> conferences who need to seek speakers and has less relevance to numpy,
> which largely doesn't run much except sprints. I think we can resolve that
> elsewhere, if not another document, then at least another clause. A CoC has
> to pull a kind of double duty: be friendly enough to digest for a newcomer
> and also be helpful to project organizers to make tough 

Re: [Numpy-discussion] Extending C with Python

[Sylvain Corlay]

You can also check out pybind11, xtensor, and xtensor-python

The latter enables a high-level numpy-like API on the C++ side.

You can check out the numpy to xtensor cheat sheet:

http://xtensor.readthedocs.io/en/latest/numpy.html

Best,

Sylvain


On Thu, Feb 1, 2018 at 12:11 AM, Stefan Seefeld  wrote:

> On 31.01.2018 17:58, Chris Barker wrote:
>
> I'm guessing you could use Cython to make this easier.
>
>
> ... or Boost.Python (http://boostorg.github.io/python), which has
> built-in support for NumPy (http://boostorg.github.io/
> python/doc/html/numpy/index.html), and supports both directions:
> extending Python with C++, as well as embedding Python into C++
> applications.
>
>
> [image: Stefan]
>
> --
>
>   ...ich hab' noch einen Koffer in Berlin...
>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [SciPy-Dev] PyRSB: Python interface to librsb sparse matrices library

[Sylvain Corlay]

Hi Michele,

Regarding xtensor, the article focused on how we can wrap an existing data
structure in an "xtensor expression", i.e. something that you can operate
upon with the numpy style API in C++ (see the numpy to xtensor cheat sheet
http://xtensor.readthedocs.io/en/latest/numpy.html). Since it is an
expression system, you can do "a + b * sin(t)" with numpy-style
broadcasting and it returns an un-evaluated expression, which is computed
only upon access or assignment to a container. "a", "b", and "c" could all
be compound expressions, backed by filesystem operations, be an in-memory
container, or a wrapper on your sparse matrices...

Now, we are working on automating the "*C++ tensor expression-type ->
python wrapper with access and math operators" *generation of code, which
would be what you are describing below...

On the subject of the LGPL, regarding the distinction with the GPL that
were brought up earlier in the thread, it is indeed not as problematic as
the GPL, but still less permissive as BSD and MIT. In general, I think that
using a BSD or MIT license averts these concerns in the Python ecosystem,
and is preferable *if you don't care about copyleft*...

Best,

Sylvain

On Sun, Jun 25, 2017 at 12:01 PM, Michele Martone <
michelemart...@users.sourceforge.net> wrote:

> On 20170624@16:29, Sylvain Corlay wrote:
> > Hi Michele,
>
> Hi Sylvain,
>
> > This is really interesting. I am a co-author of the xtensor project and
> one
> > thing that could be interesting is to wrap the various sparse matrix data
> > structures in the form of xtensor expressions. A byproduct of doing so is
> > that it would simplify creating bindings for multiple scientific
> computing
> > languages (Python, Julia, R, and more coming). You can see the blog post
> > http://quantstack.net/c++/2017/05/30/polyglot-scientific-computing-with-
> > xtensor.html for reference...
> This article exemplifies manipulation of numerical arrays.
> Now I ask you: Given an interactive language $L of the one you cite above,
> can xtensor provide objects with custom type and operators for manipulation
> in *that* language, like in e.g. the pyrsb case:
>   a=rsb_matrix((4,4))
>   print(a+a) # + operator and 'print' interfacing
> ?
>
> > Also, one quick question: is the LGPL license a deliberate choice or is
> it
> > not important to you? Most projects in the Python scientific stack are
> BSD
> > licensed. So the LGPL choice makes it unlikely that a higher-level
> project
> > adopts it as a dependency. If you are the only copyright holder, you
> would
> > still have the possibility to license it under a more permissive license
> > such as BSD or MIT...
> No important choice.
> No problems relicensing the PyRSB prototype to BSD or MIT.
>
> > Congratulations on the release!
> Thanks for the interest and welcome constructive feedback :-)
>
> > Sylvain
> >
> >
> >
> >
> >
> > On Wed, Jun 21, 2017 at 11:51 AM, Michele Martone <michelemartone@users.
> > sourceforge.net> wrote:
> >
> > > Hi.
> > >
> > >  I'm the author of the high performance multithreaded sparse matrix
> > > library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/
> > >
> > >  I'm *not* a user of SciPy/NumPy/Python, but using Cython I have
> > > written a proof-of-concept interface to librsb, named `PyRSB':
> > >  https://github.com/michelemartone/pyrsb
> > >
> > >  PyRSB is in a prototypal state; e.g. still lacks good error handling.
> > >  Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'.
> > >  Advantages over csr_matrix are in fast multithreaded multiplication
> > > of huge sparse matrices.
> > >  Intended application area is iterative solution of linear systems;
> > > particularly fast if with symmetric matrices and many rhs.
> > >
> > > With this email I am looking for prospective:
> > >  - users/testers
> > >  - developers (any interest to collaborate/adopt/include the project?)
> > >
> > > Looking forward for your feedback,
> > > Michele
> > >
> > > ___
> > > SciPy-Dev mailing list
> > > scipy-...@python.org
> > > https://mail.python.org/mailman/listinfo/scipy-dev
> > >
> > >
>
> > ___
> > SciPy-Dev mailing list
> > scipy-...@python.org
> > https://mail.python.org/mailman/listinfo/scipy-dev
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion