Re: [Numpy-discussion] Fwd: Numpy for data manipulation

[Alex Rogozhnikov]

Thanks for comments, I've fixed the named issues.

Code is python2&3 compatible, I aliased numpy and used better inversion.
Specially thanks for pointing at histogram equalization - I've added 
example for images.
Probably some other 'visual' examples would help - I'll try to invent 
something to other points, but this is not simple.


(I left %matplolib inline due to more appropriate rendering)

Alex.

02.10.15 10:50, Kiko пишет:



2015-10-02 9:48 GMT+02:00 Kiko >:




2015-10-02 9:38 GMT+02:00 Alex Rogozhnikov
mailto:alex.rogozhni...@yandex.ru>>:

I would suggest

%matplotlib notebook

It will still have to a nice png, but you get an
interactive figure when it is live.


Amazing, thanks. I was using mpld3 for this.
(for some strange reason I need to put %matplotlib notebook
before each plot)


You should create a figure before each plot instead of putthon
%matplotlib notebook
plt.figure()



putthon == putting


The recommendation of inverting a permutation by
argsort'ing it, while it works, is suboptimal, as it takes
O(n log(n)) time, and you can do it in linear time:

Actually, there is (later in post) a linear solution using
bincount, but your code is definitely better. Thanks!

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org 
https://mail.scipy.org/mailman/listinfo/numpy-discussion





___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reorganizing numpy internal extensions (was: Re: Should we drop support for "one file" compilation mode?)

[Charles R Harris]

On Tue, Oct 6, 2015 at 1:04 PM, Nathaniel Smith  wrote:

> On Tue, Oct 6, 2015 at 11:52 AM, David Cournapeau 
> wrote:
> >
> >
> > On Tue, Oct 6, 2015 at 7:30 PM, Nathaniel Smith  wrote:
> >>
> >> [splitting this off into a new thread]
> >>
> >> On Tue, Oct 6, 2015 at 3:00 AM, David Cournapeau 
> >> wrote:
> >> [...]
> >> > I also agree the current situation is not sustainable -- as we
> discussed
> >> > privately before, cythonizing numpy.core is made quite more
> complicated
> >> > by
> >> > this. I have myself quite a few issues w/ cythonizing the other parts
> of
> >> > umath. I would also like to support the static link better than we do
> >> > now
> >> > (do we know some static link users we can contact to validate our
> >> > approach
> >> > ?)
> >> >
> >> > Currently, what we have in numpy core is the following:
> >> >
> >> > numpy.core.multiarray -> compilation units in
> numpy/core/src/multiarray/
> >> > +
> >> > statically link npymath
> >> > numpy.core.umath -> compilation units in numpy/core/src/umath +
> >> > statically
> >> > link npymath/npysort + some shenanigans to use things in
> >> > numpy.core.multiarray
> >>
> >> There are also shenanigans in the other direction - supposedly umath
> >> is layered "above" multiarray, but in practice there are circular
> >> dependencies (see e.g. np.set_numeric_ops).
> >
> > Indeed, I am not arguing about merging umath and multiarray.
>
> Oh, okay :-).
>
> >> > I would suggest to have a more layered approach, to enable both
> 'normal'
> >> > build and static build, without polluting the public namespace too
> much.
> >> > This is an approach followed by most large libraries (e.g. MKL), and
> is
> >> > fairly flexible.
> >> >
> >> > Concretely, we could start by putting more common functionalities (aka
> >> > the
> >> > 'core' library) into its own static library. The API would be
> considered
> >> > private to numpy (no stability guaranteed outside numpy), and every
> >> > exported
> >> > symbol from that library would be decorated appropriately to avoid
> >> > potential
> >> > clashes (e.g. '_npy_internal_').
> >>
> >> I don't see why we need this multi-layered complexity, though.
> >
> >
> > For several reasons:
> >
> >  - when you want to cythonize either extension, it is much easier to
> > separate it as cython for CPython API, C for the rest.
>
> I don't think this will help much, because I think we'll want to have
> multiple cython files, and that we'll probably move individual
> functions between being implemented in C and Cython (including utility
> functions). So that means we need to solve the problem of mixing C and
> Cython files inside a single library.
>
> If you look at Stefan's PR:
>   https://github.com/numpy/numpy/pull/6408
> it does solve most of these problems. It would help if Cython added a
> few tweaks to officially support compiling multiple modules into one
> .so, and I'm not sure whether the current code quite handles
> initialization of the submodule correctly, but it's actually
> surprisingly easy to make work.
>
> (Obviously we won't want to go overboard here -- but the point of
> removing the technical constraints is that then it frees us to pick
> whatever arrangement makes the most sense, instead of deciding based
> on what makes the build system and linker easiest.)
>
> >  - if numpy.core.multiarray.so is built as cython-based .o + a 'large' C
> > static library, it should become much simpler to support static link.
>
> I don't see this at all, so I must be missing something? Either way
> you build a bunch of .o files, and then you have to either combine
> them into a shared library or combine them into a static library. Why
> does pre-combining some of them into a static library make this
> easier?
>
> >  - maybe that's just personal, but I find the whole multiarray + umath
> quite
> > beyond manageable in terms of intertwined complexity. You may argue it is
> > not that big, and we all have different preferences in terms of
> > organization, but if I look at the binary size of multiarray + umath, it
> is
> > quite larger than the median size of the .so I have in my /usr/lib.
>
> The binary size isn't a good measure here -- most of that is the
> bazillions of copies of slightly tweaked loops that we auto-generate,
> which take up a lot of space but don't add much intertwined
> complexity. (Though now that I think about it, my LOC estimate was
> probably a bit low because cloc is probably ignoring those
> autogeneration template files.)
>
> We definitely could do a better job with our internal APIs -- I just
> think that'll be easiest if everything is in the same directory so
> there are minimal obstacles to rearranging and refactoring things.
>
> Anyway, it sounds like we agree that the next step is to merge
> multiarray and umath, so possibly we should worry about doing that and
> then see what makes sense from there :-).
>
>
What about removing the single file build? That seems somewhat orthogonal
to this discussion. W

Re: [Numpy-discussion] reorganizing numpy internal extensions (was: Re: Should we drop support for "one file" compilation mode?)

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 11:52 AM, David Cournapeau  wrote:
>
>
> On Tue, Oct 6, 2015 at 7:30 PM, Nathaniel Smith  wrote:
>>
>> [splitting this off into a new thread]
>>
>> On Tue, Oct 6, 2015 at 3:00 AM, David Cournapeau 
>> wrote:
>> [...]
>> > I also agree the current situation is not sustainable -- as we discussed
>> > privately before, cythonizing numpy.core is made quite more complicated
>> > by
>> > this. I have myself quite a few issues w/ cythonizing the other parts of
>> > umath. I would also like to support the static link better than we do
>> > now
>> > (do we know some static link users we can contact to validate our
>> > approach
>> > ?)
>> >
>> > Currently, what we have in numpy core is the following:
>> >
>> > numpy.core.multiarray -> compilation units in numpy/core/src/multiarray/
>> > +
>> > statically link npymath
>> > numpy.core.umath -> compilation units in numpy/core/src/umath +
>> > statically
>> > link npymath/npysort + some shenanigans to use things in
>> > numpy.core.multiarray
>>
>> There are also shenanigans in the other direction - supposedly umath
>> is layered "above" multiarray, but in practice there are circular
>> dependencies (see e.g. np.set_numeric_ops).
>
> Indeed, I am not arguing about merging umath and multiarray.

Oh, okay :-).

>> > I would suggest to have a more layered approach, to enable both 'normal'
>> > build and static build, without polluting the public namespace too much.
>> > This is an approach followed by most large libraries (e.g. MKL), and is
>> > fairly flexible.
>> >
>> > Concretely, we could start by putting more common functionalities (aka
>> > the
>> > 'core' library) into its own static library. The API would be considered
>> > private to numpy (no stability guaranteed outside numpy), and every
>> > exported
>> > symbol from that library would be decorated appropriately to avoid
>> > potential
>> > clashes (e.g. '_npy_internal_').
>>
>> I don't see why we need this multi-layered complexity, though.
>
>
> For several reasons:
>
>  - when you want to cythonize either extension, it is much easier to
> separate it as cython for CPython API, C for the rest.

I don't think this will help much, because I think we'll want to have
multiple cython files, and that we'll probably move individual
functions between being implemented in C and Cython (including utility
functions). So that means we need to solve the problem of mixing C and
Cython files inside a single library.

If you look at Stefan's PR:
  https://github.com/numpy/numpy/pull/6408
it does solve most of these problems. It would help if Cython added a
few tweaks to officially support compiling multiple modules into one
.so, and I'm not sure whether the current code quite handles
initialization of the submodule correctly, but it's actually
surprisingly easy to make work.

(Obviously we won't want to go overboard here -- but the point of
removing the technical constraints is that then it frees us to pick
whatever arrangement makes the most sense, instead of deciding based
on what makes the build system and linker easiest.)

>  - if numpy.core.multiarray.so is built as cython-based .o + a 'large' C
> static library, it should become much simpler to support static link.

I don't see this at all, so I must be missing something? Either way
you build a bunch of .o files, and then you have to either combine
them into a shared library or combine them into a static library. Why
does pre-combining some of them into a static library make this
easier?

>  - maybe that's just personal, but I find the whole multiarray + umath quite
> beyond manageable in terms of intertwined complexity. You may argue it is
> not that big, and we all have different preferences in terms of
> organization, but if I look at the binary size of multiarray + umath, it is
> quite larger than the median size of the .so I have in my /usr/lib.

The binary size isn't a good measure here -- most of that is the
bazillions of copies of slightly tweaked loops that we auto-generate,
which take up a lot of space but don't add much intertwined
complexity. (Though now that I think about it, my LOC estimate was
probably a bit low because cloc is probably ignoring those
autogeneration template files.)

We definitely could do a better job with our internal APIs -- I just
think that'll be easiest if everything is in the same directory so
there are minimal obstacles to rearranging and refactoring things.

Anyway, it sounds like we agree that the next step is to merge
multiarray and umath, so possibly we should worry about doing that and
then see what makes sense from there :-).

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reorganizing numpy internal extensions (was: Re: Should we drop support for "one file" compilation mode?)

[David Cournapeau]

On Tue, Oct 6, 2015 at 7:30 PM, Nathaniel Smith  wrote:

> [splitting this off into a new thread]
>
> On Tue, Oct 6, 2015 at 3:00 AM, David Cournapeau 
> wrote:
> [...]
> > I also agree the current situation is not sustainable -- as we discussed
> > privately before, cythonizing numpy.core is made quite more complicated
> by
> > this. I have myself quite a few issues w/ cythonizing the other parts of
> > umath. I would also like to support the static link better than we do now
> > (do we know some static link users we can contact to validate our
> approach
> > ?)
> >
> > Currently, what we have in numpy core is the following:
> >
> > numpy.core.multiarray -> compilation units in numpy/core/src/multiarray/
> +
> > statically link npymath
> > numpy.core.umath -> compilation units in numpy/core/src/umath +
> statically
> > link npymath/npysort + some shenanigans to use things in
> > numpy.core.multiarray
>
> There are also shenanigans in the other direction - supposedly umath
> is layered "above" multiarray, but in practice there are circular
> dependencies (see e.g. np.set_numeric_ops).
>

Indeed, I am not arguing about merging umath and multiarray.


> > I would suggest to have a more layered approach, to enable both 'normal'
> > build and static build, without polluting the public namespace too much.
> > This is an approach followed by most large libraries (e.g. MKL), and is
> > fairly flexible.
> >
> > Concretely, we could start by putting more common functionalities (aka
> the
> > 'core' library) into its own static library. The API would be considered
> > private to numpy (no stability guaranteed outside numpy), and every
> exported
> > symbol from that library would be decorated appropriately to avoid
> potential
> > clashes (e.g. '_npy_internal_').
>
> I don't see why we need this multi-layered complexity, though.
>

For several reasons:

 - when you want to cythonize either extension, it is much easier to
separate it as cython for CPython API, C for the rest.
 - if numpy.core.multiarray.so is built as cython-based .o + a 'large' C
static library, it should become much simpler to support static link.
 - maybe that's just personal, but I find the whole multiarray + umath
quite beyond manageable in terms of intertwined complexity. You may argue
it is not that big, and we all have different preferences in terms of
organization, but if I look at the binary size of multiarray + umath, it is
quite larger than the median size of the .so I have in my /usr/lib.

I am also hoping that splitting up numpy.core in separate elements that
communicate through internal APIs would make participating into numpy
easier.

We could also swap the argument: assuming it does not make the build more
complex, and that it does help static linking, why not doing it ?

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] reorganizing numpy internal extensions (was: Re: Should we drop support for "one file" compilation mode?)

[Nathaniel Smith]

[splitting this off into a new thread]

On Tue, Oct 6, 2015 at 3:00 AM, David Cournapeau  wrote:
[...]
> I also agree the current situation is not sustainable -- as we discussed
> privately before, cythonizing numpy.core is made quite more complicated by
> this. I have myself quite a few issues w/ cythonizing the other parts of
> umath. I would also like to support the static link better than we do now
> (do we know some static link users we can contact to validate our approach
> ?)
>
> Currently, what we have in numpy core is the following:
>
> numpy.core.multiarray -> compilation units in numpy/core/src/multiarray/ +
> statically link npymath
> numpy.core.umath -> compilation units in numpy/core/src/umath + statically
> link npymath/npysort + some shenanigans to use things in
> numpy.core.multiarray

There are also shenanigans in the other direction - supposedly umath
is layered "above" multiarray, but in practice there are circular
dependencies (see e.g. np.set_numeric_ops).

> I would suggest to have a more layered approach, to enable both 'normal'
> build and static build, without polluting the public namespace too much.
> This is an approach followed by most large libraries (e.g. MKL), and is
> fairly flexible.
>
> Concretely, we could start by putting more common functionalities (aka the
> 'core' library) into its own static library. The API would be considered
> private to numpy (no stability guaranteed outside numpy), and every exported
> symbol from that library would be decorated appropriately to avoid potential
> clashes (e.g. '_npy_internal_').

I don't see why we need this multi-layered complexity, though.

npymath is a well-defined utility library that other people use, so
sure, it makes sense to keep that somewhat separate as a static
library (as discussed in the other thread).

Beyond that -- NumPy is really not a large library. multiarray is <50k
lines of code, and umath is only ~6k (!). And there's no particular
reason to keep them split up from the user point of view -- all their
functionality gets combined into the flat numpy namespace anyway. So
we *could* rewrite them as three libraries, with a "common core" that
then gets exported via two different wrapper libraries -- but it's
much simpler just to do

mv umath/* multiarray/
rmdir umath

and then make multiarray work the way we want. (After fixing up the
build system of course :-).)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 6:18 PM, David Cournapeau  wrote:

>
>
> On Tue, Oct 6, 2015 at 6:14 PM, Nathaniel Smith  wrote:
>
>> On Tue, Oct 6, 2015 at 10:10 AM, David Cournapeau 
>> wrote:
>> >
>> >
>> > On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith  wrote:
>> >>
>> >> On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou 
>> >> wrote:
>> >> > On Tue, 6 Oct 2015 09:40:43 -0700
>> >> > Nathaniel Smith  wrote:
>> >> >>
>> >> >> If you need some npy_* function it'd be much better to let us know
>> >> >> what it is and let us export it in an intentional way, instead of
>> just
>> >> >> relying on whatever stuff we accidentally exposed?
>> >> >
>> >> > Ok, we seem to be using only the complex math functions (npy_cpow and
>> >> > friends, I could make a complete list if required).
>> >>
>> >> And how are you getting at them? Are you just relying the way that on
>> >> ELF systems, if two libraries are loaded into the same address space
>> >> then they automatically get access to each other's symbols, even if
>> >> they aren't linked to each other? What do you do on Windows?
>> >
>> >
>> > It is possible (and documented) to use any of the npy_ symbols from
>> npymath
>> > from outside numpy:
>> >
>> http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension
>> >
>> > The design is not perfect (I was young and foolish :) ), but it has
>> worked
>> > fairly well and has been used in at least scipy since the 1.4/1.5 days
>> IIRC
>> > (including windows).
>>
>> Okay, so just to confirm, it looks like this does indeed implement the
>> static linking thing I just suggested (so perhaps I am also young and
>> foolish ;-)) -- from looking at the output of get_info("npymath"), it
>> seems to add -I.../numpy/core/include to the compiler flags, add
>> -lnpymath -L.../numpy/core/lib to the linker flags, and then
>> .../numpy/core/lib contains only libnpymath.a, so it's static linking.
>>
>
> Yes, I was not trying to argue otherwise. If you thought I was, blame it
> on my poor English (which sadly does not get better as I get less young...).
>
> My proposal is to extend this technique for *internal* API, but with the
> following differences:
>  * the declarations are not put in any public header
>  * we don't offer any way to link to this library, and name it something
> scary enough that people would have to be foolish (young or not) to use it.
>

I am stupid: we of course do not even ship that internal library, it would
just be linked into multiarray/umath and never installed or part of binary
packages.

David

>
> David
>
>
>> -n
>>
>> --
>> Nathaniel J. Smith -- http://vorpus.org
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Antoine Pitrou]

On Tue, 6 Oct 2015 10:07:13 -0700
Nathaniel Smith  wrote:
> 
> And how are you getting at them? Are you just relying the way that on
> ELF systems, if two libraries are loaded into the same address space
> then they automatically get access to each other's symbols, even if
> they aren't linked to each other? What do you do on Windows?

Well it seems to work on Windows too, thanks to
numpy.distutils.misc_util.get_info('npymath').

Under Windows, I seem to have a
"\site-packages\numpy\core\lib\npymath.lib" static library,
and there's also a "npy-pkg-config" subdirectory there with some INI
files in it.  Hopefully you know better than me what this all is :-)

> > And, of course, we would also benefit from the CBLAS functions (or any
> > kind of C wrappers around them) :-)
> > https://github.com/numpy/numpy/issues/6324
> 
> This is difficult to do from NumPy itself -- we don't necessarily have
> access to a full BLAS or LAPACK API -- in some configurations we fall
> back on our minimal internal implementations that just have what we
> need.

I'm thinking about the functions exposed in
"numpy/core/src/private/npy_cblas.h".
My knowledge of the Numpy build system doesn't allow me to tell if it's
always available or not :-)

> There was an interesting idea that came up in some discussions here a
> few weeks ago -- we already know that we want to package up BLAS
> inside a Python package that (numpy / scipy / scikit-learn / ...) can
> depend on and assume is there to link against.
> 
> Maybe this new package would also be a good place for exposing these wrappers?

Yeah, why not - as long as there's something well-known and
well-supported to depend on.  But given Numpy is a hard dependency for
all the other packages you mentioned, it may make sense (and simplify
dependency management) to bundle it with Numpy.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 6:14 PM, Nathaniel Smith  wrote:

> On Tue, Oct 6, 2015 at 10:10 AM, David Cournapeau 
> wrote:
> >
> >
> > On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith  wrote:
> >>
> >> On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou 
> >> wrote:
> >> > On Tue, 6 Oct 2015 09:40:43 -0700
> >> > Nathaniel Smith  wrote:
> >> >>
> >> >> If you need some npy_* function it'd be much better to let us know
> >> >> what it is and let us export it in an intentional way, instead of
> just
> >> >> relying on whatever stuff we accidentally exposed?
> >> >
> >> > Ok, we seem to be using only the complex math functions (npy_cpow and
> >> > friends, I could make a complete list if required).
> >>
> >> And how are you getting at them? Are you just relying the way that on
> >> ELF systems, if two libraries are loaded into the same address space
> >> then they automatically get access to each other's symbols, even if
> >> they aren't linked to each other? What do you do on Windows?
> >
> >
> > It is possible (and documented) to use any of the npy_ symbols from
> npymath
> > from outside numpy:
> >
> http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension
> >
> > The design is not perfect (I was young and foolish :) ), but it has
> worked
> > fairly well and has been used in at least scipy since the 1.4/1.5 days
> IIRC
> > (including windows).
>
> Okay, so just to confirm, it looks like this does indeed implement the
> static linking thing I just suggested (so perhaps I am also young and
> foolish ;-)) -- from looking at the output of get_info("npymath"), it
> seems to add -I.../numpy/core/include to the compiler flags, add
> -lnpymath -L.../numpy/core/lib to the linker flags, and then
> .../numpy/core/lib contains only libnpymath.a, so it's static linking.
>

Yes, I was not trying to argue otherwise. If you thought I was, blame it on
my poor English (which sadly does not get better as I get less young...).

My proposal is to extend this technique for *internal* API, but with the
following differences:
 * the declarations are not put in any public header
 * we don't offer any way to link to this library, and name it something
scary enough that people would have to be foolish (young or not) to use it.

David


> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 10:10 AM, David Cournapeau  wrote:
>
>
> On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith  wrote:
>>
>> On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou 
>> wrote:
>> > On Tue, 6 Oct 2015 09:40:43 -0700
>> > Nathaniel Smith  wrote:
>> >>
>> >> If you need some npy_* function it'd be much better to let us know
>> >> what it is and let us export it in an intentional way, instead of just
>> >> relying on whatever stuff we accidentally exposed?
>> >
>> > Ok, we seem to be using only the complex math functions (npy_cpow and
>> > friends, I could make a complete list if required).
>>
>> And how are you getting at them? Are you just relying the way that on
>> ELF systems, if two libraries are loaded into the same address space
>> then they automatically get access to each other's symbols, even if
>> they aren't linked to each other? What do you do on Windows?
>
>
> It is possible (and documented) to use any of the npy_ symbols from npymath
> from outside numpy:
> http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension
>
> The design is not perfect (I was young and foolish :) ), but it has worked
> fairly well and has been used in at least scipy since the 1.4/1.5 days IIRC
> (including windows).

Okay, so just to confirm, it looks like this does indeed implement the
static linking thing I just suggested (so perhaps I am also young and
foolish ;-)) -- from looking at the output of get_info("npymath"), it
seems to add -I.../numpy/core/include to the compiler flags, add
-lnpymath -L.../numpy/core/lib to the linker flags, and then
.../numpy/core/lib contains only libnpymath.a, so it's static linking.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith  wrote:

> On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou 
> wrote:
> > On Tue, 6 Oct 2015 09:40:43 -0700
> > Nathaniel Smith  wrote:
> >>
> >> If you need some npy_* function it'd be much better to let us know
> >> what it is and let us export it in an intentional way, instead of just
> >> relying on whatever stuff we accidentally exposed?
> >
> > Ok, we seem to be using only the complex math functions (npy_cpow and
> > friends, I could make a complete list if required).
>
> And how are you getting at them? Are you just relying the way that on
> ELF systems, if two libraries are loaded into the same address space
> then they automatically get access to each other's symbols, even if
> they aren't linked to each other? What do you do on Windows?
>

It is possible (and documented) to use any of the npy_ symbols from npymath
from outside numpy:
http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension

The design is not perfect (I was young and foolish :) ), but it has worked
fairly well and has been used in at least scipy since the 1.4/1.5 days IIRC
(including windows).

David


>
> > And, of course, we would also benefit from the CBLAS functions (or any
> > kind of C wrappers around them) :-)
> > https://github.com/numpy/numpy/issues/6324
>
> This is difficult to do from NumPy itself -- we don't necessarily have
> access to a full BLAS or LAPACK API -- in some configurations we fall
> back on our minimal internal implementations that just have what we
> need.
>
> There was an interesting idea that came up in some discussions here a
> few weeks ago -- we already know that we want to package up BLAS
> inside a Python package that (numpy / scipy / scikit-learn / ...) can
> depend on and assume is there to link against.
>
> Maybe this new package would also be a good place for exposing these
> wrappers?
>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 9:51 AM, David Cournapeau  wrote:
>
> On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith  wrote:
>>
>> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau 
>> wrote:
>> > The npy_ functions in npymath were designed to be exported. Those would
>> > stay
>> > that way.
>>
>> If we want to export these then I vote that we either:
>> - use the usual API export mechanism, or else
>> - provide a static library for people to link to, instead of trying to
>> do runtime binding. (I.e. drop it in some known place, and then
>> provide some functions for extension modules to find it at build time
>> -- similar to how np.get_include() works.)
>
> Unless something changed, that's more or less how it works already (npymath
> is used in scipy, for example, which was one of the rationale for writing it
> in the first place !).

Okay... in fact multiarray.so right now *does* export tons and tons of
random junk into the global symbol namespace (on systems like Linux
that do have a global symbol namespace), so it isn't obvious whether
people are asking for that to continue :-). I'm just specifically
saying that we should try to get this back down to the 1 exported
symbol.

(Try:
   objdump -T $(python -c 'import numpy; print(numpy.core.multiarray.__file__)')
This *should* print 1 line... I currently get ~700. numpy.core.umath
is similar.)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 5:58 PM, Nathaniel Smith  wrote:

> On Tue, Oct 6, 2015 at 9:51 AM, David Cournapeau 
> wrote:
> >
> > On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith  wrote:
> >>
> >> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau 
> >> wrote:
> >> > The npy_ functions in npymath were designed to be exported. Those
> would
> >> > stay
> >> > that way.
> >>
> >> If we want to export these then I vote that we either:
> >> - use the usual API export mechanism, or else
> >> - provide a static library for people to link to, instead of trying to
> >> do runtime binding. (I.e. drop it in some known place, and then
> >> provide some functions for extension modules to find it at build time
> >> -- similar to how np.get_include() works.)
> >
> > Unless something changed, that's more or less how it works already
> (npymath
> > is used in scipy, for example, which was one of the rationale for
> writing it
> > in the first place !).
>
> Okay... in fact multiarray.so right now *does* export tons and tons of
> random junk into the global symbol namespace (on systems like Linux
> that do have a global symbol namespace), so it isn't obvious whether
> people are asking for that to continue :-). I'm just specifically
> saying that we should try to get this back down to the 1 exported
> symbol.
>
> (Try:
>objdump -T $(python -c 'import numpy;
> print(numpy.core.multiarray.__file__)')
> This *should* print 1 line... I currently get ~700. numpy.core.umath
> is similar.)
>
>
I think this overestimates the amount by quite a bit, since you see GLIBC
symbols, etc... I am using nm -Dg --defined-only $(python -c 'import numpy;
print(numpy.core.multiarray.__file__)')instead.

I see around 290 symboles: the npy_ from npymath don't bother me, but the
ones from npysort do. We should at least prefix those with npy_ (I don't
think npysort API has ever been publisher in our header like npymath was ?)

David

-n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou  wrote:
> On Tue, 6 Oct 2015 09:40:43 -0700
> Nathaniel Smith  wrote:
>>
>> If you need some npy_* function it'd be much better to let us know
>> what it is and let us export it in an intentional way, instead of just
>> relying on whatever stuff we accidentally exposed?
>
> Ok, we seem to be using only the complex math functions (npy_cpow and
> friends, I could make a complete list if required).

And how are you getting at them? Are you just relying the way that on
ELF systems, if two libraries are loaded into the same address space
then they automatically get access to each other's symbols, even if
they aren't linked to each other? What do you do on Windows?

> And, of course, we would also benefit from the CBLAS functions (or any
> kind of C wrappers around them) :-)
> https://github.com/numpy/numpy/issues/6324

This is difficult to do from NumPy itself -- we don't necessarily have
access to a full BLAS or LAPACK API -- in some configurations we fall
back on our minimal internal implementations that just have what we
need.

There was an interesting idea that came up in some discussions here a
few weeks ago -- we already know that we want to package up BLAS
inside a Python package that (numpy / scipy / scikit-learn / ...) can
depend on and assume is there to link against.

Maybe this new package would also be a good place for exposing these wrappers?

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Antoine Pitrou]

On Tue, 6 Oct 2015 09:40:43 -0700
Nathaniel Smith  wrote:
> 
> If you need some npy_* function it'd be much better to let us know
> what it is and let us export it in an intentional way, instead of just
> relying on whatever stuff we accidentally exposed?

Ok, we seem to be using only the complex math functions (npy_cpow and
friends, I could make a complete list if required).

And, of course, we would also benefit from the CBLAS functions (or any
kind of C wrappers around them) :-)
https://github.com/numpy/numpy/issues/6324

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 9:51 AM, David Cournapeau  wrote:
>
> On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith  wrote:
>>
>> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau 
>> wrote:
>> > The npy_ functions in npymath were designed to be exported. Those would
>> > stay
>> > that way.
>>
>> If we want to export these then I vote that we either:
>> - use the usual API export mechanism, or else
>> - provide a static library for people to link to, instead of trying to
>> do runtime binding. (I.e. drop it in some known place, and then
>> provide some functions for extension modules to find it at build time
>> -- similar to how np.get_include() works.)
>
> Unless something changed, that's more or less how it works already (npymath
> is used in scipy, for example, which was one of the rationale for writing it
> in the first place !).

Okay... in fact multiarray.so right now *does* export tons and tons of
random junk into the global symbol namespace (on systems like Linux
that do have a global symbol namespace), so it isn't obvious whether
people are asking for that to continue :-). I'm just specifically
saying that we should try to get this back down to the 1 exported
symbol.

(Try:
   objdump -T $(python -c 'import numpy; print(numpy.core.multiarray.__file__)')
This *should* print 1 line... I currently get ~700. numpy.core.umath
is similar.)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 5:51 PM, David Cournapeau  wrote:

>
>
> On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith  wrote:
>
>> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau 
>> wrote:
>> > The npy_ functions in npymath were designed to be exported. Those would
>> stay
>> > that way.
>>
>> If we want to export these then I vote that we either:
>> - use the usual API export mechanism, or else
>> - provide a static library for people to link to, instead of trying to
>> do runtime binding. (I.e. drop it in some known place, and then
>> provide some functions for extension modules to find it at build time
>> -- similar to how np.get_include() works.)
>>
>
> Unless something changed, that's more or less how it works already
> (npymath is used in scipy, for example, which was one of the rationale for
> writing it in the first place !).
>
> You access the compilation/linking issues through the numpy distutils
> get_info function.
>

And my suggestion is to use a similar mechanism for multiarray and umath,
so that in the end the exported Python C API is just a thin layer on top of
the underlying static library. That would make cython and I suspect static
linking quite a bit easier. The API between the low layer and python C API
of multiarray/umath would be considered private and outside any API/ABI
stability.

IOW, it would be an internal change, and should not cause visible changes
to the users, except that some _npy_private_ symbols would be exported (but
you would be crazy to use them, and the prototype declarations would not be
available when you install numpy anyway). Think of those as the internal
driver API/ABI of Linux or similar.

David

>
> David
>
>
>> -n
>>
>> --
>> Nathaniel J. Smith -- http://vorpus.org
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith  wrote:

> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau 
> wrote:
> > The npy_ functions in npymath were designed to be exported. Those would
> stay
> > that way.
>
> If we want to export these then I vote that we either:
> - use the usual API export mechanism, or else
> - provide a static library for people to link to, instead of trying to
> do runtime binding. (I.e. drop it in some known place, and then
> provide some functions for extension modules to find it at build time
> -- similar to how np.get_include() works.)
>

Unless something changed, that's more or less how it works already (npymath
is used in scipy, for example, which was one of the rationale for writing
it in the first place !).

You access the compilation/linking issues through the numpy distutils
get_info function.

David


> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 4:07 AM, Antoine Pitrou  wrote:
> On Tue, 6 Oct 2015 11:00:30 +0100
> David Cournapeau  wrote:
>>
>> Assuming one of the rumour is related to some comments I made some time
>> (years ?) earlier, the context was the ability to hide exported symbols. As
>> you know, the issue is not to build extensions w/ multiple compilation
>> units, but sharing functionalities between them without sharing them
>> outside the extension.
>
> Can't you use the visibility attribute with gcc for this?
> Other Unix compilers probably provide something similar. The issue
> doesn't exist on Windows by construction.
>
> https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Function-Attributes.html#Function-Attributes

This is what we do normally when building a shared extension, but in
the exceptional case where people want to statically link numpy into
cpython, it doesn't work -- the visibility attribute and related
machinery only works on shared libraries, not static libraries.
(Recall that traditionally, doing 'a.o + b.o -> static.a; static.a +
c.o -> final' is just a shorthand for doing 'a.o + b.o + c.o ->
final'.)

But this is still a solved problem, you just have to use the static
linking version instead of the shared library version :-). E.g. with
GNU tools the magic incantation is
  objcopy --keep-symbol-name PyInit_multiarray multiarray.a

It's possible that there's some system somewhere that both needs
static linking *and* doesn't have access to objcopy-or-equivalent, but
then we're back with the thing where it's not a great plan to keep
spending energy on supporting purely theoretical platforms.

> By the way, external packages may reuse the npy_* functions, so I would
> like them not the be hidden :-)

This is a separate issue from the one file/multi-file compilation mode, but:

I'd really prefer that we stick to just one way of exporting stuff to
external packages, and that that be the (admittedly somewhat
cumbersome) public API / import_array() mechanism.

Trying to manage numpy's API/ABI exposure is a huge challenge in
general, so having two mechanisms is not really sustainable. And
trying to use the linker directly creates huge cross-platform
headaches -- no-one can agree on what's exported by default, or how
you find shared libraries, and the numpy extensions will certainly not
be on the default library search path, so you need some
platform-specific hack to find them... OTOH the "public API" mechanism
takes some ugly stuff on numpy's side to set things up, but then the
result is a uniform mechanism that uses Python's familiar package
lookup rules to find the relevant symbols.

If you need some npy_* function it'd be much better to let us know
what it is and let us export it in an intentional way, instead of just
relying on whatever stuff we accidentally exposed?

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau  wrote:
> The npy_ functions in npymath were designed to be exported. Those would stay
> that way.

If we want to export these then I vote that we either:
- use the usual API export mechanism, or else
- provide a static library for people to link to, instead of trying to
do runtime binding. (I.e. drop it in some known place, and then
provide some functions for extension modules to find it at build time
-- similar to how np.get_include() works.)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.10.0 release

[Neal Becker]

Sebastian Berg wrote:

> On Di, 2015-10-06 at 07:53 -0400, Neal Becker wrote:
>> 1 test failure:
>> 
>> FAIL: test_blasdot.test_blasdot_used
>> --
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in
>>   runTest
>> self.test(*self.arg)
>>   File "/home/nbecker/.local/lib/python2.7/site-
>> packages/numpy/testing/decorators.py", line 146, in skipper_func
>> return f(*args, **kwargs)
>>   File "/home/nbecker/.local/lib/python2.7/site-
>> packages/numpy/core/tests/test_blasdot.py", line 31, in test_blasdot_used
>> assert_(dot is _dotblas.dot)
>>   File "/home/nbecker/.local/lib/python2.7/site-
>> packages/numpy/testing/utils.py", line 53, in assert_
>> raise AssertionError(smsg)
>> AssertionError
>> 
> 
> My first guess would be, that it sounds like you got some old test files
> flying around. Can you try cleaning up everything and reinstall? It can
> happen that old installed test files survive the new version.
> 
Yes, I rm'd the old ~/.local/lib/python2.7/site-packages/numpy, reinstalled, 
and now no test failure:

Ran 5955 tests in 30.778s

OK (KNOWNFAIL=3, SKIP=1)

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.10.0 release

[Daπid]

I don't get any failures on Fedora 22. I have installed it with pip,
setting my CFLAGS to "-march=core-avx-i -O2 -pipe -mtune=native" and
linking against openblas.

With the new Numpy, Scipy full suite shows two errors, I am sorry I didn't
think of running that in the RC phase:

==
FAIL: test_weighting (test_stats.TestHistogram)
--
Traceback (most recent call last):
  File
"/home/david/.local/virtualenv/py27/lib/python2.7/site-packages/scipy/stats/tests/test_stats.py",
line 892, in test_weighting
decimal=2)
  File
"/home/david/.local/virtualenv/py27/lib/python2.7/site-packages/numpy/testing/utils.py",
line 886, in assert_array_almost_equal
precision=decimal)
  File
"/home/david/.local/virtualenv/py27/lib/python2.7/site-packages/numpy/testing/utils.py",
line 708, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 2 decimals

(mismatch 40.0%)
 x: array([   4. ,0. ,4.5,   -0.9,0. ,0.3,  110.2,0. ,
  0. ,   42. ])
 y: array([   4. ,0. ,4.5,   -0.9,0.3,0. ,7. ,  103.2,
  0. ,   42. ])

==
FAIL: test_nanmedian_all_axis (test_stats.TestNanFunc)
--
Traceback (most recent call last):
  File
"/home/david/.local/virtualenv/py27/lib/python2.7/site-packages/scipy/stats/tests/test_stats.py",
line 226, in test_nanmedian_all_axis
assert_equal(len(w), 4)
  File
"/home/david/.local/virtualenv/py27/lib/python2.7/site-packages/numpy/testing/utils.py",
line 354, in assert_equal
raise AssertionError(msg)
AssertionError:
Items are not equal:
 ACTUAL: 1
 DESIRED: 4

I am almost sure these errors weren't there before.

On 6 October 2015 at 13:53, Neal Becker  wrote:

> 1 test failure:
>
> FAIL: test_blasdot.test_blasdot_used
> --
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in
> runTest
> self.test(*self.arg)
>   File "/home/nbecker/.local/lib/python2.7/site-
> packages/numpy/testing/decorators.py", line 146, in skipper_func
> return f(*args, **kwargs)
>   File "/home/nbecker/.local/lib/python2.7/site-
> packages/numpy/core/tests/test_blasdot.py", line 31, in test_blasdot_used
> assert_(dot is _dotblas.dot)
>   File "/home/nbecker/.local/lib/python2.7/site-
> packages/numpy/testing/utils.py", line 53, in assert_
> raise AssertionError(smsg)
> AssertionError
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.10.0 release

[Sebastian Berg]

On Di, 2015-10-06 at 07:53 -0400, Neal Becker wrote:
> 1 test failure:
> 
> FAIL: test_blasdot.test_blasdot_used
> --
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
> self.test(*self.arg)
>   File "/home/nbecker/.local/lib/python2.7/site-
> packages/numpy/testing/decorators.py", line 146, in skipper_func
> return f(*args, **kwargs)
>   File "/home/nbecker/.local/lib/python2.7/site-
> packages/numpy/core/tests/test_blasdot.py", line 31, in test_blasdot_used
> assert_(dot is _dotblas.dot)
>   File "/home/nbecker/.local/lib/python2.7/site-
> packages/numpy/testing/utils.py", line 53, in assert_
> raise AssertionError(smsg)
> AssertionError
> 

My first guess would be, that it sounds like you got some old test files
flying around. Can you try cleaning up everything and reinstall? It can
happen that old installed test files survive the new version.

And most of all, thanks a lot Chuck!

- Sebastian


> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.10.0 release

[Neal Becker]

1 test failure:

FAIL: test_blasdot.test_blasdot_used
--
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
  File "/home/nbecker/.local/lib/python2.7/site-
packages/numpy/testing/decorators.py", line 146, in skipper_func
return f(*args, **kwargs)
  File "/home/nbecker/.local/lib/python2.7/site-
packages/numpy/core/tests/test_blasdot.py", line 31, in test_blasdot_used
assert_(dot is _dotblas.dot)
  File "/home/nbecker/.local/lib/python2.7/site-
packages/numpy/testing/utils.py", line 53, in assert_
raise AssertionError(smsg)
AssertionError


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 12:07 PM, Antoine Pitrou  wrote:

> On Tue, 6 Oct 2015 11:00:30 +0100
> David Cournapeau  wrote:
> >
> > Assuming one of the rumour is related to some comments I made some time
> > (years ?) earlier, the context was the ability to hide exported symbols.
> As
> > you know, the issue is not to build extensions w/ multiple compilation
> > units, but sharing functionalities between them without sharing them
> > outside the extension.
>
> Can't you use the visibility attribute with gcc for this?
>

We do that already for gcc, I think the question was whether every platform
supported this or not (and whether we should care).


> Other Unix compilers probably provide something similar. The issue
> doesn't exist on Windows by construction.
>
>
> https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Function-Attributes.html#Function-Attributes
>
> By the way, external packages may reuse the npy_* functions, so I would
> like them not the be hidden :-)
>

The npy_ functions in npymath were designed to be exported. Those would
stay that way.

David

>
> Regards
>
> Antoine.
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.10.0 release

[Neal Becker]

Are extra_compile_args actually used in all compile steps?

I set:
extra_compile_args = -march=native -O3

On some compile steps, it echos:
compile options: '-DHAVE_CBLAS -Inumpy/core/include -Ibuild/src.linux-
x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private -
Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -
Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -
I/usr/include/python2.7 -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private 
-Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-
x86_64-2.7/numpy/core/src/private -c'
extra options: '-march=native -O3'

But on at least one it doesn't:
building 'numpy.random.mtrand' extension
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -
Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-
protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -
mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -
Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-
protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -
mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -march=native -O3 -fPIC

creating build/temp.linux-x86_64-2.7/numpy/random
creating build/temp.linux-x86_64-2.7/numpy/random/mtrand
compile options: '-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -
D_LARGEFILE64_SOURCE=1 -Inumpy/core/include -Ibuild/src.linux-
x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private -
Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -
Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -
I/usr/include/python2.7 -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private 
-Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-
x86_64-2.7/numpy/core/src/private -c'

Building with:
CFLAGS='-march=native -O3' python setup.py build

Does seem to use my CFLAGS, as it always did on previous numpy versions.

It seems a bit difficult to verify what the exact compile/link steps were.  
Is there a verbose flag?

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.10.0 release

[Neal Becker]

lots of warning with openblas

 python setup.py build
Running from numpy source directory.
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown 
distribution option: 'test_suite'
  warnings.warn(msg)
blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in ['/usr/local/lib64', 
'/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

openblas_info:
/home/nbecker/numpy-1.10.0/numpy/distutils/system_info.py:635: UserWarning: 
Specified path  is invalid.
  warnings.warn('Specified path %s is invalid.' % d)
  libraries openblas not found in []
Runtime library openblas was not found. Ignoring
  FOUND:
libraries = ['openblas', 'openblas']
library_dirs = ['/usr/lib64']
language = c
define_macros = [('HAVE_CBLAS', None)]

  FOUND:
libraries = ['openblas', 'openblas']
extra_compile_args = ['-march=native -O3']
library_dirs = ['/usr/lib64']
language = c
define_macros = [('HAVE_CBLAS', None)]
...
cc /tmp/tmpoQq7Hu/tmp/tmpoQq7Hu/source.o -L/usr/lib64 -lopenblas -o 
/tmp/tmpoQq7Hu/a.out
  libraries openblas not found in []
Runtime library openblas was not found. Ignoring
  FOUND:
libraries = ['openblas', 'openblas']
library_dirs = ['/usr/lib64']
language = c
define_macros = [('HAVE_CBLAS', None)]

  FOUND:
libraries = ['openblas', 'openblas']
extra_compile_args = ['-march=native -O3']
library_dirs = ['/usr/lib64']
language = c
define_macros = [('HAVE_CBLAS', None)]
...

But I think openblas was used, despite the warnings, because later on I see 
-lopenblas in the link step.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Antoine Pitrou]

On Tue, 6 Oct 2015 11:00:30 +0100
David Cournapeau  wrote:
> 
> Assuming one of the rumour is related to some comments I made some time
> (years ?) earlier, the context was the ability to hide exported symbols. As
> you know, the issue is not to build extensions w/ multiple compilation
> units, but sharing functionalities between them without sharing them
> outside the extension.

Can't you use the visibility attribute with gcc for this?
Other Unix compilers probably provide something similar. The issue
doesn't exist on Windows by construction.

https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Function-Attributes.html#Function-Attributes

By the way, external packages may reuse the npy_* functions, so I would
like them not the be hidden :-)

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Juha Jeronen]

Hi all,

Thanks Jon and Ryan for the suggestions. Both asanyarray() or 
atleast_1d() sound good.


There's the technical detail that Cython needs to know the object type 
(e.g. in the parameters to quartic_z_array()), so I think atleast_1d() 
may be safer. I've taken this option for now.


This simplified the code somewhat. The *_scalar() functions have been 
removed, as they are no longer needed. The *_array() versions have been 
renamed, removing the _array suffix.


The return values have stayed as they were - if there is only one 
problem to solve, the singleton dimension is dropped, and otherwise a 2D 
array is returned.


(The exception is linear(), which does not need the second dimension, 
since the solution for each problem is unique. It will return a scalar 
in the case of a single problem, or a 1D array in the case of multiple 
problems.)



I've pushed the new version. It's available from the same repository:

https://yousource.it.jyu.fi/jjrandom2/miniprojects/trees/master/misc/polysolve_for_numpy


Other comments? The next step?

 -J


P.S.:

I'm not sure how exact the object type must be - i.e. whether Cython 
wants to know that the object stores its data somewhat like an ndarray, 
or that its C API exactly matches that of an ndarray.


In Cython there are some surprising details regarding this kind of 
things, such as the ctypedef'ing of scalar types. For example, see the 
comments about complex support near the beginning of polysolve2.pyx, and 
the commented-out SSE2 intrinsics experiment in 
https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/tworods.pyx 
.


In the latter, it was somewhat tricky to get Cython to recognize __m128d 
- turns out it's close enough for Cython to know that it behaves like a 
double, although it actually contains two doubles.


(Note that these ctypedefs never end up in the generated C code; Cython 
uses them as extra context information when mapping the Python code into C.)


(And no need to worry, I'm not planning to put any SSE into polysolve2 :) )



On 02.10.2015 17:18, Ryan May wrote:
numpy.asanyarray() would be my preferred goto, as it will leave 
subclasses of ndarray untouched; asarray() and atleast_1d() force 
ndarray. It's nice to do the whenever possible.


Ryan

On Fri, Oct 2, 2015 at 6:52 AM, Slavin, Jonathan 
mailto:jsla...@cfa.harvard.edu>> wrote:


​ Personally I like atleast_1d, which will convert a scalar into a
1d array but will leave arrays untouched (i.e. won't change the
dimensions.  Not sure what the advantages/disadvantages are
relative to asarray.

Jon​


On Fri, Oct 2, 2015 at 7:05 AM,
mailto:numpy-discussion-requ...@scipy.org>> wrote:

From: Juha Jeronen mailto:juha.jero...@jyu.fi>>
To: Discussion of Numerical Python mailto:numpy-discussion@scipy.org>>
Cc:
Date: Fri, 2 Oct 2015 13:31:47 +0300
Subject: Re: [Numpy-discussion] Cython-based
OpenMP-accelerated quartic polynomial solver

On 02.10.2015 13:07, Daπid wrote:


On 2 October 2015 at 11:58, Juha Jeronen mailto:juha.jero...@jyu.fi>> wrote:



First version done and uploaded:


https://yousource.it.jyu.fi/jjrandom2/miniprojects/trees/master/misc/polysolve_for_numpy


Small comment: now you are checking if the input is a scalar
or a ndarray, but it should also accept any array-like. If I
pass a list, I expect it to work, internally converting it
into an array.


Good catch.

Is there an official way to test for array-likes? Or should I
always convert with asarray()? Or something else?


 -J





-- 


Jonathan D. Slavin Harvard-Smithsonian CfA
jsla...@cfa.harvard.edu 60
Garden Street, MS 83
phone: (617) 496-7981Cambridge,
MA 02138-1516
cell: (781) 363-0035  USA



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org 
https://mail.scipy.org/mailman/listinfo/numpy-discussion




--
Ryan May



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Mon, Oct 5, 2015 at 11:26 PM, Nathaniel Smith  wrote:

> Hi all,
>
> For a long time, NumPy has supported two different ways of being compiled:
>
> "Separate compilation" mode: like most C projects, each .c file gets
> compiled to a .o file, and then the .o files get linked together to
> make a shared library. (This has been the default since 1.8.0.)
>
> "One file" mode: first concatenate all the .c files together to make
> one monster .c file, and then compile that .c file to make a shared
> library. (This was the default before 1.8.0.)
>
> Supporting these two different build modes creates a drag on
> development progress; in particular Stefan recently ran into this in
> this experiments with porting parts of the NumPy internals to Cython:
>   https://github.com/numpy/numpy/pull/6408
> (I suspect the particular problem he's running into can be fixed b/c
> so far he only has one .pyx file, but I also suspect that it will be
> impossible to support "one file" mode once we have multiple .pyx
> files.)
>
> There are some rumors that "one file" mode might be needed on some
> obscure platform somewhere, or that it might be necessary for
> statically linking numpy into the CPython executable, but we can't
> continue supporting things forever based only on rumors. If all we can
> get are rumors, then eventually we have to risk breaking things just
> to force anyone who cares to actually show up and explain what they
> need so we can support it properly :-).
>

Assuming one of the rumour is related to some comments I made some time
(years ?) earlier, the context was the ability to hide exported symbols. As
you know, the issue is not to build extensions w/ multiple compilation
units, but sharing functionalities between them without sharing them
outside the extension. I am just reiterating that point so that we all
discuss under the right context :)

I also agree the current situation is not sustainable -- as we discussed
privately before, cythonizing numpy.core is made quite more complicated by
this. I have myself quite a few issues w/ cythonizing the other parts of
umath. I would also like to support the static link better than we do now
(do we know some static link users we can contact to validate our approach
?)

Currently, what we have in numpy core is the following:

numpy.core.multiarray -> compilation units in numpy/core/src/multiarray/ +
statically link npymath
numpy.core.umath -> compilation units in numpy/core/src/umath + statically
link npymath/npysort + some shenanigans to use things in
numpy.core.multiarray

I would suggest to have a more layered approach, to enable both 'normal'
build and static build, without polluting the public namespace too much.
This is an approach followed by most large libraries (e.g. MKL), and is
fairly flexible.

Concretely, we could start by putting more common functionalities (aka the
'core' library) into its own static library. The API would be considered
private to numpy (no stability guaranteed outside numpy), and every
exported symbol from that library would be decorated appropriately to avoid
potential clashes (e.g. '_npy_internal_').

FWIW, that has always been my intention to go toward this when I split up
multiarray/umath into multiple .c files and extracted out npymath.

cheers,
David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Stephan Hoyer]

On Tue, Oct 6, 2015 at 1:14 AM, Daπid  wrote:

> One idea: what about creating a "parallel numpy"? There are a few
> algorithms that can benefit from parallelisation. This library would mimic
> Numpy's signature, and the user would be responsible for choosing the
> single threaded or the parallel one by just changing np.function(x, y) to
> pnp.function(x, y)
>

I would recommend taking a look at dask.array [1], which in many cases
works exactly like a parallel NumPy, though it also does lazy and
out-of-core computation. It's a new project, but it's remarkably mature --
we use it as an alternative array backend (to numpy) in xray, and it's also
being used by scikit-image.

[1] http://dask.pydata.org/en/latest/array.html


> If that were deemed a good one, what would be the best parallelisation
> scheme? OpenMP? Threads?
>

Dask uses threads. That works pretty well as long as all the hard work is
calling into something that releases the GIL (which includes NumPy, of
course).
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Daπid]

On 30 September 2015 at 18:20, Nathaniel Smith  wrote:

> - parallel code in general is not very composable. If someone is calling a
> numpy operation from one thread, great, transparently using multiple
> threads internally is a win. If they're exploiting some higher-level
> structure in their problem to break it into pieces and process each in
> parallel, and then using numpy on each piece, then numpy spawning threads
> internally will probably destroy performance. And numpy is too low-level to
> know which case it's in. This problem exists to some extent already with
> multi-threaded BLAS, so people use various BLAS-specific knobs to manage it
> in ad hoc ways, but this doesn't scale.
>
One idea: what about creating a "parallel numpy"? There are a few
algorithms that can benefit from parallelisation. This library would mimic
Numpy's signature, and the user would be responsible for choosing the
single threaded or the parallel one by just changing np.function(x, y) to
pnp.function(x, y)

If that were deemed a good one, what would be the best parallelisation
scheme? OpenMP? Threads?
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion