Re: [Numpy-discussion] numpy all unexpected result (generator)

[josef . pktd]

On Tue, Jan 31, 2012 at 5:35 PM, Travis Oliphant  wrote:
> Actually i believe the NumPy 'any' and 'all' names pre-date the Python usage 
> which first appeared in Python 2.5
>
> I agree with Chris that namespaces are a great idea.  I don't agree with 
> deprecating 'any' and 'all'

I completely agree here.

I also like to keep np.all, np.any, np.max, ...

>>> np.max((i>  0 for i in xrange (10)))
 at 0x046493F0>
>>> max((i>  0 for i in xrange (10)))
True

I used an old-style matplotlib example as recipe yesterday, and the
first thing I did is getting rid of the missing name spaces, and I had
to think twice what amax and amin are.

aall, aany ??? ;)

Josef

>
> It also seems useful to revisit under what conditions 'array' could correctly 
> interpret a generator expression, but in the context of streaming or deferred 
> arrays.
>
> Travis
>
>
> --
> Travis Oliphant
> (on a mobile)
> 512-826-7480
>
>
> On Jan 31, 2012, at 4:22 PM, Robert Kern  wrote:
>
>> On Tue, Jan 31, 2012 at 22:17, Travis Oliphant  wrote:
>>> I also agree that an exception should be raised at the very least.
>>>
>>> It might also be possible to make the NumPy any, all, and sum functions
>>> behave like the builtins when given a generator.  It seems worth exploring
>>> at least.
>>
>> I would rather we deprecate the all() and any() functions in favor of
>> the alltrue() and sometrue() aliases that date back to Numeric.
>> Renaming them to match the builtin names was a mistake.
>>
>> --
>> Robert Kern
>>
>> "I have come to believe that the whole world is an enigma, a harmless
>> enigma that is made terrible by our own mad attempt to interpret it as
>> though it had an underlying truth."
>>   -- Umberto Eco
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Travis Oliphant]

Actually i believe the NumPy 'any' and 'all' names pre-date the Python usage 
which first appeared in Python 2.5

I agree with Chris that namespaces are a great idea.  I don't agree with 
deprecating 'any' and 'all'

It also seems useful to revisit under what conditions 'array' could correctly 
interpret a generator expression, but in the context of streaming or deferred 
arrays.

Travis 


--
Travis Oliphant
(on a mobile)
512-826-7480


On Jan 31, 2012, at 4:22 PM, Robert Kern  wrote:

> On Tue, Jan 31, 2012 at 22:17, Travis Oliphant  wrote:
>> I also agree that an exception should be raised at the very least.
>> 
>> It might also be possible to make the NumPy any, all, and sum functions
>> behave like the builtins when given a generator.  It seems worth exploring
>> at least.
> 
> I would rather we deprecate the all() and any() functions in favor of
> the alltrue() and sometrue() aliases that date back to Numeric.
> Renaming them to match the builtin names was a mistake.
> 
> -- 
> Robert Kern
> 
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>   -- Umberto Eco
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Warren Weckesser]

On Tue, Jan 31, 2012 at 4:22 PM, Robert Kern  wrote:

> On Tue, Jan 31, 2012 at 22:17, Travis Oliphant 
> wrote:
> > I also agree that an exception should be raised at the very least.
> >
> > It might also be possible to make the NumPy any, all, and sum functions
> > behave like the builtins when given a generator.  It seems worth
> exploring
> > at least.
>
> I would rather we deprecate the all() and any() functions in favor of
> the alltrue() and sometrue() aliases that date back to Numeric.
>


+1  (Maybe 'anytrue' for consistency?  (And a royal blue bike shed?))

Warren
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Robert Kern]

On Tue, Jan 31, 2012 at 22:17, Travis Oliphant  wrote:
> I also agree that an exception should be raised at the very least.
>
> It might also be possible to make the NumPy any, all, and sum functions
> behave like the builtins when given a generator.  It seems worth exploring
> at least.

I would rather we deprecate the all() and any() functions in favor of
the alltrue() and sometrue() aliases that date back to Numeric.
Renaming them to match the builtin names was a mistake.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Travis Oliphant]

I also agree that an exception should be raised at the very least.

It might also be possible to make the NumPy any, all, and sum functions behave 
like the builtins when given a generator.  It seems worth exploring at least.

Travis 

--
Travis Oliphant
(on a mobile)
512-826-7480


On Jan 31, 2012, at 10:46 AM, Benjamin Root  wrote:

> 
> On Tue, Jan 31, 2012 at 10:05 AM, Olivier Delalleau  wrote:
> Le 31 janvier 2012 10:50, Robert Kern  a écrit :
> On Tue, Jan 31, 2012 at 15:35, Benjamin Root  wrote:
> >
> >
> > On Tue, Jan 31, 2012 at 9:18 AM, Robert Kern  wrote:
> >>
> >> On Tue, Jan 31, 2012 at 15:13, Benjamin Root  wrote:
> >>
> >> > Is np.all() using np.array() or np.asanyarray()?  If the latter, I would
> >> > expect it to return a numpy array from a generator.
> >>
> >> Why would you expect that?
> >>
> >> [~/scratch]
> >> |37> np.asanyarray(i>5 for i in range(10))
> >> array( at 0xdc24a08>, dtype=object)
> >>
> >> --
> >> Robert Kern
> >
> >
> > What possible use-case could there be for a numpy array of generators?
> 
> Not many. This isn't an intentional feature, just a logical
> consequence of all of the other intentional features being applied
> consistently.
> 
> > Furthermore, from the documentation:
> >
> > numpy.asanyarray = asanyarray(a, dtype=None, order=None, maskna=None,
> > ownmaskna=False)
> >  Convert the input to an ndarray, but pass ndarray subclasses through.
> >
> >  Parameters
> >  --
> >  a : array_like
> >  Input data, in any form that can be converted to an array.  This
> >  includes scalars, lists, lists of tuples, tuples, tuples of tuples,
> >  tuples of lists, and ndarrays.
> >
> > Emphasis mine.  A generator is an input that could be converted into an
> > array.  (Setting aside the issue of non-terminating generators such as those
> > from cycle()).
> 
> I'm sorry, but this is not true. In general, it's too hard to do all
> of the magic autodetermination that asarray() and array() do when
> faced with an indeterminate-length iterable. We tried. That's why we
> have fromiter(). By restricting the domain to an iterable yielding
> scalars and requiring that the user specify the desired dtype,
> fromiter() can figure out the rest.
> 
> Like it or not, "array_like" is practically defined by the behavior of
> np.asarray(), not vice-versa.
> 
> In that case I agree with whoever said ealier it would be best to detect this 
> case and throw an exception, as it'll probably save some headaches.
> 
> -=- Olivier
> 
> 
> I'll agree with this statement.  This bug has popped up a few times in the 
> mpl bug tracker due to the pylab mode.  While I would prefer if it were 
> possible to evaluate the generator into an array, silently returning True 
> incorrectly for all() and any() is probably far worse.
> 
> That said, is it still impossible to make np.all() and np.any() special to 
> have similar behavior to the built-in all() and any()?  Maybe it could catch 
> the above exception and then return the result from python's built-ins?
> 
> Cheers!
> Ben Root
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Unexpected reorganization of internal data

[Chris Barker]

On Tue, Jan 31, 2012 at 6:14 AM, Malcolm Reynolds
 wrote:
> Not exactly an answer to your question, but I can highly recommend
> using Boost.python, PyUblas and Ublas for your C++ vectors and
> matrices. It gives you a really good interface on the C++ side to
> numpy arrays and matrices, which can be passed in both directions over
> the language threshold with no copying.

or use Cython...

> If I had to guess I'd say sometimes when transposing numpy simply sets
> a flag internally to avoid copying the data, but in some cases (such
> as perhaps when multiplication needs to take place) the data has to be
> placed in a new object.

good guess:

> V = numpy.dot(R, U.transpose()).transpose()

>>> a
array([[1, 2],
   [3, 4],
   [5, 6]])
>>> a.flags
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

>>> b = a.transpose()
>>> b.flags
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

so the transpose() simple re-arranges the strides to Fortran order,
rather than changing anything in memory.

np.dot() produces a new array, so it is C-contiguous, then you
transpose it, so you get a fortran-ordered array.

> Now when I call my C++ function from the Python side, all the data in V is 
> printed, but it has been transposed.

as mentioned, if you are working with arrays in C++ (or fortran, orC,
or...) and need to count on the ordering of the data, you need to
check it in your extension code. There are utilities for this.

> However, if I do:

> V = numpy.array(U.transpose()).transpose()

right:

In [7]: a.flags
Out[7]:
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [8]: a.transpose().flags
Out[8]:
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [9]: np.array( a.transpose() ).flags
Out[9]:
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False


so the np.array call doesn't re-arrange the order if it doesn't need
to. If you want to force it, you can specify the order:

In [10]: np.array( a.transpose(), order='C' ).flags
Out[10]:
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False


(note: this does surprise me a bit, as it is making a copy, but there
you go -- if order matters, specify it)

In general, numpy does a lot of things for the sake of efficiency --
avoiding copies when it can, for instance -- this give efficiency and
flexibility, but you do need to be careful, particularly when
interfacing with the binary data directly.

-Chris






-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Chris Barker]

On Tue, Jan 31, 2012 at 6:33 AM, Neal Becker  wrote:
> The reason it surprised me, is that python 'all' doesn't behave as numpy 'all'
> in this respect - and using ipython, I didn't even notice that 'all' was
> numpy.all rather than standard python all.

"namespaces are one honking great idea"

-- sorry, I couldn't help myself




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Benjamin Root]

On Tue, Jan 31, 2012 at 10:05 AM, Olivier Delalleau  wrote:

> Le 31 janvier 2012 10:50, Robert Kern  a écrit :
>
>> On Tue, Jan 31, 2012 at 15:35, Benjamin Root  wrote:
>> >
>> >
>> > On Tue, Jan 31, 2012 at 9:18 AM, Robert Kern 
>> wrote:
>> >>
>> >> On Tue, Jan 31, 2012 at 15:13, Benjamin Root  wrote:
>> >>
>> >> > Is np.all() using np.array() or np.asanyarray()?  If the latter, I
>> would
>> >> > expect it to return a numpy array from a generator.
>> >>
>> >> Why would you expect that?
>> >>
>> >> [~/scratch]
>> >> |37> np.asanyarray(i>5 for i in range(10))
>> >> array( at 0xdc24a08>, dtype=object)
>> >>
>> >> --
>> >> Robert Kern
>> >
>> >
>> > What possible use-case could there be for a numpy array of generators?
>>
>> Not many. This isn't an intentional feature, just a logical
>> consequence of all of the other intentional features being applied
>> consistently.
>>
>> > Furthermore, from the documentation:
>> >
>> > numpy.asanyarray = asanyarray(a, dtype=None, order=None, maskna=None,
>> > ownmaskna=False)
>> >  Convert the input to an ndarray, but pass ndarray subclasses
>> through.
>> >
>> >  Parameters
>> >  --
>> >  a : array_like
>> >  Input data, in any form that can be converted to an array.
>> This
>> >  includes scalars, lists, lists of tuples, tuples, tuples of
>> tuples,
>> >  tuples of lists, and ndarrays.
>> >
>> > Emphasis mine.  A generator is an input that could be converted into an
>> > array.  (Setting aside the issue of non-terminating generators such as
>> those
>> > from cycle()).
>>
>> I'm sorry, but this is not true. In general, it's too hard to do all
>> of the magic autodetermination that asarray() and array() do when
>> faced with an indeterminate-length iterable. We tried. That's why we
>> have fromiter(). By restricting the domain to an iterable yielding
>> scalars and requiring that the user specify the desired dtype,
>> fromiter() can figure out the rest.
>>
>> Like it or not, "array_like" is practically defined by the behavior of
>> np.asarray(), not vice-versa.
>
>
> In that case I agree with whoever said ealier it would be best to detect
> this case and throw an exception, as it'll probably save some headaches.
>
> -=- Olivier
>
>
I'll agree with this statement.  This bug has popped up a few times in the
mpl bug tracker due to the pylab mode.  While I would prefer if it were
possible to evaluate the generator into an array, silently returning True
incorrectly for all() and any() is probably far worse.

That said, is it still impossible to make np.all() and np.any() special to
have similar behavior to the built-in all() and any()?  Maybe it could
catch the above exception and then return the result from python's
built-ins?

Cheers!
Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Robert Kern]

On Tue, Jan 31, 2012 at 15:35, Benjamin Root  wrote:

> Furthermore, from the documentation:
>
> numpy.asanyarray = asanyarray(a, dtype=None, order=None, maskna=None,
> ownmaskna=False)
>  Convert the input to an ndarray, but pass ndarray subclasses through.
>
>  Parameters
>  --
>  a : array_like
>  Input data, in any form that can be converted to an array.  This
>  includes scalars, lists, lists of tuples, tuples, tuples of tuples,
>  tuples of lists, and ndarrays.

I should also add that this verbiage is also in np.asarray(). The only
additional feature of np.asanyarray() is that is does not convert
ndarray subclasses like matrix to ndarray objects. np.asanyarray()
does not accept more types of objects than np.asarray().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Olivier Delalleau]

Le 31 janvier 2012 10:50, Robert Kern  a écrit :

> On Tue, Jan 31, 2012 at 15:35, Benjamin Root  wrote:
> >
> >
> > On Tue, Jan 31, 2012 at 9:18 AM, Robert Kern 
> wrote:
> >>
> >> On Tue, Jan 31, 2012 at 15:13, Benjamin Root  wrote:
> >>
> >> > Is np.all() using np.array() or np.asanyarray()?  If the latter, I
> would
> >> > expect it to return a numpy array from a generator.
> >>
> >> Why would you expect that?
> >>
> >> [~/scratch]
> >> |37> np.asanyarray(i>5 for i in range(10))
> >> array( at 0xdc24a08>, dtype=object)
> >>
> >> --
> >> Robert Kern
> >
> >
> > What possible use-case could there be for a numpy array of generators?
>
> Not many. This isn't an intentional feature, just a logical
> consequence of all of the other intentional features being applied
> consistently.
>
> > Furthermore, from the documentation:
> >
> > numpy.asanyarray = asanyarray(a, dtype=None, order=None, maskna=None,
> > ownmaskna=False)
> >  Convert the input to an ndarray, but pass ndarray subclasses
> through.
> >
> >  Parameters
> >  --
> >  a : array_like
> >  Input data, in any form that can be converted to an array.  This
> >  includes scalars, lists, lists of tuples, tuples, tuples of
> tuples,
> >  tuples of lists, and ndarrays.
> >
> > Emphasis mine.  A generator is an input that could be converted into an
> > array.  (Setting aside the issue of non-terminating generators such as
> those
> > from cycle()).
>
> I'm sorry, but this is not true. In general, it's too hard to do all
> of the magic autodetermination that asarray() and array() do when
> faced with an indeterminate-length iterable. We tried. That's why we
> have fromiter(). By restricting the domain to an iterable yielding
> scalars and requiring that the user specify the desired dtype,
> fromiter() can figure out the rest.
>
> Like it or not, "array_like" is practically defined by the behavior of
> np.asarray(), not vice-versa.


In that case I agree with whoever said ealier it would be best to detect
this case and throw an exception, as it'll probably save some headaches.

-=- Olivier
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Robert Kern]

On Tue, Jan 31, 2012 at 15:35, Benjamin Root  wrote:
>
>
> On Tue, Jan 31, 2012 at 9:18 AM, Robert Kern  wrote:
>>
>> On Tue, Jan 31, 2012 at 15:13, Benjamin Root  wrote:
>>
>> > Is np.all() using np.array() or np.asanyarray()?  If the latter, I would
>> > expect it to return a numpy array from a generator.
>>
>> Why would you expect that?
>>
>> [~/scratch]
>> |37> np.asanyarray(i>5 for i in range(10))
>> array( at 0xdc24a08>, dtype=object)
>>
>> --
>> Robert Kern
>
>
> What possible use-case could there be for a numpy array of generators?

Not many. This isn't an intentional feature, just a logical
consequence of all of the other intentional features being applied
consistently.

> Furthermore, from the documentation:
>
> numpy.asanyarray = asanyarray(a, dtype=None, order=None, maskna=None,
> ownmaskna=False)
>  Convert the input to an ndarray, but pass ndarray subclasses through.
>
>  Parameters
>  --
>  a : array_like
>  Input data, in any form that can be converted to an array.  This
>  includes scalars, lists, lists of tuples, tuples, tuples of tuples,
>  tuples of lists, and ndarrays.
>
> Emphasis mine.  A generator is an input that could be converted into an
> array.  (Setting aside the issue of non-terminating generators such as those
> from cycle()).

I'm sorry, but this is not true. In general, it's too hard to do all
of the magic autodetermination that asarray() and array() do when
faced with an indeterminate-length iterable. We tried. That's why we
have fromiter(). By restricting the domain to an iterable yielding
scalars and requiring that the user specify the desired dtype,
fromiter() can figure out the rest.

Like it or not, "array_like" is practically defined by the behavior of
np.asarray(), not vice-versa.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Alan G Isaac]

On 1/31/2012 10:35 AM, Benjamin Root wrote:
> A generator is an input that could be converted into an array.


def mygen():
   i = 0
   while True:
 yield i
 i += 1

Alan Isaac

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Dag Sverre Seljebotn]

On 01/31/2012 04:35 PM, Benjamin Root wrote:
>
>
> On Tue, Jan 31, 2012 at 9:18 AM, Robert Kern  > wrote:
>
> On Tue, Jan 31, 2012 at 15:13, Benjamin Root  > wrote:
>
>  > Is np.all() using np.array() or np.asanyarray()?  If the latter,
> I would
>  > expect it to return a numpy array from a generator.
>
> Why would you expect that?
>
> [~/scratch]
> |37> np.asanyarray(i>5 for i in range(10))
> array( at 0xdc24a08>, dtype=object)
>
> --
> Robert Kern
>
>
> What possible use-case could there be for a numpy array of generators?
> Furthermore, from the documentation:
>
> numpy.asanyarray = asanyarray(a, dtype=None, order=None, maskna=None,
> ownmaskna=False)
>   Convert the input to an ndarray, but pass ndarray subclasses through.
>
>   Parameters
>   --
>   a : array_like
> *Input data, in any form that can be converted to an array*.  This
>   includes scalars, lists, lists of tuples, tuples, tuples of
> tuples,
>   tuples of lists, and ndarrays.
>
> Emphasis mine.  A generator is an input that could be converted into an
> array.  (Setting aside the issue of non-terminating generators such as
> those from cycle()).

Splitting semantic hairs doesn't help here -- it *does* return an array, 
it just happens to be a completely useless 0-dimensional one.

The question is, is the current confusing and less than useful? (I vot 
for "yes"). list and tuple are special-cased, why not generators (at 
least to raise an exception)

Going OT, look at this gem:



In [3]: a
Out[3]: array([1, 2, 3], dtype=object)

In [4]: a.shape
Out[4]: ()

???

In [9]: b
Out[9]: array([1, 2, 3], dtype=object)

In [10]: b.shape
Out[10]: (3,)

Figuring out the "???" is left as an exercise to the reader :-)

Dag Sverre
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Benjamin Root]

On Tue, Jan 31, 2012 at 9:18 AM, Robert Kern  wrote:

> On Tue, Jan 31, 2012 at 15:13, Benjamin Root  wrote:
>
> > Is np.all() using np.array() or np.asanyarray()?  If the latter, I would
> > expect it to return a numpy array from a generator.
>
> Why would you expect that?
>
> [~/scratch]
> |37> np.asanyarray(i>5 for i in range(10))
> array( at 0xdc24a08>, dtype=object)
>
> --
> Robert Kern
>

What possible use-case could there be for a numpy array of generators?
Furthermore, from the documentation:

numpy.asanyarray = asanyarray(a, dtype=None, order=None, maskna=None,
ownmaskna=False)
 Convert the input to an ndarray, but pass ndarray subclasses through.

 Parameters
 --
 a : array_like
 *Input data, in any form that can be converted to an array*.  This
 includes scalars, lists, lists of tuples, tuples, tuples of tuples,
 tuples of lists, and ndarrays.

Emphasis mine.  A generator is an input that could be converted into an
array.  (Setting aside the issue of non-terminating generators such as
those from cycle()).

Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Dag Sverre Seljebotn]

On 01/31/2012 04:13 PM, Benjamin Root wrote:
>
>
> On Tuesday, January 31, 2012, Alan G Isaac  > wrote:
>  > On 1/31/2012 8:26 AM, Neal Becker wrote:
>  >> I was just bitten by this unexpected behavior:
>  >>
>  >> In [24]: all ([i>   0 for i in xrange (10)])
>  >> Out[24]: False
>  >>
>  >> In [25]: all (i>   0 for i in xrange (10))
>  >> Out[25]: True
>  >>
>  >> Turns out:
>  >> In [31]: all is numpy.all
>  >> Out[31]: True
>  >
>  >
>   np.array([i>  0 for i in xrange (10)])
>  > array([False,  True,  True,  True,  True,  True,  True,  True,  True,
>   True], dtype=bool)
>   np.array(i>  0 for i in xrange (10))
>  > array( at 0x0267A210>, dtype=object)
>   import this
>  >
>  >
>  > Cheers,
>  > Alan
>  >
>
> Is np.all() using np.array() or np.asanyarray()?  If the latter, I would
> expect it to return a numpy array from a generator.  If the former, why
> isn't it using asanyarray()?

Your expectation is probably wrong:

In [12]: np.asanyarray(i for i in range(10))
Out[12]: array( at 0x455d9b0>, dtype=object)

Dag Sverre
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Robert Kern]

On Tue, Jan 31, 2012 at 15:13, Benjamin Root  wrote:

> Is np.all() using np.array() or np.asanyarray()?  If the latter, I would
> expect it to return a numpy array from a generator.

Why would you expect that?

[~/scratch]
|37> np.asanyarray(i>5 for i in range(10))
array( at 0xdc24a08>, dtype=object)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Benjamin Root]

On Tuesday, January 31, 2012, Alan G Isaac  wrote:
> On 1/31/2012 8:26 AM, Neal Becker wrote:
>> I was just bitten by this unexpected behavior:
>>
>> In [24]: all ([i>   0 for i in xrange (10)])
>> Out[24]: False
>>
>> In [25]: all (i>   0 for i in xrange (10))
>> Out[25]: True
>>
>> Turns out:
>> In [31]: all is numpy.all
>> Out[31]: True
>
>
 np.array([i>  0 for i in xrange (10)])
> array([False,  True,  True,  True,  True,  True,  True,  True,  True,
 True], dtype=bool)
 np.array(i>  0 for i in xrange (10))
> array( at 0x0267A210>, dtype=object)
 import this
>
>
> Cheers,
> Alan
>

Is np.all() using np.array() or np.asanyarray()?  If the latter, I would
expect it to return a numpy array from a generator.  If the former, why
isn't it using asanyarray()?

Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Alan G Isaac]

On 1/31/2012 8:26 AM, Neal Becker wrote:
> I was just bitten by this unexpected behavior:
>
> In [24]: all ([i>   0 for i in xrange (10)])
> Out[24]: False
>
> In [25]: all (i>   0 for i in xrange (10))
> Out[25]: True
>
> Turns out:
> In [31]: all is numpy.all
> Out[31]: True


>>> np.array([i>  0 for i in xrange (10)])
array([False,  True,  True,  True,  True,  True,  True,  True,  True,  True], 
dtype=bool)
>>> np.array(i>  0 for i in xrange (10))
array( at 0x0267A210>, dtype=object)
>>> import this


Cheers,
Alan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Unexpected reorganization of internal data

[Matthew Brett]

Hi,

On Tue, Jan 31, 2012 at 8:29 AM, Mads Ipsen  wrote:
> Hi,
>
> I am confused. Here's the reason:
>
> The following structure is a representation of N points in 3D space:
>
> U = numpy.array([[x1,y1,z1], [x1,y1,z1],...,[xn,yn,zn]])
>
> So the array U has shape (N,3). This order makes sense to me since U[i] will
> give you the i'th point in the set. Now, I want to pass this array to a C++
> function that does some stuff with the points. Here's how I do that
>
> void Foo::doStuff(int n, PyObject * numpy_data)
> {
>     // Get pointer to data
>     double * const positions = (double *) PyArray_DATA(numpy_data);
>
>     // Print positions
>     for (int i=0; i     {
>     float x = static_cast(positions[3*i+0])
>     float y = static_cast(positions[3*i+1])
>     float z = static_cast(positions[3*i+2])
>
>     printf("Pos[%d] = %f %f %f\n", x, y, z);
>     }
> }
>
> When I call this routine, using a swig wrapped Python interface to the C++
> class, everything prints out nice.
>
> Now, I want to apply a rotation to all the positions. So I set up some
> rotation matrix R like this:
>
> R = numpy.array([[r11,r12,r13],
>  [r21,r22,r23],
>  [r31,r32,r33]])
>
> To apply the matrix to the data in one crunch, I do
>
> V = numpy.dot(R, U.transpose()).transpose()
>
> Now when I call my C++ function from the Python side, all the data in V is
> printed, but it has been transposed. So apparently the internal data
> structure handled by numpy has been reorganized, even though I called
> transpose() twice, which I would expect to cancel out each other.
>
> However, if I do:
>
> V = numpy.array(U.transpose()).transpose()
>
> and call the C++ routine, everything is perfectly fine, ie. the data
> structure is as expected.
>
> What went wrong?

The numpy array reserves the right to organize its data internally.
For example, a numpy array can be in Fortran order in memory, or C
order in memory, and many more complicated schemes.  You might want to
have a look at:

http://docs.scipy.org/doc/numpy/reference/arrays.ndarray.html#internal-memory-layout-of-an-ndarray

If you depend on a particular order for your array memory, you might
want to look at:

http://docs.scipy.org/doc/numpy/reference/generated/numpy.ascontiguousarray.html

Best,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Neal Becker]

Dag Sverre Seljebotn wrote:

> On 01/31/2012 03:07 PM, Robert Kern wrote:
>> On Tue, Jan 31, 2012 at 13:26, Neal Becker  wrote:
>>> I was just bitten by this unexpected behavior:
>>>
>>> In [24]: all ([i>0 for i in xrange (10)])
>>> Out[24]: False
>>>
>>> In [25]: all (i>0 for i in xrange (10))
>>> Out[25]: True
>>>
>>> Turns out:
>>> In [31]: all is numpy.all
>>> Out[31]: True
>>>
>>> So numpy.all doesn't seem to do what I would expect when given a generator.
>>> Bug?
>>
>> Expected behavior. numpy.all(), like nearly all numpy functions,
>> converts the input to an array using numpy.asarray(). numpy.asarray()
>> knows nothing special about generators and other iterables that are
>> not sequences, so it thinks it's a single scalar object. This scalar
>> object happens to have a __nonzero__() method that returns True like
>> most Python objects that don't override this.
>>
>> In order to use generic iterators that are not sequences, you need to
>> explicitly use numpy.fromiter() to convert them to ndarrays. asarray()
>> and array() can't do it in general because they need to autodiscover
>> the shape and dtype all at the same time.
> 
> Perhaps np.asarray could specifically check for a generator argument and
> raise an exception? I imagine that would save people some time when
> running into this...
> 
> If you really want
> 
> In [7]: x = np.asarray(None)
> 
> In [8]: x[()] = (i for i in range(10))
> 
> In [9]: x
> Out[9]: array( at 0x4553fa0>, dtype=object)
> 
> ...then one can type it out?
> 
> Dag

The reason it surprised me, is that python 'all' doesn't behave as numpy 'all' 
in this respect - and using ipython, I didn't even notice that 'all' was 
numpy.all rather than standard python all.  All in all, rather unfortunate :)

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Unexpected reorganization of internal data

[Malcolm Reynolds]

Not exactly an answer to your question, but I can highly recommend
using Boost.python, PyUblas and Ublas for your C++ vectors and
matrices. It gives you a really good interface on the C++ side to
numpy arrays and matrices, which can be passed in both directions over
the language threshold with no copying.

If I had to guess I'd say sometimes when transposing numpy simply sets
a flag internally to avoid copying the data, but in some cases (such
as perhaps when multiplication needs to take place) the data has to be
placed in a new object. Accessing the data via raw pointers in C++ may
not be checking for the 'transpose' flag and therefore you see an
unexpected result.

Disclaimer: this is just a guess, someone more familiar with Numpy
internals will no doubt be able to correct me.

Malcolm
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Dag Sverre Seljebotn]

On 01/31/2012 03:07 PM, Robert Kern wrote:
> On Tue, Jan 31, 2012 at 13:26, Neal Becker  wrote:
>> I was just bitten by this unexpected behavior:
>>
>> In [24]: all ([i>0 for i in xrange (10)])
>> Out[24]: False
>>
>> In [25]: all (i>0 for i in xrange (10))
>> Out[25]: True
>>
>> Turns out:
>> In [31]: all is numpy.all
>> Out[31]: True
>>
>> So numpy.all doesn't seem to do what I would expect when given a generator.
>> Bug?
>
> Expected behavior. numpy.all(), like nearly all numpy functions,
> converts the input to an array using numpy.asarray(). numpy.asarray()
> knows nothing special about generators and other iterables that are
> not sequences, so it thinks it's a single scalar object. This scalar
> object happens to have a __nonzero__() method that returns True like
> most Python objects that don't override this.
>
> In order to use generic iterators that are not sequences, you need to
> explicitly use numpy.fromiter() to convert them to ndarrays. asarray()
> and array() can't do it in general because they need to autodiscover
> the shape and dtype all at the same time.

Perhaps np.asarray could specifically check for a generator argument and 
raise an exception? I imagine that would save people some time when 
running into this...

If you really want

In [7]: x = np.asarray(None)

In [8]: x[()] = (i for i in range(10))

In [9]: x
Out[9]: array( at 0x4553fa0>, dtype=object)

...then one can type it out?

Dag
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy all unexpected result (generator)

[Robert Kern]

On Tue, Jan 31, 2012 at 13:26, Neal Becker  wrote:
> I was just bitten by this unexpected behavior:
>
> In [24]: all ([i>  0 for i in xrange (10)])
> Out[24]: False
>
> In [25]: all (i>  0 for i in xrange (10))
> Out[25]: True
>
> Turns out:
> In [31]: all is numpy.all
> Out[31]: True
>
> So numpy.all doesn't seem to do what I would expect when given a generator.
> Bug?

Expected behavior. numpy.all(), like nearly all numpy functions,
converts the input to an array using numpy.asarray(). numpy.asarray()
knows nothing special about generators and other iterables that are
not sequences, so it thinks it's a single scalar object. This scalar
object happens to have a __nonzero__() method that returns True like
most Python objects that don't override this.

In order to use generic iterators that are not sequences, you need to
explicitly use numpy.fromiter() to convert them to ndarrays. asarray()
and array() can't do it in general because they need to autodiscover
the shape and dtype all at the same time.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Unexpected reorganization of internal data

[Mads Ipsen]

Hi,

I am confused. Here's the reason:

The following structure is a representation of N points in 3D space:

U = numpy.array([[x1,y1,z1], [x1,y1,z1],...,[xn,yn,zn]])

So the array U has shape (N,3). This order makes sense to me since U[i] 
will give you the i'th point in the set. Now, I want to pass this array 
to a C++ function that does some stuff with the points. Here's how I do 
that


void Foo::doStuff(int n, PyObject * numpy_data)
{
// Get pointer to data
double * const positions = (double *) PyArray_DATA(numpy_data);

// Print positions
for (int i=0; i(positions[3*i+0])
float y = static_cast(positions[3*i+1])
float z = static_cast(positions[3*i+2])

printf("Pos[%d] = %f %f %f\n", x, y, z);
}
}

When I call this routine, using a swig wrapped Python interface to the 
C++ class, everything prints out nice.


Now, I want to apply a rotation to all the positions. So I set up some 
rotation matrix R like this:


R = numpy.array([[r11,r12,r13],
 [r21,r22,r23],
 [r31,r32,r33]])

To apply the matrix to the data in one crunch, I do

V = numpy.dot(R, U.transpose()).transpose()

Now when I call my C++ function from the Python side, all the data in V 
is printed, but it has been transposed. So apparently the internal data 
structure handled by numpy has been reorganized, even though I called 
transpose() twice, which I would expect to cancel out each other.


However, if I do:

V = numpy.array(U.transpose()).transpose()

and call the C++ routine, everything is perfectly fine, ie. the data 
structure is as expected.


What went wrong?

Best regards,

Mads

--
+-+
| Mads Ipsen  |
+--+--+
| Gåsebæksvej 7, 4. tv |  |
| DK-2500 Valby| phone:  +45-29716388 |
| Denmark  | email:  mads.ip...@gmail.com |
+--+--+


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] histogram help

[Nadav Horesh]

Do you want a histogramm of z for each (x,y) ?

   Nadav


From: numpy-discussion-boun...@scipy.org [numpy-discussion-boun...@scipy.org] 
On Behalf Of Ruby Stevenson [ruby...@gmail.com]
Sent: 30 January 2012 21:27
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] histogram help

Sorry, I realize I didn't describe the problem completely clear or correct.

the (x,y) in this case is just many co-ordinates, and  each coordinate
has a list of values (Z value) associated with it.  The bins are
allocated for the Z.

I hope this clarify things a little. Thanks again.

Ruby




On Mon, Jan 30, 2012 at 2:21 PM, Ruby Stevenson  wrote:
> hi, all
>
> I am trying to figure out how to do histogram with numpy
>
> I have a three-dimension array A[x,y,z],  another array (bins) has
> been allocated along Z dimension, z'
>
> how can I get the histogram of H[ x, y, z' ]?
>
> thanks for your help.
>
> Ruby
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] numpy all unexpected result (generator)

[Neal Becker]

I was just bitten by this unexpected behavior:

In [24]: all ([i>  0 for i in xrange (10)])
Out[24]: False

In [25]: all (i>  0 for i in xrange (10))
Out[25]: True

Turns out:
In [31]: all is numpy.all
Out[31]: True

So numpy.all doesn't seem to do what I would expect when given a generator.  
Bug?

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] [ANN] Summer School "Advanced Scientific Programming in Python" in Kiel, Germany

[Tiziano Zito]

Advanced Scientific Programming in Python
=
a Summer School by the G-Node and the Institute of Experimental and
Applied Physics, Christian-Albrechts-Universität zu Kiel

Scientists spend more and more time writing, maintaining, and
debugging software. While techniques for doing this efficiently have
evolved, only few scientists actually use them. As a result, instead
of doing their research, they spend far too much time writing
deficient code and reinventing the wheel. In this course we will
present a selection of advanced programming techniques,
incorporating theoretical lectures and practical exercises tailored
to the needs of a programming scientist. New skills will be tested
in a real programming project: we will team up to develop an
entertaining scientific computer game.

We use the Python programming language for the entire course. Python
works as a simple programming language for beginners, but more
importantly, it also works great in scientific simulations and data
analysis. We show how clean language design, ease of extensibility,
and the great wealth of open source libraries for scientific
computing and data visualization are driving Python to become a
standard tool for the programming scientist.

This school is targeted at Master or PhD students and Post-docs from
all areas of science. Competence in Python or in another language
such as Java, C/C++, MATLAB, or Mathematica is absolutely required.
Basic knowledge of Python is assumed. Participants without any prior
experience with Python should work through the proposed introductory
materials before the course. 

Date and Location
=
September 2—7, 2012. Kiel, Germany.

Preliminary Program
===
Day 0 (Sun Sept 2) — Best Programming Practices
  - Best Practices, Development Methodologies and the Zen of Python
  - Version control with git
  - Object-oriented programming & design patterns
Day 1 (Mon Sept 3) — Software Carpentry
  - Test-driven development, unit testing & quality assurance
  - Debugging, profiling and benchmarking techniques
  - Best practices in data visualization
  - Programming in teams
Day 2 (Tue Sept 4) — Scientific Tools for Python
  - Advanced NumPy
  - The Quest for Speed (intro): Interfacing to C with Cython
  - Advanced Python I: idioms, useful built-in data structures, 
generators
Day 3 (Wed Sept 5) — The Quest for Speed
  - Writing parallel applications in Python
  - Programming project
Day 4 (Thu Sept 6) — Efficient Memory Management
  - When parallelization does not help:
the starving CPUs problem
  - Advanced Python II: decorators and context managers
  - Programming project
Day 5 (Fri Sept 7) — Practical Software Development
  - Programming project
  - The Pelita Tournament

Every evening we will have the tutors' consultation hour: Tutors
will answer your questions and give suggestions for your own
projects.

Applications

You can apply on-line at http://python.g-node.org

Applications must be submitted before 23:59 UTC, May 1, 2012.
Notifications of acceptance will be sent by June 1, 2012.

No fee is charged but participants should take care of travel,
living, and accommodation expenses.  Candidates will be selected on
the basis of their profile. Places are limited: acceptance rate last
time was around 20%.  Prerequisites: You are supposed to know the
basics of Python to participate in the lectures. You are encouraged
to go through the introductory material available on the website.

Faculty
===
  - Francesc Alted, Continuum Analytics Inc., USA
  - Pietro Berkes, Enthought Inc., UK
  - Valentin Haenel,  Blue Brain Project, École Polytechnique
Fédérale de Lausanne, Switzerland
  - Zbigniew Jędrzejewski-Szmek, Faculty of Physics, University of
Warsaw, Poland
  - Eilif Muller, Blue Brain Project, École Polytechnique Fédérale
de Lausanne, Switzerland
  - Emanuele Olivetti, NeuroInformatics Laboratory, Fondazione Bruno
Kessler and University of Trento, Italy
  - Rike-Benjamin Schuppner,  Technologit GbR, Germany
  - Bartosz Teleńczuk, Unité de Neurosciences Information et
Complexité, Centre National de la Recherche Scientifique, France
  - Stéfan van der Walt, Helen Wills Neuroscience Institute,
University of California Berkeley, USA
  - Bastian Venthur, Berlin Institute of Technology and Bernstein
Focus Neurotechnology, Germany
  - Niko Wilbert, TNG Technology Consulting GmbH, Germany
  - Tiziano Zito, Institute for Theoretical Biology,
Humboldt-Universität zu Berlin, Germany

Organized by Christian T. Steigies and Christian Drews of the
Institute of Experimental and Applied Physics,
Christian-Albrechts-Universität zu Kiel , and by Zbigniew
Jędrzejewski-Szmek and Tiziano Zito for the German Neuroinformatics
Node of the INCF. 

Website:  http://python.g-node.org
Contact:  python-i...@g-node.org

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org