Re: [Numpy-discussion] Question about broadcasting vs for loop performance

2014-09-15 Thread Benjamin Root
Broadcasting, by itself, should not be creating large arrays in memory. It
uses stride tricks to make the array appear larger, while simply reusing
the same memory block. This is why it is so valuable because it doesn't
make a copy.

Now, what may be happening is that the resulting calculation from the
broadcasted arrays is too large to easily fit into the cpu cache, so the
subsequent summation might be hitting performance penalties for that.
Essentially, your first example may be a poor-man's implementation of data
chunking. I bet if you ran these performance metrics over a wide range of
sizes, you will see some interesting results.

Cheers!
Ben Root


On Sun, Sep 14, 2014 at 10:53 PM, Ryan Nelson  wrote:

> I think I figured out my own question. I guess that the broadcasting
> approach is generating a very large 2D array in memory, which takes a bit
> of extra time. I gathered this from reading the last example on the
> following site:
> http://wiki.scipy.org/EricsBroadcastingDoc
> I tried this again with a much smaller "xs" array (~100 points) and the
> broadcasting version was much faster.
> Thanks
>
> Ryan
>
> Note: The link to the Scipy wiki page above is broken at the bottom of
> Numpy's broadcasting page, otherwise I would have seen that earlier. Sorry
> for the noise.
>
> On Sun, Sep 14, 2014 at 10:22 PM, Ryan Nelson 
> wrote:
>
>> Hello all,
>>
>> I have a question about the performance of broadcasting versus Python for
>> loops. I have the following sample code that approximates some simulation
>> I'd like to do:
>>
>> ## Test Code ##
>>
>> import numpy as np
>>
>>
>> def lorentz(x, pos, inten, hwhm):
>>
>> return inten*( hwhm**2 / ( (x - pos)**2 + hwhm**2 ) )
>>
>>
>> poss = np.random.rand(100)
>>
>> intens = np.random.rand(100)
>>
>> xs = np.linspace(0,10,1)
>>
>>
>> def first_try():
>>
>> sim_inten = np.zeros(xs.shape)
>>
>> for freq, inten in zip(poss, intens):
>>
>> sim_inten += lorentz(xs, freq, inten, 5.0)
>>
>> return sim_inten
>>
>>
>> def second_try():
>>
>> sim_inten2 = lorentz(xs.reshape((-1,1)), poss, intens, 5.0)
>>
>> sim_inten2 = sim_inten2.sum(axis=1)
>>
>> return sim_inten2
>>
>>
>> print np.array_equal(first_try(), second_try())
>>
>>
>> ## End Test ##
>>
>>
>> Running this script prints "True" for the final equality test. However,
>> IPython's %timeit magic, gives ~10 ms for first_try and ~30 ms for
>> second_try. I tried this on Windows 7 (Anaconda Python) and on a Linux
>> machine both with Python 2.7 and Numpy 1.8.2.
>>
>>
>> I understand in principle why broadcasting should be faster than Python
>> loops, but I'm wondering why I'm getting worse results with the pure Numpy
>> function. Is there some general rules for when broadcasting might give
>> worse performance than a Python loop?
>>
>>
>> Thanks
>>
>>
>> Ryan
>>
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Linear algebra functions on empty arrays

2014-09-15 Thread josef.pktd
On Mon, Sep 15, 2014 at 7:26 AM, Sebastian Berg
 wrote:
> On Mo, 2014-09-15 at 07:07 -0400, josef.p...@gmail.com wrote:
>> On Mon, Sep 15, 2014 at 5:48 AM, Sebastian Berg
>>  wrote:
>> > Hey all,
>> >
>> > for https://github.com/numpy/numpy/pull/3861/files I would like to allow
>> > 0-sized dimensions for generalized ufuncs, meaning that the gufunc has
>> > to be able to handle this, but also that it *can* handle it at all.
>> > However lapack does not support this, so it needs some explicit fixing.
>> > Also some of the linalg functions currently explicitly allow and others
>> > explicitly disallow empty arrays.
>> >
>> > For example the QR and eigvals does not allow it, but on the other hand
>> > solve explicitly does (most probably never did, simply because lapack
>> > does not). So I am wondering if there is some convention for this, or
>> > what convention we should implement.
>>
>> What does an empty square matrix/array look like?
>>
>> np.linalg.solve   can have empty rhs, but shape of empty lhs, `a`, is ?
>>
>> If I do a QR(arr)  with arr.shape=(0, 5), what is R supposed to be ?
>>
>
> QR may be more difficult since R may itself could not be empty, begging
> the question if you want to error out or fill it sensibly.

I shouldn't have tried it again (I got this a few times last week):

>>> ze = np.ones((z.shape[1], 0))
>>> np.linalg.qr(ze)
 ** On entry to DGEQRF parameter number  7 had an illegal value

crash

z.shape[1]  is 3
>>> np.__version__
'1.6.1'

I think, I would prefer an exception if the output would require a
empty square matrix with shape > (0, 0)
I don't see any useful fill value.


> Cholesky would require (0, 0) for example and for eigenvalues it would
> somewhat make sense too, the (0, 0) matrix has 0 eigenvalues.
> I did not go through them all, but I would like to figure out whether we
> should aim to generally allow it, or maybe just allow it for some
> special ones.

If the return square array has shape (0, 0), then it would make sense,
but I haven't run into a case for it yet.

np.cholesky(np.ones((0, 0)))  ?
(I didn't try since my interpreter is crashed. :)

Josef

>
> - Sebastian
>
>>
>> I just wrote some loops over linalg.qr, but I always initialized explicitly.
>>
>> I didn't manage to figure out how empty arrays would be useful.
>>
>> If an empty square matrix can only only be of shape (0, 0), then it's
>> no use (in my applications).
>>
>>
>> Josef
>>
>>
>> >
>> > Regards,
>> >
>> > Sebastian
>> >
>> > ___
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion@scipy.org
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Tracking and inspecting numpy objects

2014-09-15 Thread Robert Kern
On Mon, Sep 15, 2014 at 12:31 PM, Eelco Hoogendoorn
 wrote:
> I figured the reference to the object through the local scope would also be
> tracked by the GC somehow, since the entire stack frame can be regarded as a
> separate object itself, but apparently not.

Objects are "tracked", not references. An object is only considered
"tracked" by the cyclic GC if the cyclic GC needs to traverse the
object's referents when looking for cycles. It is not necessarily
"tracked" just because it is referred to by an object that is
"tracked". If an object does not refer to other objects, *OR* if such
references cannot create a cycle for whatever reason, then the object
does not need to be tracked by the GC. Most ndarrays fall in that
second category. They do have references to their base ndarray, if a
view, and their dtype object, but under most circumstances these
cannot cause a cycle. As Sebastian mentioned, though, dtype=object
arrays can create cycles and should be tracked. We need to implement
the "tp_traverse" slot in order to do so.

-- 
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Tracking and inspecting numpy objects

2014-09-15 Thread Eelco Hoogendoorn
I figured the reference to the object through the local scope would also be
tracked by the GC somehow, since the entire stack frame can be regarded as
a separate object itself, but apparently not.

On Mon, Sep 15, 2014 at 1:06 PM, Mads Ipsen  wrote:

> Thanks to everybody for taking time to answer!
>
> Best regards,
>
> Mads
>
> On 15/09/14 12:11, Sebastian Berg wrote:
> > On Mo, 2014-09-15 at 12:05 +0200, Eelco Hoogendoorn wrote:
> >>
> >>
> >>
> >> On Mon, Sep 15, 2014 at 11:55 AM, Sebastian Berg
> >>  wrote:
> >>  On Mo, 2014-09-15 at 10:16 +0200, Mads Ipsen wrote:
> >>  > Hi,
> >>  >
> >>  > I am trying to inspect the reference count of numpy arrays
> >>  generated by
> >>  > my application.
> >>  >
> >>  > Initially, I thought I could inspect the tracked objects
> >>  using
> >>  > gc.get_objects(), but, with respect to numpy objects, the
> >>  returned
> >>  > container is empty. For example:
> >>  >
> >>  > import numpy
> >>  > import gc
> >>  >
> >>  > data = numpy.ones(1024).reshape((32,32))
> >>  >
> >>  > objs = [o for o in gc.get_objects() if isinstance(o,
> >>  numpy.ndarray)]
> >>  >
> >>  > print objs# Prints empty list
> >>  > print gc.is_tracked(data) # Print False
> >>  >
> >>  > Why is this? Also, is there some other technique I can use
> >>  to inspect
> >>  > all numpy generated objects?
> >>  >
> >>
> >>  Two reasons. First of all, unless your array is an object
> >>  arrays (or a
> >>  structured one with objects in it), there are no objects to
> >>  track. The
> >>  array is a single python object without any referenced objects
> >>  (except
> >>  possibly its `arr.base`).
> >>
> >>  Second of all -- and this is an issue -- numpy doesn't
> >>  actually
> >>  implement the traverse slot, so it won't even work for object
> >>  arrays
> >>  (numpy object arrays do not support circular garbage
> >>  collection at this
> >>  time, please feel free to implement it ;)).
> >>
> >>  - Sebastian
> >>
> >>
> >>
> >>
> >>
> >>
> >> Does this answer why the ndarray object itself isn't tracked though? I
> >> must say I find this puzzling; the only thing I can think of is that
> >> the python compiler notices that data isn't used anymore after its
> >> creation, and deletes it right after its creation as an optimization,
> >> but that conflicts with my own experience of the GC.
> >>
> >>
> >
> > Not sure if it does, but my quick try and error says:
> > In [15]: class T(tuple):
> > : pass
> > :
> >
> > In [16]: t = T()
> >
> > In [17]: objs = [o for o in gc.get_objects() if isinstance(o, T)]
> >
> > In [18]: objs
> > Out[18]: [()]
> >
> > In [19]: a = 123.
> >
> > In [20]: objs = [o for o in gc.get_objects() if isinstance(o, float)]
> >
> > In [21]: objs
> > Out[21]: []
> >
> > So I guess nothing is tracked, unless it contains things, and numpy
> > arrays don't say they can contain things (i.e. no traverse).
> >
> > - Sebastian
> >
> >
> >
> >> ___
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion@scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> --
> +-+
> | Mads Ipsen  |
> +--+--+
> | Gåsebæksvej 7, 4. tv | phone:  +45-29716388 |
> | DK-2500 Valby| email:  mads.ip...@gmail.com |
> | Denmark  | map  :   www.tinyurl.com/ns52fpa |
> +--+--+
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Linear algebra functions on empty arrays

2014-09-15 Thread Sebastian Berg
On Mo, 2014-09-15 at 07:07 -0400, josef.p...@gmail.com wrote:
> On Mon, Sep 15, 2014 at 5:48 AM, Sebastian Berg
>  wrote:
> > Hey all,
> >
> > for https://github.com/numpy/numpy/pull/3861/files I would like to allow
> > 0-sized dimensions for generalized ufuncs, meaning that the gufunc has
> > to be able to handle this, but also that it *can* handle it at all.
> > However lapack does not support this, so it needs some explicit fixing.
> > Also some of the linalg functions currently explicitly allow and others
> > explicitly disallow empty arrays.
> >
> > For example the QR and eigvals does not allow it, but on the other hand
> > solve explicitly does (most probably never did, simply because lapack
> > does not). So I am wondering if there is some convention for this, or
> > what convention we should implement.
> 
> What does an empty square matrix/array look like?
> 
> np.linalg.solve   can have empty rhs, but shape of empty lhs, `a`, is ?
> 
> If I do a QR(arr)  with arr.shape=(0, 5), what is R supposed to be ?
> 

QR may be more difficult since R may itself could not be empty, begging
the question if you want to error out or fill it sensibly.
Cholesky would require (0, 0) for example and for eigenvalues it would
somewhat make sense too, the (0, 0) matrix has 0 eigenvalues.
I did not go through them all, but I would like to figure out whether we
should aim to generally allow it, or maybe just allow it for some
special ones.

- Sebastian

> 
> I just wrote some loops over linalg.qr, but I always initialized explicitly.
> 
> I didn't manage to figure out how empty arrays would be useful.
> 
> If an empty square matrix can only only be of shape (0, 0), then it's
> no use (in my applications).
> 
> 
> Josef
> 
> 
> >
> > Regards,
> >
> > Sebastian
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Linear algebra functions on empty arrays

2014-09-15 Thread josef.pktd
On Mon, Sep 15, 2014 at 5:48 AM, Sebastian Berg
 wrote:
> Hey all,
>
> for https://github.com/numpy/numpy/pull/3861/files I would like to allow
> 0-sized dimensions for generalized ufuncs, meaning that the gufunc has
> to be able to handle this, but also that it *can* handle it at all.
> However lapack does not support this, so it needs some explicit fixing.
> Also some of the linalg functions currently explicitly allow and others
> explicitly disallow empty arrays.
>
> For example the QR and eigvals does not allow it, but on the other hand
> solve explicitly does (most probably never did, simply because lapack
> does not). So I am wondering if there is some convention for this, or
> what convention we should implement.

What does an empty square matrix/array look like?

np.linalg.solve   can have empty rhs, but shape of empty lhs, `a`, is ?

If I do a QR(arr)  with arr.shape=(0, 5), what is R supposed to be ?


I just wrote some loops over linalg.qr, but I always initialized explicitly.

I didn't manage to figure out how empty arrays would be useful.

If an empty square matrix can only only be of shape (0, 0), then it's
no use (in my applications).


Josef


>
> Regards,
>
> Sebastian
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Tracking and inspecting numpy objects

2014-09-15 Thread Mads Ipsen
Thanks to everybody for taking time to answer!

Best regards,

Mads

On 15/09/14 12:11, Sebastian Berg wrote:
> On Mo, 2014-09-15 at 12:05 +0200, Eelco Hoogendoorn wrote:
>>
>>
>>
>> On Mon, Sep 15, 2014 at 11:55 AM, Sebastian Berg
>>  wrote:
>>  On Mo, 2014-09-15 at 10:16 +0200, Mads Ipsen wrote:
>>  > Hi,
>>  >
>>  > I am trying to inspect the reference count of numpy arrays
>>  generated by
>>  > my application.
>>  >
>>  > Initially, I thought I could inspect the tracked objects
>>  using
>>  > gc.get_objects(), but, with respect to numpy objects, the
>>  returned
>>  > container is empty. For example:
>>  >
>>  > import numpy
>>  > import gc
>>  >
>>  > data = numpy.ones(1024).reshape((32,32))
>>  >
>>  > objs = [o for o in gc.get_objects() if isinstance(o,
>>  numpy.ndarray)]
>>  >
>>  > print objs# Prints empty list
>>  > print gc.is_tracked(data) # Print False
>>  >
>>  > Why is this? Also, is there some other technique I can use
>>  to inspect
>>  > all numpy generated objects?
>>  >
>>
>>  Two reasons. First of all, unless your array is an object
>>  arrays (or a
>>  structured one with objects in it), there are no objects to
>>  track. The
>>  array is a single python object without any referenced objects
>>  (except
>>  possibly its `arr.base`).
>>
>>  Second of all -- and this is an issue -- numpy doesn't
>>  actually
>>  implement the traverse slot, so it won't even work for object
>>  arrays
>>  (numpy object arrays do not support circular garbage
>>  collection at this
>>  time, please feel free to implement it ;)).
>>
>>  - Sebastian
>>
>>
>>
>>
>>
>>
>> Does this answer why the ndarray object itself isn't tracked though? I
>> must say I find this puzzling; the only thing I can think of is that
>> the python compiler notices that data isn't used anymore after its
>> creation, and deletes it right after its creation as an optimization,
>> but that conflicts with my own experience of the GC.
>>
>>
>
> Not sure if it does, but my quick try and error says:
> In [15]: class T(tuple):
> : pass
> :
>
> In [16]: t = T()
>
> In [17]: objs = [o for o in gc.get_objects() if isinstance(o, T)]
>
> In [18]: objs
> Out[18]: [()]
>
> In [19]: a = 123.
>
> In [20]: objs = [o for o in gc.get_objects() if isinstance(o, float)]
>
> In [21]: objs
> Out[21]: []
>
> So I guess nothing is tracked, unless it contains things, and numpy
> arrays don't say they can contain things (i.e. no traverse).
>
> - Sebastian
>
>
>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

-- 
+-+
| Mads Ipsen  |
+--+--+
| Gåsebæksvej 7, 4. tv | phone:  +45-29716388 |
| DK-2500 Valby| email:  mads.ip...@gmail.com |
| Denmark  | map  :   www.tinyurl.com/ns52fpa |
+--+--+
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Tracking and inspecting numpy objects

2014-09-15 Thread Sebastian Berg
On Mo, 2014-09-15 at 12:05 +0200, Eelco Hoogendoorn wrote:
> 
> 
> 
> On Mon, Sep 15, 2014 at 11:55 AM, Sebastian Berg
>  wrote:
> On Mo, 2014-09-15 at 10:16 +0200, Mads Ipsen wrote:
> > Hi,
> >
> > I am trying to inspect the reference count of numpy arrays
> generated by
> > my application.
> >
> > Initially, I thought I could inspect the tracked objects
> using
> > gc.get_objects(), but, with respect to numpy objects, the
> returned
> > container is empty. For example:
> >
> > import numpy
> > import gc
> >
> > data = numpy.ones(1024).reshape((32,32))
> >
> > objs = [o for o in gc.get_objects() if isinstance(o,
> numpy.ndarray)]
> >
> > print objs# Prints empty list
> > print gc.is_tracked(data) # Print False
> >
> > Why is this? Also, is there some other technique I can use
> to inspect
> > all numpy generated objects?
> >
> 
> Two reasons. First of all, unless your array is an object
> arrays (or a
> structured one with objects in it), there are no objects to
> track. The
> array is a single python object without any referenced objects
> (except
> possibly its `arr.base`).
> 
> Second of all -- and this is an issue -- numpy doesn't
> actually
> implement the traverse slot, so it won't even work for object
> arrays
> (numpy object arrays do not support circular garbage
> collection at this
> time, please feel free to implement it ;)).
> 
> - Sebastian
> 
> 
> 
> 
> 
> 
> Does this answer why the ndarray object itself isn't tracked though? I
> must say I find this puzzling; the only thing I can think of is that
> the python compiler notices that data isn't used anymore after its
> creation, and deletes it right after its creation as an optimization,
> but that conflicts with my own experience of the GC.
> 
> 

Not sure if it does, but my quick try and error says:
In [15]: class T(tuple):
   : pass
   : 

In [16]: t = T()

In [17]: objs = [o for o in gc.get_objects() if isinstance(o, T)]

In [18]: objs
Out[18]: [()]

In [19]: a = 123.

In [20]: objs = [o for o in gc.get_objects() if isinstance(o, float)]

In [21]: objs
Out[21]: []

So I guess nothing is tracked, unless it contains things, and numpy
arrays don't say they can contain things (i.e. no traverse).

- Sebastian



> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Tracking and inspecting numpy objects

2014-09-15 Thread Robert Kern
On Mon, Sep 15, 2014 at 11:05 AM, Eelco Hoogendoorn
 wrote:

> Does this answer why the ndarray object itself isn't tracked though? I must
> say I find this puzzling; the only thing I can think of is that the python
> compiler notices that data isn't used anymore after its creation, and
> deletes it right after its creation as an optimization, but that conflicts
> with my own experience of the GC.

The "gc" that "gc.is_tracked()" refers to is just the cyclical garbage
detector, not the usual reference counting memory management that all
Python objects participate in. If your object cannot participate in
reference cycles (like most ndarrays), it doesn't need to be tracked.

-- 
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Tracking and inspecting numpy objects

2014-09-15 Thread Eelco Hoogendoorn
On Mon, Sep 15, 2014 at 11:55 AM, Sebastian Berg  wrote:

> On Mo, 2014-09-15 at 10:16 +0200, Mads Ipsen wrote:
> > Hi,
> >
> > I am trying to inspect the reference count of numpy arrays generated by
> > my application.
> >
> > Initially, I thought I could inspect the tracked objects using
> > gc.get_objects(), but, with respect to numpy objects, the returned
> > container is empty. For example:
> >
> > import numpy
> > import gc
> >
> > data = numpy.ones(1024).reshape((32,32))
> >
> > objs = [o for o in gc.get_objects() if isinstance(o, numpy.ndarray)]
> >
> > print objs# Prints empty list
> > print gc.is_tracked(data) # Print False
> >
> > Why is this? Also, is there some other technique I can use to inspect
> > all numpy generated objects?
> >
>
> Two reasons. First of all, unless your array is an object arrays (or a
> structured one with objects in it), there are no objects to track. The
> array is a single python object without any referenced objects (except
> possibly its `arr.base`).
>
> Second of all -- and this is an issue -- numpy doesn't actually
> implement the traverse slot, so it won't even work for object arrays
> (numpy object arrays do not support circular garbage collection at this
> time, please feel free to implement it ;)).
>
> - Sebastian
>
>

Does this answer why the ndarray object itself isn't tracked though? I must
say I find this puzzling; the only thing I can think of is that the python
compiler notices that data isn't used anymore after its creation, and
deletes it right after its creation as an optimization, but that conflicts
with my own experience of the GC.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Tracking and inspecting numpy objects

2014-09-15 Thread Sebastian Berg
On Mo, 2014-09-15 at 10:16 +0200, Mads Ipsen wrote:
> Hi,
> 
> I am trying to inspect the reference count of numpy arrays generated by 
> my application.
> 
> Initially, I thought I could inspect the tracked objects using 
> gc.get_objects(), but, with respect to numpy objects, the returned 
> container is empty. For example:
> 
> import numpy
> import gc
> 
> data = numpy.ones(1024).reshape((32,32))
> 
> objs = [o for o in gc.get_objects() if isinstance(o, numpy.ndarray)]
> 
> print objs# Prints empty list
> print gc.is_tracked(data) # Print False
> 
> Why is this? Also, is there some other technique I can use to inspect
> all numpy generated objects?
> 

Two reasons. First of all, unless your array is an object arrays (or a
structured one with objects in it), there are no objects to track. The
array is a single python object without any referenced objects (except
possibly its `arr.base`).

Second of all -- and this is an issue -- numpy doesn't actually
implement the traverse slot, so it won't even work for object arrays
(numpy object arrays do not support circular garbage collection at this
time, please feel free to implement it ;)).

- Sebastian


> Thanks in advance.
> 
> Best regards,
> 
> Mads
> 



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


[Numpy-discussion] Linear algebra functions on empty arrays

2014-09-15 Thread Sebastian Berg
Hey all,

for https://github.com/numpy/numpy/pull/3861/files I would like to allow
0-sized dimensions for generalized ufuncs, meaning that the gufunc has
to be able to handle this, but also that it *can* handle it at all.
However lapack does not support this, so it needs some explicit fixing.
Also some of the linalg functions currently explicitly allow and others
explicitly disallow empty arrays.

For example the QR and eigvals does not allow it, but on the other hand
solve explicitly does (most probably never did, simply because lapack
does not). So I am wondering if there is some convention for this, or
what convention we should implement.

Regards,

Sebastian


signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


[Numpy-discussion] Tracking and inspecting numpy objects

2014-09-15 Thread Mads Ipsen
Hi,

I am trying to inspect the reference count of numpy arrays generated by 
my application.

Initially, I thought I could inspect the tracked objects using 
gc.get_objects(), but, with respect to numpy objects, the returned 
container is empty. For example:

import numpy
import gc

data = numpy.ones(1024).reshape((32,32))

objs = [o for o in gc.get_objects() if isinstance(o, numpy.ndarray)]

print objs# Prints empty list
print gc.is_tracked(data) # Print False

Why is this? Also, is there some other technique I can use to inspect
all numpy generated objects?

Thanks in advance.

Best regards,

Mads

-- 
+-+
| Mads Ipsen  |
+--+--+
| Gåsebæksvej 7, 4. tv | phone:  +45-29716388 |
| DK-2500 Valby| email:  mads.ip...@gmail.com |
| Denmark  | map  :   www.tinyurl.com/ns52fpa |
+--+--+
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion