Re: [Numpy-discussion] isnan() equivalent for np.NaT?

[Gerrit Holl]

On 18 July 2016 at 22:20, Scott Sanderson  wrote:
> I'm working on upgrading Zipline (github.com/quantopian/zipline) to the
> latest numpy, and I'm getting a FutureWarnings about the upcoming change in
> the behavior of comparisons on np.NaT.  I'd like to be able to do checks for
> NaT in a way that's forwards-compatible, but I couldn't find a function
> analogous to `np.isnan` for NaT.  Am I missing something that already
> exists?  If not, is there interest in such a function? I'd like to be able
> to pre-emptively fix the warnings in Zipline so that we're ready when the
> change actually happens, but it doesn't seem like the necessary tools are
> available right now.

Hi Scott,

see https://github.com/numpy/numpy/issues/5610

Gerrit.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Should dtypes have an ndim attribute?

[Gerrit Holl]

Hi,

I have made a modest proposal in issue #6752
.  Basically, the proposal
is to add an `ndim` attribute to dtypes.  Currently, arrays have a
shape and an ndim attribute, where ndim equals len(shape).  dtype
objects have a shape attribute, but no corresponding ndim.

An ndim attribute would help in immediately determining whether a
field in a structured dtype is multidimensional or not.

Thoughts?

Gerrit.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Indexing structured masked arrays with multidimensional fields; what with fill_value?

[Gerrit Holl]

Hello,

usually, a masked array's .fill_value attribute has ndim=0 and the
same dtype as the data attribute:

In [27]: ar = array((0, [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]], 0.0),
dtype="int, (2,3)float, float")

In [28]: arm = ma.masked_array(ar)

In [29]: arm.fill_value.ndim
Out[29]: 0

In [31]: arm.fill_value.dtype
Out[31]: dtype([('f0', '

[Numpy-discussion] subdivide array

[Gerrit Holl]

On 30 November 2010 17:58, Pierre GM pgmdevl...@gmail.com wrote:
 On Nov 30, 2010, at 5:40 PM, John wrote:
 I have an array of data for a global grid at 1 degree resolution. It's
 filled with 1s and 0s, and it is just a land sea mask (not only, but
 as an example). I want to be able to regrid the data to higher or
 lower resolutions (i.e. 0.5 or 2 degrees). But if I try to use any
 standard interp functions, such as mpl_toolkits.basemap.interp it
 fails -- I assume due to the data being binary.

 I guess there may be a fairly easy routine to do this?? Does someone
 have an example?

 Just a random idea: have you tried to convert your input data to float? 
 Hopefully you could get some values between 0 and 1 for your interpolated 
 values, that you'll have to transform back to integers following a scheme of 
 your choosing...

I would argue that some float between 0 and 1 is an excellent
representation when regridding a binary land-sea-mask onto a higher
resolution. After all, this information is not there. Why should a
land-sea mask be binary anyway? As if a grid-cell can only be fully
ocean or fully land...

BTW, I just realised Pythons convention that negative indices count
from the end of the array is perfect when using a 180x360
land-sea-mask, as lon[-30] and lon[330] mean and should mean the same
:)

Gerrit.

--
Gerrit Holl
PhD student at Department of Space Science, Luleå University of
Technology, Kiruna, Sweden
http://www.sat.ltu.se/members/gerrit/
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] merge_arrays is very slow; alternatives?

[Gerrit Holl]

Hi,

upon profiling my code, I found that
numpy.lib.recfunctions.merge_arrays is extremely slow; it does some
7000 rows/second. This is not acceptable for me.

I have two large record arrays, or arrays with a complicated dtype.
All I want to do is to merge them into one. I don't think that should
have to be a very slow operation, I don't need to copy anything, I
just want to view the two record arrays as one.

How can I do this in a faster way?

In [45]: cProfile.runctx(numpy.lib.recfunctions.merge_arrays([metarows,
targetrows2], flatten=True), globals(), locals())
        225381902 function calls (150254635 primitive calls) in
166.620 CPU seconds

  Ordered by: standard name

  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       1    0.031    0.031  166.620  166.620 string:1(module)
    68/1    0.000    0.000    0.000    0.000 _internal.py:82(_array_descr)
       2    0.000    0.000    0.000    0.000 numeric.py:286(asanyarray)
       2    0.000    0.000    0.000    0.000 recfunctions.py:135(flatten_descr)
       1    0.000    0.000    0.001    0.001 recfunctions.py:161(zip_descr)
149165600/74038400  117.195    0.000  139.701    0.000
recfunctions.py:235(_izip_fields_flat)
 1088801   12.146    0.000  151.847    0.000 recfunctions.py:263(izip_records)
       3    0.000    0.000    0.000    0.000 recfunctions.py:277(sentinel)
       1    4.599    4.599  166.589  166.589 recfunctions.py:328(merge_arrays)
       3    0.000    0.000    0.000    0.000 recfunctions.py:406(genexpr)
 75127201   22.506    0.000   22.506    0.000 {isinstance}
      69    0.000    0.000    0.000    0.000 {len}
       1    0.000    0.000    0.000    0.000 {map}
       1    0.000    0.000    0.000    0.000 {max}
       2    0.000    0.000    0.000    0.000 {method '__array__' of
'numpy.ndarray' objects}
     136    0.000    0.000    0.000    0.000 {method 'append' of
'list' objects}
       1    0.000    0.000    0.000    0.000 {method 'disable' of
'_lsprof.Profiler' objects}
       2    0.000    0.000    0.000    0.000 {method 'extend' of
'list' objects}
       2    0.000    0.000    0.000    0.000 {method 'pop' of 'list' objects}
       2    0.000    0.000    0.000    0.000 {method 'ravel' of
'numpy.ndarray' objects}
       2    0.000    0.000    0.000    0.000 {numpy.core.multiarray.array}
       1   10.142   10.142   10.142   10.142 {numpy.core.multiarray.fromiter}


Gerrit.

--
Gerrit Holl
PhD student at Department of Space Science, Luleå University of
Technology, Kiruna, Sweden
http://www.sat.ltu.se/members/gerrit/
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] merge_arrays is very slow; alternatives?

[Gerrit Holl]

On 26 November 2010 20:16, Gerrit Holl gerrit.h...@gmail.com wrote:
 Hi,

 upon profiling my code, I found that
 numpy.lib.recfunctions.merge_arrays is extremely slow; it does some
 7000 rows/second. This is not acceptable for me.
...
 How can I do this in a faster way?

Replying to my own code here. Either I have written a much faster
implementation of this, or I am missing something. I consider it
unlikely that I write a much faster implementation of an established
numpy function with little effort, so I suspect I am missing something
here.

I wrote this implementation of the flattened version of merge_arrays:

def merge_arrays(arr1, arr2):
t1 = arr1.dtype
t2 = arr2.dtype
newdtype = numpy.dtype(t1.descr + t2.descr)
newarray = numpy.empty(shape=arr1.shape, dtype=newdtype)
for field in t1.names:
newarray[field] = arr1[field]
for field in t2.names:
newarray[field] = arr2[field]
return newarray

and benchmarks show it's almost 100 times faster for a medium-sized array:

In [211]: %timeit merged1 =
numpy.lib.recfunctions.merge_arrays([metarows[:1],
targetrows2[:1]], flatten=True)
1 loops, best of 3: 1.01 s per loop

In [212]: %timeit merged2 =
pyatmlab.tools.merge_arrays(metarows[:1], targetrows2[:1])
100 loops, best of 3: 10.8 ms per loop

In [214]: (merged1.view(dtype=uint64).reshape(-1, 100) ==
merged2.view(dtype=uint64).reshape(-1, 100)).all()
Out[214]: True

# and still 4 times faster for a small array:

In [215]: %timeit merged1 =
numpy.lib.recfunctions.merge_arrays([metarows[:10], targetrows2[:10]],
flatten=True)
1000 loops, best of 3: 1.31 ms per loop

In [216]: %timeit merged2 = pyatmlab.tools.merge_arrays(metarows[:10],
targetrows2[:10])
1000 loops, best of 3: 344 us per loop

# and 15 times faster for a large array (1.5 million elements):

In [218]: %timeit merged1 =
numpy.lib.recfunctions.merge_arrays([metarows, targetrows2],
flatten=True)
1 loops, best of 3: 110 s per loop

In [217]: %timeit merged2 = pyatmlab.tools.merge_arrays(metarows, targetrows2)
1 loops, best of 3: 7.26 s per loop

I wonder, am I missing something or have I really written a
significant improvement in less than 10 LOC? Should I file a patch for
this?

Gerrit.

-- 
Exploring space at http://gerrit-explores.blogspot.com/
Personal homepage at http://www.topjaklont.org/
Asperger Syndroom: http://www.topjaklont.org/nl/asperger.html
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy.test() errors

[Gerrit Holl]

2010/11/23 Stéfan van der Walt ste...@sun.ac.za:
 On Tue, Nov 23, 2010 at 9:28 AM, Nils Wagner
 nwag...@iam.uni-stuttgart.de wrote:
 /data/home/nwagner/local/lib/python2.5/site-packages/numpy/lib/npyio.py,
 line 66, in seek_gzip_factory
     g.name = f.name
 AttributeError: GzipFile instance has no attribute 'name'

 This one is mine--the change was made to avoid a deprecationwarning.
 Which version of Python are you using?

I hope 2.5, as his site-packages directory is in lib/python2.5 :)

Gerrit.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Advise for numerical programming content (New python user)

[Gerrit Holl]

Hi,

On 19 November 2010 03:48, Sachin Kumar Sharma ssharm...@slb.com wrote:
 Does graphic output like maps, histogram, crossplot, tornado charts is good
 enough with basic installation or needs some additional packages?

You might want to ask this question at the scipy mailig-list.

For maps, you need basemap or PyNCL.

I am presently using basemap. In the future, I will look at PyNCL,
Python bindings to NCL. I think there are also Python bindings to GMT.

Interestingly, both basemap and PyNCL are written by the same person
(Jeff Whitaker). Probably those packages complement each other rather
than doing the same thing. Others might correct me here.

Gerrit.

-- 
Exploring space at http://gerrit-explores.blogspot.com/
Personal homepage at http://www.topjaklont.org/
Asperger Syndroom: http://www.topjaklont.org/nl/asperger.html
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Printing formatted numerical values

[Gerrit Holl]

On 15 November 2010 15:32,  pv+nu...@math.duke.edu wrote:
 Hi, what is the best way to print (to a file or to stdout) formatted
 numerical values? Analogously to C's printf(%d %g,x,y) etc?

Use the .tofile() method:

numpy.random.random(5).tofile(sys.stdout, ' ', '%s')
0.230466435867 0.609443784908 0.353855676828 0.552641723317 0.186418931597

(works only on real files, afaik, not on StringIO or similar)

cheers,
Gerrit.

-- 
Exploring space at http://gerrit-explores.blogspot.com/
Personal homepage at http://www.topjaklont.org/
Asperger Syndroom: http://www.topjaklont.org/nl/asperger.html
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] List with numpy semantics

[Gerrit Holl]

On 31 October 2010 17:10, Nikolaus Rath nikol...@rath.org wrote:
 Hello,

 I have a couple of numpy arrays which belong together. Unfortunately
 they have different dimensions, so I can't bundle them into a higher
 dimensional array.

 My solution was to put them into a Python list instead. But
 unfortunately this makes it impossible to use any ufuncs.

 Has someone else encountered a similar problem and found a nice
 solution? Something like a numpy list maybe?

You could try a record array with a clever dtype, maybe?

Gerrit.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion