Re: [Numpy-discussion] 2D array indexing

[Gökhan Sever]

Thanks Julian. Mistakenly, I have (a[:1:] + a[:1,:])/2 type of construct
somewhere in my code. It works fine, however I wasn't sure if this is
something leading to a wrong calculation. Now your explanation makes it
clearer.


On Fri, Feb 28, 2014 at 6:48 PM, Julian Taylor <
jtaylor.deb...@googlemail.com> wrote:

> On 01.03.2014 00:32, Gökhan Sever wrote:
> >
> > Hello,
> >
> > Given this simple 2D array:
> >
> > In [1]: np.arange(9).reshape((3,3))
> > Out[1]:
> > array([[0, 1, 2],
> >[3, 4, 5],
> >[6, 7, 8]])
> >
> > In [2]: a = np.arange(9).reshape((3,3))
> >
> > In [3]: a[:1:]
> > Out[3]: array([[0, 1, 2]])
> >
> > In [4]: a[:1,:]
> > Out[4]: array([[0, 1, 2]])
> >
> > Could you tell me why the last two indexing (note the comma!) results in
> > the same array? Thanks.
> >
>
>
> if you specify less indices than dimensions the latter dimensions are
> implicitly all selected.
> so these are identical for three dimensional arrays:
> d = np.ones((3,3,3))
> d[1]
> d[1,:]
> d[1,:,:]
> d[1,...] (... or Ellipsis selects all remaining dimensions)
>
> this only applies to latter dimensions in the shape, if you want to
> select all earlier dimensions they have to be explicitly selected:
> d[:,1] == d[:,1,:]
> d[..., 1] = d[:,:,1]
>
>
> as for :1: vs 1:, its standard python rules: start:stop:step, with all
> three having defaults of 0:len(sequence):1
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] 2D array indexing

[Gökhan Sever]

Hello,

Given this simple 2D array:

In [1]: np.arange(9).reshape((3,3))
Out[1]:
array([[0, 1, 2],
   [3, 4, 5],
   [6, 7, 8]])

In [2]: a = np.arange(9).reshape((3,3))

In [3]: a[:1:]
Out[3]: array([[0, 1, 2]])

In [4]: a[:1,:]
Out[4]: array([[0, 1, 2]])

Could you tell me why the last two indexing (note the comma!) results in
the same array? Thanks.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Modern Fortran vs NumPy syntax

[Gökhan Sever]

Hi Ondřej,

Any ideas that your manual syntax mapping would evolve to an automatic
translation tool like i2py [http://code.google.com/p/i2py/]

Thanks.

On Thu, Feb 7, 2013 at 12:22 PM, Ondřej Čertík wrote:

> Hi,
>
> I have recently setup a page about modern Fortran:
>
> http://fortran90.org/
>
> and in particular, it has a long section with side by side syntax
> examples of Python/NumPy vs Fortran:
>
> http://fortran90.org/src/rosetta.html
>
> I would be very interested if some NumPy gurus would provide me
> feedback. I personally knew
> NumPy long before I learned Fortran, and I was amazed that the modern
> Fortran pretty much
> allows 1:1 syntax with NumPy, including most of all the fancy indexing etc.
>
> Is there some NumPy feature that is not covered there? I would like it
> to be a nice resource
> for people who know NumPy to feel like at home with Fortran, and vice
> versa. I personally
> use both every day (Fortran a bit more than NumPy).
>
> Or of you have any other comments or tips for the site, please let me
> know. Eventually I'd like
> to also put there C++ way of doing the same things, but at the moment
> I want to get Fortran
> and Python/NumPy done first.
>
> Ondrej
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] float32 to float64 casting

[Gökhan Sever]

On Sat, Nov 17, 2012 at 9:47 AM, Nathaniel Smith  wrote:

> On Fri, Nov 16, 2012 at 9:53 PM, Gökhan Sever 
> wrote:
> > Thanks for the explanations.
> >
> > For either case, I was expecting to get float32 as a resulting data type.
> > Since, float32 is large enough to contain the result. I am wondering if
> > changing casting rule this way, requires a lot of modification in the
> NumPy
> > code. Maybe as an alternative to the current casting mechanism?
> >
> > I like the way that NumPy can convert to float64. As if these data-types
> are
> > continuation of each other. But just the conversation might happen too
> early
> > --at least in my opinion, as demonstrated in my example.
> >
> > For instance comparing this example to IDL surprises me:
> >
> > I16 np.float32()*5e38
> > O16 2.77749998e+42
> >
> > I17 (np.float32()*5e38).dtype
> > O17 dtype('float64')
>
> In this case, what's going on is that 5e38 is a Python float object,
> and Python float objects have double-precision, i.e., they're
> equivalent to np.float64's. So you're multiplying a float32 and a
> float64. I think most people will agree that in this situation it's
> better to use float64 for the output?
>
> -n
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

OK, I see your point. Python numeric data objects and NumPy data objects
mixed operations require more attention.

The following causes float32 overflow --rather than casting to float64 as
in the case for Python float multiplication, and behaves like in IDL.

I3 (np.float32()*np.float32(5e38))
O3 inf

However, these two still surprises me:

I5 (np.float32()*1).dtype
O5 dtype('float64')

I6 (np.float32()*np.int32(1)).dtype
O6 dtype('float64')
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] float32 to float64 casting

[Gökhan Sever]

Thanks for the explanations.

For either case, I was expecting to get float32 as a resulting data type.
Since, float32 is large enough to contain the result. I am wondering if
changing casting rule this way, requires a lot of modification in the NumPy
code. Maybe as an alternative to the current casting mechanism?

I like the way that NumPy can convert to float64. As if these data-types
are continuation of each other. But just the conversation might happen too
early --at least in my opinion, as demonstrated in my example.

For instance comparing this example to IDL surprises me:

I16 np.float32()*5e38
O16 2.77749998e+42

I17 (np.float32()*5e38).dtype
O17 dtype('float64')

IDL> help, 5e38*float()
FLOAT =   Inf

In IDL, the expression doesn't get converted to DOUBLE. Perhaps, its a
design decision.


On Thu, Nov 15, 2012 at 8:24 PM, Gökhan Sever  wrote:

> Hello,
>
> Could someone briefly explain why are these two operations are casting my
> float32 arrays to float64?
>
> I1 (np.arange(5, dtype='float32')).dtype
> O1 dtype('float32')
>
> I2 (10*np.arange(5, dtype='float32')).dtype
> O2 dtype('float64')
>
>
>
> I3 (np.arange(5, dtype='float32')[0]).dtype
> O3 dtype('float32')
>
> I4 (1*np.arange(5, dtype='float32')[0]).dtype
> O4 dtype('float64')
>
>
>
> Thanks.
>
>
>
>
> --
> Gökhan
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] float32 to float64 casting

[Gökhan Sever]

Hello,

Could someone briefly explain why are these two operations are casting my
float32 arrays to float64?

I1 (np.arange(5, dtype='float32')).dtype
O1 dtype('float32')

I2 (10*np.arange(5, dtype='float32')).dtype
O2 dtype('float64')



I3 (np.arange(5, dtype='float32')[0]).dtype
O3 dtype('float32')

I4 (1*np.arange(5, dtype='float32')[0]).dtype
O4 dtype('float64')



Thanks.




-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question on numpy.ma.masked_values

[Gökhan Sever]

Yes, that's the behaviour that I expect setting the 'shrink' keyword to 'False'

> Now, just to be clear, you'd want
> 'np.ma.masked_values(...,shrink=False) to create a maked array w/ a
> full boolean mask by default, right ?
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question on numpy.ma.masked_values

[Gökhan Sever]

Submitted the ticket at http://projects.scipy.org/numpy/ticket/2082



On Thu, Mar 15, 2012 at 1:24 PM, Gökhan Sever  wrote:

>
>
> On Thu, Mar 15, 2012 at 1:12 PM, Pierre GM  wrote:
>
>> Ciao Gökhan,
>> AFAIR, shrink is used only to force a collapse of a mask full of False,
>> not to force the creation of such a mask.
>> Now, it should work as you expected, meaning that it needs to be fixed.
>> Could you open a ticket? And put me in copy, just in case.
>> Anyhow:
>> Your trick is a tad dangerous, as it erases the previous mask. I'd prefer
>> to create x w/ a full mask, then use masked_values w/ shrink=False... Now,
>> if you're sure there's x= no masked values, go for it.
>> Cheers
>>
>> This condition checking should make it stronger:
>
> I7 x = np.array([1, 1.1, 2, 1.1, 3])
>
> I8 y = np.ma.masked_values(x, 1.5)
>
> I9 if y.mask == False:
> y.mask = np.zeros(len(x), dtype=np.bool)*True
>...:
>
> I10 y.mask
> O10 array([False, False, False, False, False], dtype=bool)
>
> I11 y
> O11
> masked_array(data = [1.0 1.1 2.0 1.1 3.0],
>  mask = [False False False False False],
>fill_value = 1.5)
>
> How do you create "x w/ a full mask"?
>
> --
> Gökhan
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question on numpy.ma.masked_values

[Gökhan Sever]

On Thu, Mar 15, 2012 at 1:12 PM, Pierre GM  wrote:

> Ciao Gökhan,
> AFAIR, shrink is used only to force a collapse of a mask full of False,
> not to force the creation of such a mask.
> Now, it should work as you expected, meaning that it needs to be fixed.
> Could you open a ticket? And put me in copy, just in case.
> Anyhow:
> Your trick is a tad dangerous, as it erases the previous mask. I'd prefer
> to create x w/ a full mask, then use masked_values w/ shrink=False... Now,
> if you're sure there's x= no masked values, go for it.
> Cheers
>
This condition checking should make it stronger:

I7 x = np.array([1, 1.1, 2, 1.1, 3])

I8 y = np.ma.masked_values(x, 1.5)

I9 if y.mask == False:
y.mask = np.zeros(len(x), dtype=np.bool)*True
   ...:

I10 y.mask
O10 array([False, False, False, False, False], dtype=bool)

I11 y
O11
masked_array(data = [1.0 1.1 2.0 1.1 3.0],
 mask = [False False False False False],
   fill_value = 1.5)

How do you create "x w/ a full mask"?

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question on numpy.ma.masked_values

[Gökhan Sever]

On Thu, Mar 15, 2012 at 12:56 PM, Gökhan Sever wrote:

If not so, how can I return a set of False values if my masking condition
> is not met?
>

Self-answer: I can force the mask to be filled with False's, however unsure
if this is a safe operation.

I50 x = np.array([1, 1.1, 2, 1.1, 3])

I51 y = np.ma.masked_values(x, 1.5, shrink=0)

I52 y
O52
masked_array(data = [1.0 1.1 2.0 1.1 3.0],
 mask = False,
   fill_value = 1.5)


I53 y.mask = np.zeros(len(x), dtype=np.bool)*True

I54 y
O54
masked_array(data = [1.0 1.1 2.0 1.1 3.0],
 mask = [False False False False False],
   fill_value = 1.5)
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Question on numpy.ma.masked_values

[Gökhan Sever]

Hello,

>From the masked_values() documentation ->
http://docs.scipy.org/doc/numpy/reference/generated/numpy.ma.masked_values.html

I10 np.ma.masked_values(x, 1.5)
O10
masked_array(data = [ 1.   1.1  2.   1.1  3. ],
 mask = False,
   fill_value = 1.5)


I12 np.ma.masked_values(x, 1.5, shrink=False)
O12
masked_array(data = [ 1.   1.1  2.   1.1  3. ],
 mask = False,
   fill_value = 1.5)

Shouldn't setting the 'shrink' to False return an array of False values for
the mask field?
If not so, how can I return a set of False values if my masking condition
is not met?

Using:
I16 np.__version__
O16 '2.0.0.dev-7e202a2'

Thanks.


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Re place array values

[Gökhan Sever]

Hi,

You can try masked_array module:

x = np.array([[0,1,2],[3,4,5],[6,7,8]])

I3 np.ma.masked_where(x<1, x)
O3
masked_array(data =
 [[-- 1 2]
 [3 4 5]
 [6 7 8]],
 mask =
 [[ True False False]
 [False False False]
 [False False False]],
   fill_value = 99)

There might be a smarter solution than this, since ma tends to get slower
when you deal with big data arrays. But you retain the 2D information
instead of getting a flattened np.array.


On Wed, Mar 14, 2012 at 5:35 PM, jonasr  wrote:

>
> Hello,
>
> my problem is that i want to remove some small numbers of an 2d array,
> for example if i want to sort out all numbers smaller then 1 of an array i
> get
>
> x=[[0,1,2],[3,4,5][6,7,8]]
>
> c=x>=1
>
> In [213]: c
> Out[213]:
> array([[False,  True,  True],
>   [ True,  True,  True],
>   [ True,  True,  True]], dtype=bool)
>
> In [214]: x[c]
> Out[214]: array([1, 2, 3, 4, 5, 6, 7, 8])
>
> the problem ist that i now have a 1d array, is there any possibility to
> keep the 2d structure ?
>
> greets jonas
> --
> View this message in context:
> http://old.nabble.com/Replace-array-values-tp33506446p33506446.html
> Sent from the Numpy-discussion mailing list archive at Nabble.com.
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] wanted: decent matplotlib alternative

[Gökhan Sever]

On Thu, Oct 13, 2011 at 4:15 PM, Benjamin Root  wrote:

> Myself and other developers would greatly appreciate help from the
> community to point out which examples are too confusing or out of date. We
>

It would be nice to have a social interface for the mpl gallery like the one
similar to the R-gallery [
http://www.r-bloggers.com/the-r-graph-gallery-goes-social/]


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] wanted: decent matplotlib alternative

[Gökhan Sever]

On Thu, Oct 13, 2011 at 4:03 PM, Gökhan Sever  wrote:

>
> I think, IPython is great for interaction with the OO interface of the
> matlab.  Just starting simple with:
>
> fig=plt.figure()
> ax=plt.gca()
> and keep tabbing ax., fig. or any object you create on the canvas
> .tab to get its methods and attributes. Another approach is start with the
> pylab interface and query detailed help/code with ?? in IPython (e.g.
> plt.xlabel??)
>

Sorry s/matlab/matplotlib. I am not sure if matlab has IPython like
interface to introspect objects. Definitely IDL doesn't.


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] wanted: decent matplotlib alternative

[Gökhan Sever]

On Thu, Oct 13, 2011 at 3:21 PM, Zachary Pincus wrote:

> I keep meaning to use matplotlib as well, but every time I try I also get
> really turned off by the matlabish interface in the examples. I get that
> it's a selling point for matlab refugees, but I find it counterintuitive in
> the same way Christoph seems to.
>
> I'm glad to hear the OO interface isn't as clunky as it looks on some of
> the doc pages, though. This is good news. Can anyone point out any good
> tutorials/docs on using matplotlib idiomatically via its OO interface?
>
> Zach


I think, IPython is great for interaction with the OO interface of the
matlab.  Just starting simple with:

fig=plt.figure()
ax=plt.gca()
and keep tabbing ax., fig. or any object you create on the canvas
.tab to get its methods and attributes. Another approach is start with the
pylab interface and query detailed help/code with ?? in IPython (e.g.
plt.xlabel??)


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray with double comparison

[Gökhan Sever]

On Thu, Oct 13, 2011 at 10:13 AM, Chao YUE  wrote:

> Dear all,
>
> sorry for this stupid question but I cannot find it in numpy tutorial or
> google.
> suppose I have a=np.arange(11).
>
> In [32]: a < 8
> Out[32]:
> array([ True,  True,  True,  True,  True,  True,  True,  True, False,
>False, False], dtype=bool)
>
> In [34]: a > 4
> Out[34]:
> array([False, False, False, False, False,  True,  True,  True,  True,
> True,  True], dtype=bool)
>
> how can I have boolean index like 4 < a < 8
> np.where(a>4 and a<8);or plainly input "a>4 and a<8" doesn't work.
>
> thanks,
>
> Chao
>

I1 a=np.arange(11)

I2 a[(a<8) & (a>4)]
O2 array([5, 6, 7])

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reading a big netcdf file

[Gökhan Sever]

Back to the reality. After clearing the cache using Warren's suggestion:

In [1]: timeit -n1 -r1 a = np.fromfile('temp.npa', dtype=np.uint16)
1 loops, best of 1: 7.23 s per loop


On Wed, Aug 3, 2011 at 4:52 PM, Eric Firing  wrote:

> On 08/03/2011 11:24 AM, Gökhan Sever wrote:
>
> > I[1]: timeit a = np.fromfile('temp.npa', dtype=np.uint16)
> > 1 loops, best of 3: 263 ms per loop
>
> You need to clear your cache and then run timeit with options "-n1 -r1".
>
> Eric
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reading a big netcdf file

[Gökhan Sever]

On Wed, Aug 3, 2011 at 3:15 PM, Christopher Barker wrote:

> On 8/3/11 1:57 PM, Gökhan Sever wrote:
> > This is what I get here:
> >
> > In [1]: a = np.zeros((21601, 10801), dtype=np.uint16)
> >
> > In [2]: a.tofile('temp.npa')
> >
> > In [3]: del a
> >
> > In [4]: timeit a = np.fromfile('temp.npa', dtype=np.uint16)
> > 1 loops, best of 3: 251 ms per loop
>
> so that's about 10 times faster than my machine. I didn't think disks
> had gotten much faster -- they are still generally 7200 rpm (or slower
> in laptops).
>
> So I've either got a really slow disk, or you have a really fast one (or
> both), or maybe you're getting cache effect, as you wrote the file just
> before reading it.
>
> repeating, doing just what you did:
>
> In [8]: timeit a = np.fromfile('temp.npa', dtype=np.uint16)
> 1 loops, best of 3: 2.53 s per loop
>
> then I wrote a bunch of others to disk, and tried again:
>
> In [17]: timeit a = np.fromfile('temp.npa', dtype=np.uint16)
> 1 loops, best of 3: 2.45 s per loop
>
> so ti seems I'm not seeing cache effects, but maybe you are.
>
> Anyway, we haven't heard from the OP -- I'm not sure what s/he thought
> was slow.
>
> -Chris



In [11]: a = np.zeros((21601, 10801), dtype=np.uint16)

In [12]: a.tofile('temp.npa')

In [13]: del a

Quitting here and restarting IPython. (this should cut the caching effect
isn't it?)

I[1]: timeit a = np.fromfile('temp.npa', dtype=np.uint16)
1 loops, best of 3: 263 ms per loop

#More information about my system:
hdparm -I /dev/sda | grep Rotation
Nominal Media Rotation Rate: 7200

uname -a  #64-bit Fedora 14
Linux ccn 2.6.35.13-92.fc14.x86_64 #1

Filesystem(s) ext4

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reading a big netcdf file

[Gökhan Sever]

I think these answer your questions.

In [3]: type f.variables['reflectivity']
--> type(f.variables['reflectivity'])
Out[3]: 

In [4]: type f.variables['reflectivity'][:]
--> type(f.variables['reflectivity'][:])
Out[4]: 

In [5]: z = f.variables['reflectivity'][:]

In [6]: type z
--> type(z)
Out[6]: 

In [10]: id f.variables['reflectivity'][:]
---> id(f.variables['reflectivity'][:])
Out[10]: 37895488

In [11]: id z
---> id(z)
Out[11]: 37901440


On Wed, Aug 3, 2011 at 12:40 PM, Christopher Barker
wrote:

> On 8/3/11 9:46 AM, Gökhan Sever wrote:
> > In [23]: from netCDF4 import Dataset
> >
> > In [24]: f = Dataset("test.nc <http://test.nc>")
> >
> > In [25]: f.variables['reflectivity'].shape
> > Out[25]: (6, 18909, 506)
> >
> > In [26]: f.variables['reflectivity'].size
> > Out[26]: 57407724
> >
> > In [27]: f.variables['reflectivity'][:].dtype
> > Out[27]: dtype('float32')
> >
> > In [28]: timeit z = f.variables['reflectivity'][:]
> > 1 loops, best of 3: 731 ms per loop
>
> that seems pretty fast, actually -- are you sure that [:] forces the
> full data read? It probably does, but I'm not totally sure.
>
> is "z" a numpy array object at that point?
>
> -Chris
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reading a big netcdf file

[Gökhan Sever]

This is what I get here:

In [1]: a = np.zeros((21601, 10801), dtype=np.uint16)

In [2]: a.tofile('temp.npa')

In [3]: del a

In [4]: timeit a = np.fromfile('temp.npa', dtype=np.uint16)
1 loops, best of 3: 251 ms per loop


On Wed, Aug 3, 2011 at 10:50 AM, Christopher Barker
wrote:

> On 8/3/11 9:30 AM, Kiko wrote:
> > I'm trying to read a big netcdf file (445 Mb) using netcdf4-python.
>
> I've never noticed that netCDF4 was particularly slow for reading
> (writing can be pretty slow some times). How slow is slow?
>
> > The data are described as:
>
> please post the results of:
>
> ncdump -h the_file_name.nc
>
> So we can see if there is anything odd in the structure (though I don't
> know what it might be)
>
> Post your code (in the simnd pplest form you can).
>
> and post your timings and machine type
>
> Is the file netcdf4 or 3 format? (the python lib will read either)
>
> As a reference, reading that much data in from a raw file into a numpy
> array takes 2.57 on my machine (a rather old Mac, but disks haven't
> gotten much  faster). YOu can test that like this:
>
> a = np.zeros((21601, 10801), dtype=np.uint16)
>
> a.tofile('temp.npa')
>
> del a
>
> timeit a = np.fromfile('temp.npa', dtype=np.uint16)
>
> (using ipython's timeit)
>
> -Chris
>
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reading a big netcdf file

[Gökhan Sever]

Just a few extra tests on my side pushing the limits of my system memory:

In [34]: k = np.zeros((21601, 10801, 3), dtype='int16')
k  ndarray 21601x10801x3: 699937203 elems, type `int16`,
1399874406 bytes (1335 Mb)

And for the first time my memory explodes with a hard kernel crash:

In [36]: k = np.zeros((21601, 10801, 13), dtype='int16')

Message from syslogd@ccn at Aug  3 10:51:43 ...
 kernel:[48715.531155] [ cut here ]

Message from syslogd@ccn at Aug  3 10:51:43 ...
 kernel:[48715.531163] invalid opcode:  [#1] SMP

Message from syslogd@ccn at Aug  3 10:51:43 ...
 kernel:[48715.531166] last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map

Message from syslogd@ccn at Aug  3 10:51:43 ...
 kernel:[48715.531253] Stack:

Message from syslogd@ccn at Aug  3 10:51:43 ...
 kernel:[48715.531265] Call Trace:

Message from syslogd@ccn at Aug  3 10:51:43 ...
 kernel:[48715.531332] Code: be 33 01 00 00 48 89 fb 48 c7 c7 67 31 7a 81 e8
b0 2d f1 ff e8 90 f2 33 00 48 89 df e8 86 db 00 00 48 83 bb 60 01 00 00 00
74 02 <0f> 0b 48 8b 83 10 02 00 00 a8 20 75 02 0f 0b a8 40 74 02 0f 0b


On Wed, Aug 3, 2011 at 10:46 AM, Gökhan Sever  wrote:

> Here are my values for your comparison:
>
> test.nc file is about 715 MB. The details are below:
>
> In [21]: netCDF4.__version__
> Out[21]: '0.9.4'
>
> In [22]: np.__version__
> Out[22]: '2.0.0.dev-b233716'
>
> In [23]: from netCDF4 import Dataset
>
> In [24]: f = Dataset("test.nc")
>
> In [25]: f.variables['reflectivity'].shape
> Out[25]: (6, 18909, 506)
>
> In [26]: f.variables['reflectivity'].size
> Out[26]: 57407724
>
> In [27]: f.variables['reflectivity'][:].dtype
> Out[27]: dtype('float32')
>
> In [28]: timeit z = f.variables['reflectivity'][:]
> 1 loops, best of 3: 731 ms per loop
>
> How long it takes in your side to read that big array?
>
> On Wed, Aug 3, 2011 at 10:30 AM, Kiko  wrote:
>
>> Hi.
>>
>> I'm trying to read a big netcdf file (445 Mb) using netcdf4-python.
>>
>> The data are described as:
>> *The GEBCO gridded data set is stored in NetCDF as a one dimensional
>> array of 2-byte signed integers that represent integer elevations in metres.
>>
>> The complete data set gives global coverage. It consists of 21601 x 10801
>> data values, one for each one minute of latitude and longitude for 233312401
>> points.
>> The data start at position 90°N, 180°W and are arranged in bands of 360
>> degrees x 60 points/degree + 1 = 21601 values. The data range eastward from
>> 180°W longitude to 180°E longitude, i.e. the 180° value is repeated.*
>>
>> The problem is that it is very slow (or I am quite newbie).
>>
>> Anyone has a suggestion to get these data in a numpy array in a faster
>> way?
>>
>> Thanks in advance.
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
>
> --
> Gökhan
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reading a big netcdf file

[Gökhan Sever]

Here are my values for your comparison:

test.nc file is about 715 MB. The details are below:

In [21]: netCDF4.__version__
Out[21]: '0.9.4'

In [22]: np.__version__
Out[22]: '2.0.0.dev-b233716'

In [23]: from netCDF4 import Dataset

In [24]: f = Dataset("test.nc")

In [25]: f.variables['reflectivity'].shape
Out[25]: (6, 18909, 506)

In [26]: f.variables['reflectivity'].size
Out[26]: 57407724

In [27]: f.variables['reflectivity'][:].dtype
Out[27]: dtype('float32')

In [28]: timeit z = f.variables['reflectivity'][:]
1 loops, best of 3: 731 ms per loop

How long it takes in your side to read that big array?

On Wed, Aug 3, 2011 at 10:30 AM, Kiko  wrote:

> Hi.
>
> I'm trying to read a big netcdf file (445 Mb) using netcdf4-python.
>
> The data are described as:
> *The GEBCO gridded data set is stored in NetCDF as a one dimensional array
> of 2-byte signed integers that represent integer elevations in metres.
> The complete data set gives global coverage. It consists of 21601 x 10801
> data values, one for each one minute of latitude and longitude for 233312401
> points.
> The data start at position 90°N, 180°W and are arranged in bands of 360
> degrees x 60 points/degree + 1 = 21601 values. The data range eastward from
> 180°W longitude to 180°E longitude, i.e. the 180° value is repeated.*
>
> The problem is that it is very slow (or I am quite newbie).
>
> Anyone has a suggestion to get these data in a numpy array in a faster way?
>
> Thanks in advance.
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Unicode characters in a numpy array

[Gökhan Sever]

On Thu, Jun 16, 2011 at 8:54 PM, Charles R Harris
 wrote:
>
>
> On Wed, Jun 15, 2011 at 1:30 PM, Gökhan Sever  wrote:
>>
>> Hello,
>>
>> The following snippet works fine for a regular string and prints out
>> the string without a problem.
>>
>> python
>> Python 2.7 (r27:82500, Sep 16 2010, 18:02:00)
>> [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> mystr = u"öööğğğ"
>> >>> mystr
>> u'\xf6\xf6\xf6\u011f\u011f\u011f'
>> >>> type(mystr)
>> 
>> >>> print mystr
>> öööğğğ
>>
>> What is the correct way to print out the following array?
>>
>> >>> import numpy as np
>> >>> arr = np.array(u"öööğğğ")
>> >>> arr
>> array(u'\xf6\xf6\xf6\u011f\u011f\u011f',
>>      dtype='> >>> print arr
>> Traceback (most recent call last):
>>  File "", line 1, in 
>>  File "/usr/lib64/python2.7/site-packages/numpy/core/numeric.py",
>> line 1379, in array_str
>>    return array2string(a, max_line_width, precision, suppress_small,
>> ' ', "", str)
>>  File "/usr/lib64/python2.7/site-packages/numpy/core/arrayprint.py",
>> line 426, in array2string
>>    lst = style(x)
>> UnicodeEncodeError: 'ascii' codec can't encode characters in position
>> 0-5: ordinal not in range(128)
>>
>
> I don't know. It might be that we need to fix the printing functions for
> unicode and maybe have some way to set the codec as well.
>
> Chuck
>

Typing
arr = np.array(u"öööğğğ")

yields UnicodeEncodeError: 'ascii' codec can't encode characters in
position 17-22: ordinal not in range(128)
in IPython 0.10. I am not sure if this is fixed in the new-coming IPython.

Typing the array in this form (with brackets) makes a difference:

>>> arr = np.array([u"öööğğğ"])
>>> print arr
[u'\xf6\xf6\xf6\u011f\u011f\u011f']
>>> print arr[0]
öööğğğ

I am wondering whether "print arr" should print out the unicode
characters in human-readable format or in this current form.

This applies to the regular Python lists as well.

>>> mylist = [u"öööğğğ"]
>>> print mylist
[u'\xf6\xf6\xf6\u011f\u011f\u011f']
>>> print mylist[0]
öööğğğ
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Unicode characters in a numpy array

[Gökhan Sever]

Hello,

The following snippet works fine for a regular string and prints out
the string without a problem.

python
Python 2.7 (r27:82500, Sep 16 2010, 18:02:00)
[GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> mystr = u"öööğğğ"
>>> mystr
u'\xf6\xf6\xf6\u011f\u011f\u011f'
>>> type(mystr)

>>> print mystr
öööğğğ

What is the correct way to print out the following array?

>>> import numpy as np
>>> arr = np.array(u"öööğğğ")
>>> arr
array(u'\xf6\xf6\xf6\u011f\u011f\u011f',
  dtype='>> print arr
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib64/python2.7/site-packages/numpy/core/numeric.py",
line 1379, in array_str
return array2string(a, max_line_width, precision, suppress_small,
' ', "", str)
  File "/usr/lib64/python2.7/site-packages/numpy/core/arrayprint.py",
line 426, in array2string
lst = style(x)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
0-5: ordinal not in range(128)


Thanks.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Segfault using "fromstring" and reading variable length string

[Gökhan Sever]

On Fri, Apr 22, 2011 at 6:32 PM, Mark Wiebe  wrote:

> I took a quick look at this issue and committed a fix. PyArray_FromString
> was doing a check to exclude object arrays, but that check was incorrect.
> Now it should appropriately raise an exception instead of creating an
> invalid array.
>
>
> https://github.com/numpy/numpy/commit/f75bfab3a2ab74ac82047f153a36c71c58fe3732
>
> -Mark
>

Thanks for the fix.

Now that line yields:

I[6]: k = np.fromstring(c.read(dt.itemsize), dt)[0]
---
ValueErrorTraceback (most recent call last)

/home/gsever/Desktop/python-repo/numpy/ in ()

ValueError: Cannot create an object array from a string


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Segfault using "fromstring" and reading variable length string

[Gökhan Sever]

On Fri, Apr 22, 2011 at 12:37 PM, Ralf Gommers
wrote:

> On Thu, Apr 21, 2011 at 10:06 PM, Gökhan Sever 
> wrote:
> > Hello,
> > Given this piece of code (I can provide the meg file off-the list for
> those
> > who wants to reproduce the error)
>
> Can you instead construct a test as simple as possible for this? It
> sounds like you need only a two line string to reproduce this. The bug
> sounds similar to http://projects.scipy.org/numpy/ticket/1689.
>
> Ralf


This simple case segfaults as well (The commented line works correctly):

import numpy as np
from StringIO import StringIO

c = StringIO(" hello \r\n world \r\n")

dt = np.dtype([('line1', '|S6'), ('line2', np.object_)])
#dt = np.dtype([('line1', '|S9'), ('line2', '|S9')])
k = np.fromstring(c.read(dt.itemsize), dt)[0]

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Segfault using "fromstring" and reading variable length string

[Gökhan Sever]

Hello,

Given this piece of code (I can provide the meg file off-the list for those
who wants to reproduce the error)


import numpy as np

f = open("a08A0122.341071.meg", "rb")

dt = np.dtype([('line1', '|S80'), ('line2', np.object_), ('line3', '|S80'),
('line4', '|S80'),
   ('line5', '|S80'), ('line6', '|S2'), ('line7', np.int32,
2000), ('line8', '|S2'),
   ('line9', np.int32, 2000), ('line10', '|S2')])

k = np.fromstring(f.read(dt.itemsize), dt)[0]

Accessing k causes a "Segmentation fault (core dumped)" and kills my python
and IPython sessions immediately.  I actually know that the culprit is
"np.object_" in this case.  The original was as ('line2', '|S81') however
those meg files (mix of text and binary content) have a funny habit of
switching from 80 characters to 81 (including "/r/n" chars). I was testing
if I could create a variable length string dtype, which seems not possible.

Little more info: that line2 has time stamps, one of which is in the form
of 22:34:59.999. I have seen in the file that 22:34:59.999 was originally
written as 22:34:59.1000 which causes that extra character flow.
(Interestingly, millisecond should cycle from 0-999 and overflow at 999
instead of 1000 which to me indicates a slight bug) Because of this reason,
I can't read the whole content of those meg files since somewhere in the
middle fromstring attempts reading a shifted (erroneous) content. Should I
go fix that millisecond overflow first or is there an alternative way to
approach this problem?

Thanks


Platform : Linux-2.6.35.12-88.fc14.x86_64-x86_64-with-fedora-14-Laughlin
Python   : ('CPython', 'tags/r27', '82500')
IPython  : 0.10
NumPy   : 2.0.0.dev-2e96d91


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Anybody going to PyCon?

[Gökhan Sever]

Yung-Yu,

We are advertised on this blog
http://pycon.blogspot.com/2011/03/pycon-2011-outside-talks-poster-session.html

I will be in the conference venue by tomorrow morning. There are many
interesting talks and posters that I look forward seeing plus meeting
those presenters.

See you in Atlanta.

On 3/9/11, Yung-Yu Chen  wrote:
> I will be there tomorrow, and giving a talk about solving PDEs on Saturday
> morning (plus a poster).  Looking forward to meet you guys and to learn more
> about contribution to the community.
>
> with regards,
> Yung-Yu Chen
>
> --
> Yung-Yu Chen
> PhD candidate of Mechanical Engineering
> The Ohio State University, Columbus, Ohio
> +1 (614) 859 2436
> http://solvcon.net/yyc/
>

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Anybody going to PyCon?

[Gökhan Sever]

Hello,

I am going to the PyCon this week. I am presenting a poster about an
atmospheric sciences related project -- the most active development
from my coding site over at http://code.google.com/p/ccnworks/

Is there anybody in the community participating there as well? Any
plans for sprinting or similar activities?

See you at PyCon.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy speed tests by NASA

[Gökhan Sever]

On Tue, Feb 22, 2011 at 2:44 PM, Alan G Isaac  wrote:
>
>
> I don't believe the matrix multiplication results.
> Maybe I misunderstand them ...
>
> >>> t = timeit.Timer("np.dot(A,B)","import numpy as
> np;N=1500;A=np.random.random((N,N));B=np.random.random((N,N))")
> >>> print t.timeit(number=10)/10.
> 1.09043075307
>
> I'm using the precompiled Windows binaries.


This is on Fedora 14 x86_64 --using Fedora provided builds of ATLAS, and
build using gcc/gfortran.

>>> t = timeit.Timer("np.dot(A,B)","import numpy as
np;N=1500;A=np.random.random((N,N));B=np.random.random((N,N))")
>>> print t.timeit(number=10)/10.
0.497710204124

I am guessing ATLAS is thread aware since with N=15000 each of the quad core
runs at %100. Probably MKL build doesn't bring much speed advantage in this
computation. Any thoughts?





-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy speed question

[Gökhan Sever]

On Thu, Nov 25, 2010 at 4:13 AM, Jean-Luc Menut  wrote:
> Hello all,
>
> I have a little question about the speed of numpy vs IDL 7.0. I did a
> very simple little check by computing just a cosine in a loop. I was
> quite surprised to see an order of magnitude of difference between numpy
> and IDL, I would have thought that for such a basic function, the speed
> would be approximatively the same.
>
> I suppose that some of the difference may come from  the default data
> type of 64bits in numpy and 32 bits in IDL. Is there a way to change the
> numpy default data type (without recompiling) ?
>
> And I'm not an expert at all, maybe there is a better explanation, like
> a better use of the several CPU core by IDL ?
>
> I'm working with windows 7 64 bits on a core i7.
>
> any hint is welcome.
> Thanks.
>
> Here the IDL code :
> Julian1 = SYSTIME( /JULIAN , /UTC )
> for j=0, do begin
>   for i=0,999 do begin
>     a=cos(2*!pi*i/100.)
>   endfor
> endfor
> Julian2 = SYSTIME( /JULIAN , /UTC )
> print, (Julian2-Julian1)*86400.0
> print,cpt
> end
>
> result:
> % Compiled module: $MAIN$.
>        2.837
>
>
> The python code:
> from numpy import *
> from time import time
> time1 = time()
> for j in range(1):
>     for i in range(1000):
>         a=cos(2*pi*i/100.)
> time2 = time()
> print time2-time1
>
> result:
> In [2]: run python_test_speed.py
> 24.180943
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

Vectorised numpy version already blow away the results.

Here is what I get using the IDL version (with IDL v7.1):

IDL> .r test_idl
% Compiled module: $MAIN$.
   4.185

I[10]: time run test_python
43.305727005

and using a Cythonized version:

from math import pi

cdef extern from "math.h":
float cos(float)

cpdef float myloop(int n1, int n2, float n3):
cdef float a
cdef int i, j
for j in range(n1):
for i in range(n2):
a=cos(2*pi*i/n3)

compiling the setup.py file python setup.py build_ext --inplace
and importing the function into IPython

from mycython import myloop

I[6]: timeit myloop(1, 1000, 100.0)
1 loops, best of 3: 2.91 s per loop


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Return values stats.mstats.linregress

[Gökhan Sever]

Hello,

mstats.linregress returns 6 values. I don't see this documented from
the function docstring. I know 0.91... is r. What is masked_array
return here?

I[29]: stats.mstats.linregress(np.ma.hstack(all_measured[0::6]),
np.ma.hstack(all_predicted[0::6]))
O[29]:
(2.6309756058562122,
 -358.84572340482669,
 0.91388013590912054,
 masked_array(data = 8.93881546632e-73,
 mask = False,
   fill_value = 1e+20)
,
 241.88030264140224,
 8.2881035327089627)

When I run with mstats:

I[30]: stats.linregress(np.ma.hstack(all_measured[0::6]),
np.ma.hstack(all_predicted[0::6]))
O[30]:
(2.5731434548927012,
 -325.98658013998079,
 0.90619871728013224,
 1.4284868970559915e-69,
 0.089246202128717186)

Any idea also why the returned p-value is so large from the mstats
version. Shouldn't p-values lie in between [0,1) ?

Thanks.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Appending/combining masked arrays

[Gökhan Sever]

On Wed, Sep 29, 2010 at 5:09 PM, Pierre GM  wrote:
>
> On Sep 29, 2010, at 11:46 PM, josef.p...@gmail.com wrote:
>>
>> any of the ma stack array function might also work, or not?
>
> In np.ma.extras ? Most likely, yes

This seems to do what I want:

I[262]: np.ma.hstack(all_measured)
O[262]:
masked_array(data = [382.4828 375.57736 387.56132 ..., 371.29348
364.26819 374.77477],
 mask = [False False False ..., False False False],
   fill_value = 1e+20)

I don't know why there are replicas under np.ma.extras.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Appending/combining masked arrays

[Gökhan Sever]

> You're using a standard numpy function on a masked array. It's hardly 
> surprising that you run into some issues. You should use the np.ma 
> equivalent. Except of course that the equivalent doesn't exist yet... Please 
> open a ticket.

Here it comes -> http://projects.scipy.org/numpy/ticket/1623

> In the mean time, do it manually:
> * combining the .data of each item (masked array) of your list is 
> straightforward
> * repeat for the masks. I advise you to use the getmaskarray function to get 
> the mask of each item, otherwise you may get a `nomask` in the middle which 
> will complicate things
> * combine the the two results in a new masked array.

Does this look good?

I[256]: a = np.ma.masked_equal([1,2,3], value=2)
I[257]: b = np.ma.masked_equal([4,3,2, 5], value=2)
I[258]: c = np.append(a.data, b.data)

I[251]: cmask = np.append(np.ma.getmaskarray (a), np.ma.getmaskarray (b))

I[252]: c
O[252]: array([1, 2, 3, 4, 3, 2, 5])

I[253]: cmask
O[253]: array([False,  True, False, False, False,  True, False], dtype=bool)

I[254]: d = np.ma.masked_array(c, cmask)

I[255]: d
O[255]:
masked_array(data = [1 -- 3 4 3 -- 5],
 mask = [False  True False False False  True False],
   fill_value = 99)

Probably I will need a loop to construct a similar ma array for 7
different masked arrays in a list.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Appending/combining masked arrays

[Gökhan Sever]

Hello,

Consider these two simple masked arrays:

I[188]: a = np.ma.masked_equal([1,2,3], value=2)

I[189]: b = np.ma.masked_equal([4,3,2], value=2)

An operation like this voids the mask:

I[190]: np.append(a,b)
O[190]:
masked_array(data = [1 2 3 4 3 2],
 mask = False,
   fill_value = 99)

In my real use case I have two lists (shown simplified versions):

I[193]: all_measured[5::14][1:]
O[193]:
[masked_array(data = [425.82268 441.8043 432.69865 433.75158 420.42552
469.73359 483.80741
 427.66887 466.7487 452.64255 438.14488 428.38871 416.38598 432.92884
 440.74705 415.00694 430.1807 446.02079 428.1408 428.21708 461.37897
 453.43518 433.90081 426.88591 451.15683 426.07399 410.7971 455.19179
 389.01905 485.69204 505.35355 523.30598 502.00168 491.85421 485.75839
 473.37061 459.24917 438.47531 424.09222 411.82773 409.27676 366.24813
 362.0136 385.61986 350.38855 357.10589 390.84878 390.53565 332.60864
 368.45913],
 mask = [False False False False False False False False
False False False False
 False False False False False False False False False False False False
 False False False False False False False False False False False False
 False False False False False False False False False False False False
 False False],
   fill_value = 1e+20)
,
 masked_array(data = [762.00897 756.79155 773.59503 757.97867
746.20204 752.0657],
 mask = [False False False False False False],
   fill_value = 1e+20)
]

I[194]: all_predicted[5::14][1:]
O[194]:
[masked_array(data = [601.587925396 615.382975948 637.565961135
662.845855035 630.180285797
 910.714363555 886.048093691 912.616380221 1087.38406572 789.0075947
 777.900831884 733.319025182 750.579326854 752.627618389 696.521605131
 633.362074267 722.437789869 730.89750503 692.179530427 703.786215707
 808.592649936 1006.89797524 818.839286207 767.260255009 787.622382926
 831.332970348 949.016807581 783.981396594 643.768619685 654.417348215
 681.516301642 753.577103851 654.092465489 628.484105951 691.461588689
 800.901347497 630.894132084 610.977386345 512.926749811 653.74866061
 587.915074604 531.106658494 562.265237436 606.32672755 563.281067561
 546.715211886 604.210379352 475.66452212 454.426293217 656.039874394],
 mask = [False False False False False False False False
False False False False
 False False False False False False False False False False False False
 False False False False False False False False False False False False
 False False False False False False False False False False False False
 False False],
   fill_value = 1e+20)
,
 masked_array(data = [891.251806903 833.185882945 840.25752
831.649215796 883.534378034
 841.970022166],
 mask = [False False False False False False],
   fill_value = 1e+20)
]

These lists have 42 varying size masked arrays in each. I want to be
able to combine each list in one array --preferably in a masked array
for not losing the mask information so that I can perform some overall
statistics.

What is the way to solve this issue?

Thanks

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] slicing / indexing question

[Gökhan Sever]

On Tue, Sep 21, 2010 at 6:20 PM, Timothy W. Hilton wrote:

> Hello,
>
> I have an indexing problem which I suspect has a simple solution, but
> I've not been able to piece together various threads I've read on this
> list to solve.
>
> I have an 80x1200x1200 nd.array of floats this_par.  I have a
> 1200x1200 boolean array idx, and an 80-element float array pars.  For
> each element of idx that is True, I wish to replace the corresponding
> 80x1x1 slice of this_par with the elements of pars.
>
> I've tried lots of variations on the theme of
> >>>this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis]
> but so far, no dice.
>
> Any help greatly appreciated!
>
> Thanks,
> Tim
>

This is a tough indexing question for me. I can visualize the arrays and the
required selections in my mind, but hard to put them into application as in
this case. Beyond 2D array operations indexing and broadcasting operations
get somewhat complex to interpret at the first look and likewise later when
I read and decipher my own code, not to mention if someone else wants to
understand that same code.

What do you do with such big arrays?  It is good to see new atmospheric
sciences people in the lists. Our department is still swarming with IDL and
Matlab users.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] unique 2d arrays

[Gökhan Sever]

On Tue, Sep 21, 2010 at 12:43 PM,  wrote:

> I'm a bit surprised, I think np.unique does some extra work to
> maintain the order.
> The tolist() might not be necessary if you iterate over rows.
>

Testing again with a smaller k array and more repeats

I[25]: k = np.array((a.tolist()*5000))

I[27]: %timeit -r 100 np.array(list(set(tuple(i) for i in k.tolist(
10 loops, best of 100: 31.3 ms per loop

I[28]: %timeit -r 100 np.array(list(set(tuple(i) for i in k)))
10 loops, best of 100: 55.4 ms per loop

I[30]: %timeit -r 100
np.unique(k.view([('',k.dtype)]*k.shape[1])).view(k.dtype).reshape(-1,k.shape[1])
10 loops, best of 100: 60.5 ms per loop

.tolist version is faster. Can you also verify this?



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] unique 2d arrays

[Gökhan Sever]

On Tue, Sep 21, 2010 at 1:55 AM, Peter Schmidtke
wrote:

> Dear all,
>
> I'd like to know if there is a pythonic / numpy way of retrieving unique
> lines of a 2d numpy array.
>
> In a way I have this :
>
> [[409 152]
>  [409 152]
>  [409 152]
>  [409 152]
>  [409 152]
>  [409 152]
>  [409 152]
>  [409 152]
>  [409 152]
>  [409 152]
>  [409 152]
>  [426 193]
>  [431 129]]
>
> And I'd like to get this :
>
> [[409 152]
>  [426 193]
>  [431 129]]
>
>
> How can I do this without workarounds like string concatenation or such
> things? Numpy.unique flattens the whole array so it's not really of use
> here.
>

Here is one alternative:

I[15]: a = np.array([[409, 152], [409, 152], [426, 193], [431, 129]])

I[16]: np.array(list(set(tuple(i) for i in a.tolist(
O[16]:
array([[409, 152],
   [426, 193],
   [431, 129]])

I[6]: %timeit
np.unique(a.view([('',a.dtype)]*a.shape[1])).view(a.dtype).reshape(-1,a.shape[1])
1 loops, best of 3: 51 us per loop

I[8]: %timeit np.array(list(set(tuple(i) for i in a.tolist(
1 loops, best of 3: 31.4 us per loop

# Try with a bigger array
I[9]: k = np.array((a.tolist()*5))

I[10]: %timeit np.array(list(set(tuple(i) for i in k.tolist(
1 loops, best of 3: 324 ms per loop

I[11]: %timeit
np.unique(k.view([('',k.dtype)]*k.shape[1])).view(k.dtype).reshape(-1,k.shape[1])
1 loops, best of 3: 790 ms per loop


Seems like faster on these tests comparing to the unique method. Also it is
more readable. Still not uber Pythonic. Haskell has "nub" to remove
duplicate list elements.
http://www.haskell.org/ghc/docs/6.12.2/html/libraries/base-4.2.0.1/Data-List.html#v%3Anub



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about masked arrays

[Gökhan Sever]

On Mon, Sep 20, 2010 at 3:34 PM, Benjamin Root  wrote:

> I have been using masked arrays quite extensively.  My take on them is that
> if a masked array makes sense in that operation, then they should still work
> with the regular functions.  However, there have been many cases where a
> developer used np.asarray() instead of np.asanyarray() for their code, which
> causes the masked array object to lose the mask.  If you encounter such
> situations, it is usually a bug and should be reported.
>

Do you think then in my case assigning the masked array to np.zeros and
getting a non-masked return is a bug or I should make sure that I am
providing a masked array equivalent zeros function before I start
doing computation?

And also if the inner execution could be clarified by asanyarray assertion
why there is ma equivalent array operation functions?


>
> Actually, that reminds me... watch out for np.polyfit() with masked
> arrays.  It doesn't behave quite nicely with masked arrays and the results
> are deceptive.  It may appear to be right, but it is not.  Use
> np.ma.polyfit().
>

Hah, you should see my scipy.optimize.leastsq in action. I estimate
geometric mean and standard deviation parameters for log-normal aerosol
population. It runs, with the values masked and unmasked (from above
np.zeros and np.ma.zeros difference) but resulting all kind interesting
estimates.


>
> Ben Root
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about masked arrays

[Gökhan Sever]

On Mon, Sep 20, 2010 at 1:05 PM, Robert Kern  wrote:

> Are you asking about when masked arrays are casted to ndarrays (and
> thus losing the mask information)? Most times when a function uses
> asarray() or array() to explicitly cast the inputs to an ndarray. The
> reason that np.mean() gives the same result as np.ma.mean() is that it
> simply defers to the .mean() method on the object, so it works as
> expected on a masked array. Many other functions will not.
>
> --
> Robert Kern
>

Right guess. It is important for me to able to preserve masked array
properties of an array. Otherwise losing the mask information yields
unexpected results in some of my calculations. I could see from np.mean??
that mean function is indeed the object method. Also in /numpy/ma there is a
conversion for np.zeros(). I guess in any case it is the user's
responsibility to make sure that the operations are performed on a desired
array type.


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Question about masked arrays

[Gökhan Sever]

Hello,

Consider these two sets of container arrays --one defined as usual np array
the others as ma arrays:

all_measured = np.ma.zeros((16, 18))
all_predicted = np.ma.zeros((16, 18))
all_measured2 = np.zeros((16, 18))
all_predicted2 = np.zeros((16, 18))

I do a computation within a for loop to assign 16 set of measurements into a
length of 18 arrays (thus constructing a 2D array to perform overall
statistics and plotting.) For the simplicity I only show a portion of
all_measured and all_measured2 as:

all_measured
masked_array(data =
 [[512.632175 527.33373 565.36541 567.53967 593.86833 570.31319 574.40965
  582.72649 588.21336 618.48789 593.09007 620.33474 591.10203 611.06443
  655.60614 638.13193 626.71769 625.63584]
 [626.6435 -- -- 1183.67671 1206.82453 1183.13248 1162.5514 1180.70062
  1086.53246 1078.78711 997.1642 856.57159 645.35167 696.86947 778.40914
  816.03059 862.88297 901.7237] ...

all_measured2
array([[  512.632175  ,   527.33373   ,   565.36541   ,   567.53967   ,
  593.86833   ,   570.31319   ,   574.40965   ,   582.72649   ,
  588.21336   ,   618.48789   ,   593.09007   ,   620.33474   ,
  591.10203   ,   611.06443   ,   655.60614   ,   638.13193   ,
  626.71769   ,   625.63584   ],
   [  626.6435, 0., 0.,  1183.67671   ,
 1206.82453   ,  1183.13248   ,  1162.5514,  1180.70062   ,
 1086.53246   ,  1078.78711   ,   997.1642,   856.57159   ,
  645.35167   ,   696.86947   ,   778.40914   ,   816.03059   ,
  862.88297   ,   901.7237],...

The issue is why masked arrays casted to regular numpy arrays as in
all_measured2 case? whereas a simple numpy function np.mean and ma
equivalent np.ma.mean yields same results on all_measured? Because the
former requires a priori knowledge about the type of arrays, however the
latter doesn't necessitate such restriction.

Hope this is clear. Thanks.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI)

[Gökhan Sever]

On Fri, Sep 17, 2010 at 12:16 AM, Mayank P Jain  wrote:

>  Currently I am exporting them to csv files, but I wonder if there is a
> viewer that can be used with native numpy array files to view and preferably
> modify the 2D arrays.
> Any help would be appreciated.
>

I would suggest using IPython + Matplotlib, and as a text editor Vim. I have
just learnt how to select the same column in Vim [
http://stackoverflow.com/questions/3736678/select-all-column]. This trio is
as simple as and yet as powerful as it gets for my array view
+ visualization tasks.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] common significant diigits

[Gökhan Sever]

On Wed, Sep 15, 2010 at 2:34 PM, Benjamin Root  wrote:

> Hello,
>
> I am trying to solve a problem in matplotlib where I would have an array of
> floating point numbers and I want to quickly determine what is the closest
> common offset to a power of 10.  In other words, if given:
>
> [12373.43, 12375.89, 12370.18],
>
> I would want returned something like either 12370.0, or the lowest common
> order of magnitude (in this case, 10).
>
> Is there some sort of neat math/numpy trick to figure this out?  I already
> have a brute-force method with a while loop, but I am looking for something
> a little bit more elegant.
>
> Thanks,
> Ben Root
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
Probably not being very fail-safe here is my first guess:

a = np.array([12373.43, 12375.89, 12370.18])
np.floor(a).min()


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] summing over more than one axis

[Gökhan Sever]

On Thu, Aug 19, 2010 at 9:01 AM, greg whittier  wrote:

> I frequently deal with 3D data and would like to sum (or find the
> mean, etc.) over the last two axes.  I.e. sum a[i,j,k] over j and k.
> I find using .sum() really convenient for 2d arrays but end up
> reshaping 2d arrays to do this.  I know there has to be a more
> convenient way.  Here's what I'm doing
>
> a = np.arange(27).reshape(3,3,3)
>
> # sum over axis 1 and 2
> result = a.reshape((a.shape[0], a.shape[1]*a.shape[2])).sum(axis=1)
>
> Is there a cleaner way to do this?  I'm sure I'm missing something obvious.
>
> Thanks,
> Greg
>

Using two sums

np.sum(np.sum(a, axis=-2), axis=1)


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Broken links on new.scipy

[Gökhan Sever]

Hi,

@ http://new.scipy.org/download.html numpy and scipy links for Fedora is
broken.

Could you update the links with these?

https://admin.fedoraproject.org/pkgdb/acls/name/numpy


https://admin.fedoraproject.org/pkgdb/acls/name/scipy

Thanks.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] {OT} Mailing trends

[Gökhan Sever]

Hello,

There is a nice e-mailing trend tool for Gmail users at
http://code.google.com/p/mail-trends/
It is a command line tool producing an html output showing your e-mailing
statistics. In my inbox, the following threads are highly ranked in the top
threads section.

[Numpy-discussion] Announcing toydist, improving distribution and packaging
situation
[SciPy-Dev] 
scipy.stats
[Numpy-discussion] curious about how people would feel about moving to
github

Just out of curiosity, are there any mailing trends (top threads, top
posters, etc...) provided for the Python related mailing archives?

Share your comments please.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Quick array value assignment based on common values

[Gökhan Sever]

On Wed, Aug 4, 2010 at 8:00 PM, Gökhan Sever  wrote:

>
>
> On Wed, Aug 4, 2010 at 6:59 PM,  wrote:
>
>> Hey folks,
>>
>> I've one array, x, that you could define as follows:
>> [[1, 2.25],
>>  [2, 2.50],
>>  [3, 2.25],
>>  [4, 0.00],
>>  [8, 0.00],
>>  [9, 2.75]]
>>
>> Then my second array, y, is:
>> [[1, 0.00],
>>  [2, 0.00],
>>  [3, 0.00],
>>  [4, 0.00],
>>  [5, 0.00],
>>  [6, 0.00],
>>  [7, 0.00],
>>  [8, 0.00],
>>  [9, 0.00],
>>  [10,0.00]]
>>
>> Is there a concise, Numpythonic way to copy the values of x[:,1] over to
>> y[:,1] where x[:,0] = y[:,0]? Resulting in, z:
>> [[1, 2.25],
>>  [2, 2.50],
>>  [3, 2.25],
>>  [4, 0.00],
>>  [5, 0.00],
>>  [6, 0.00],
>>  [7, 0.00],
>>  [8, 0.00],
>>  [9, 2.75],
>>  [10,0.00]]
>>
>> My current task has len(x) = 25000 and len(y) = 35 and looping through
>> is quite slow unfortunately.
>>
>> Many thanks,
>> -paul
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> My simplest approach would be:
>
> y[x[:0]-1] = x
>
> # Providing the arrays are nicely ordered and 1st column x is all integer.
>
> --
> Gökhan
>

With the forgotten comma ;)

y[x[:,0]-1] = x

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Quick array value assignment based on common values

[Gökhan Sever]

On Wed, Aug 4, 2010 at 6:59 PM,  wrote:

> Hey folks,
>
> I've one array, x, that you could define as follows:
> [[1, 2.25],
>  [2, 2.50],
>  [3, 2.25],
>  [4, 0.00],
>  [8, 0.00],
>  [9, 2.75]]
>
> Then my second array, y, is:
> [[1, 0.00],
>  [2, 0.00],
>  [3, 0.00],
>  [4, 0.00],
>  [5, 0.00],
>  [6, 0.00],
>  [7, 0.00],
>  [8, 0.00],
>  [9, 0.00],
>  [10,0.00]]
>
> Is there a concise, Numpythonic way to copy the values of x[:,1] over to
> y[:,1] where x[:,0] = y[:,0]? Resulting in, z:
> [[1, 2.25],
>  [2, 2.50],
>  [3, 2.25],
>  [4, 0.00],
>  [5, 0.00],
>  [6, 0.00],
>  [7, 0.00],
>  [8, 0.00],
>  [9, 2.75],
>  [10,0.00]]
>
> My current task has len(x) = 25000 and len(y) = 35 and looping through
> is quite slow unfortunately.
>
> Many thanks,
> -paul
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

My simplest approach would be:

y[x[:0]-1] = x

# Providing the arrays are nicely ordered and 1st column x is all integer.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] np.choose() question

[Gökhan Sever]

On Tue, Jun 8, 2010 at 2:32 PM, Hans Meine
wrote:
>
>
> Funny, that's exactly what I wanted to do (idx being a label/region image
> here),
> and what I tried today.
>
> You will be happy to hear that the even simpler solution is to just use
> fancy indexing (the name is justified here):
>
>  times[idx]
>
> will do the trick, too - no nead to flatten & reshape at all! :-)
> (Given that both are ndarrays of course.)
>

You have got the bonus points Hans!

I have a special ability to drag myself into list comprehensions for array
indexing questions. Numpy provides such simple operations where mortals like
me skip over easily or forget to convert lists as arrays before trying them.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] np.choose() question

[Gökhan Sever]

If we were at so or ask.scipy I would vote for Mark's solution :)

Usually in cases like yours, I tend to use the shortest version of the
solutions.

On Tue, Jun 8, 2010 at 2:08 PM, Andreas Hilboll  wrote:

> Hi,
>
> > newtimes = [times[idx[x][y]] for x in range(2) for y in range(2)]
> > np.array(newtimes).reshape(2,2)
> > array([[104, 102],
> >[103, 101]])
>
> Great, thanks a lot!
>
> Cheers,
>
> Andreas.
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] np.choose() question

[Gökhan Sever]

On Tue, Jun 8, 2010 at 11:24 AM, Andreas Hilboll  wrote:

> Hi there,
>
> I have a problem, which I'm sure can somehow be solved using np.choose()
> - but I cannot figure out how :(
>
> I have an array idx, which holds int values and has a 2d shape. All
> values inside idx are 0 <= idx < n. And I have a second array times,
> which is 1d, with times.shape = (n,).
>
> Out of these two arrays I now want to create a 2d array having the same
> shape as idx, and holding the values contained in times, as indexed by
> idx.
>
> A simple np.choose(idx,times) does not work (error "Need between 2 and
> (32) array objects (inclusive).").
>
> Example:
>
> idx = [[4,2],[3,1]]
> times = [100,101,102,103,104]
>
>  From these two I want to create an array
>
> result = [[104,102],[103,101]]
>
> How can this be done?
>
> Thanks a lot for your insight!
>
> Andreas
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

Here is a non numpy.choose solution:

newtimes = [times[idx[x][y]] for x in range(2) for y in range(2)]
np.array(newtimes).reshape(2,2)
array([[104, 102],
   [103, 101]])


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Wrong Eigenvalue (Approximation?)

[Gökhan Sever]

Floating point numbers; one of my recent favorite subjects... See this
hot Slashdot discussion subject: what every programmer should know
about floating-point arithmetic

On 5/16/10, Alan G Isaac  wrote:
> On 5/16/2010 12:03 AM, Gabriel Mihalache wrote:
>> The eigenvalue should be 1 exactly.
>
> http://floating-point-gui.de/
>
> hth,
> Alan Isaac
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about numpy.ma masking

[Gökhan Sever]

On Sun, May 9, 2010 at 2:42 PM, Eric Firing  wrote:

>
> The mask attribute can be a full array, or it can be a scalar to
> indicate that nothing is masked.  This is an optimization in masked
> arrays; it adds complexity, but it can save space and/or processing
> time. You can always access a full mask array by using
> np.ma.getmaskarray().  Or you can ensure the internal mask is an array,
> not a scalar, by using the shrink=False kwarg when making the masked
> array with np.ma.array().
>

shrink=False fits perfect for my use-case. I was guessing that leaving the
mask as scalar should something to do with optimization. Probably not many
people around write loops and check conditions based on the mask content
like I do :) I hope someone in SciPy10 will present a Numpy.MA talk or
tutorial describing all the nitty details of the module usage.


>
> Offhand, I suspect your loop can be eliminated by vectorization.
> Something like this:
>
> ns = len(shorter)
> slice0 = slice(ns)
> slice1 = slice(diff, diff+ns)
> cond1 = serialh.data['dccnTempSF'][slice0] != 0
> cond2 = np.ma.getmaskarray(basic.data['Air_Temp'][slice1]) == False
> cond = cond1 & cond2
> dccnConAmb[slice0][cond] = (serialc.data['dccnConc'][slice0][cond] *
>physical.data['STATIC_PR'][slice1][cond])
>

Bonus help :) My gmail has over 400 Python tagged e-mails collected over a
year. I get responses here (in mailing lists general) most of the time
faster than I get locally around my department. This (especially
no-appointments feature) doubles triples my learning experience. Just a
personal thanks to you and all who make these great mediums possible.

Anyways back to the topic again. The snippet I share is about a year old
from the times that I didn't know much about vectorization. Your version
looks good to my eyes, but it is little harder to read in general. Also I
don't know how would you debug this code. Sometimes I need to pause the
execution of scripts and step-by-step move through the lines and see how
values are changing in each iteration.

Lastly, this dccnConAmb is my CCN concentration normalized at ambient
pressure and temperature that I use to estimate C and k parameters from
power-law relationship using scipy's curve_fit() in case someone is curious
what I am after.


>
> Eric
>
>
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about numpy.ma masking

[Gökhan Sever]

On Fri, May 7, 2010 at 3:28 PM, Pierre GM  wrote:

> On May 4, 2010, at 8:38 PM, Gökhan Sever wrote:
> > Hello,
> >
> > I have the following arrays read as masked array.
> >
> > I[10]: basic.data['Air_Temp'].mask
> > O[10]: array([ True, False, False, ..., False, False, False], dtype=bool)
> >
> > [12]: basic.data['Press_Alt'].mask
> > O[12]: False
> >
> > I[13]: len basic.data['Air_Temp']
> > -> len(basic.data['Air_Temp'])
> > O[13]: 1758
> >
> >
> > The first item data['Air_Temp'] has only the first element masked and
> this result with mask attribute being created an equal data length bool
> array. On the other hand data['Press_Alt'] has no elements to mask yielding
> a 'False' scalar. Is this a documented behavior or intentionally designed
> this way? This is the only case out of 20 that breaks my code as following:
> :)
> >
> > IndexErrorTraceback (most recent call
> last)
> >
> > 130 for k in range(len(shorter)):
> > 131 if (serialh.data['dccnTempSF'][k] != 0) \
> > --> 132and (basic.data['Air_Temp'].mask[k+diff] == False):
> > 133 dccnConAmb[k] = serialc.data['dccnConc'][k] * \
> > 134 physical.data['STATIC_PR'][k+diff] * \
> >
> > IndexError: invalid index to scalar variable.
> >
> > since mask is a scalar in this case, nothing to loop terminating with an
> IndexError.
>
>
> Gokhan,
> Sorry for not getting back sooner, web connectivity was limited on my side.
> I must admit I can't really see what you're tring to do here, but I'll
> throw some random comments:
> * If you're using structured MaskedArrays, it's a really bad idea to call
> one of the fields "data", as it may interact in a non-obvious way with the
> actual "data" property (the one that outputs a view of the array as a pure
> ndarray).
>

Hello Pierre,

basic.data is a dictionary containing all masked array items. When I read
the original data into scripts, my main constructor-reader class
automatically converts data to masked arrays. basic.data['Air_Temp'] is a
masked array itself, little confusing for sure it also has 'data' attribute.


In the above example I check one condition looping in mask value. When mask
attribute isn't an bool-array (when there is no missing value in data) the
condition fails asserting an IndexError. I was wondering why it doesn't
yield a bool-array instead of giving me a scalar False.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Another masked array question

[Gökhan Sever]

On Sat, May 8, 2010 at 9:29 PM, Eric Firing  wrote:

> On 05/08/2010 04:16 PM, Ryan May wrote:
> > On Sat, May 8, 2010 at 7:52 PM, Gökhan Sever
>  wrote:
> >> Hello,
> >>
> >> Consider my masked arrays:
> >>
> >> I[28]: type basic.data['Air_Temp']
> >> ->  type(basic.data['Air_Temp'])
> >> O[28]: numpy.ma.core.MaskedArray
> >>
> >> I[29]: basic.data['Air_Temp']
> >> O[29]:
> >> masked_array(data = [-- -- -- ..., -- -- --],
> >>   mask = [ True  True  True ...,  True  True  True],
> >> fill_value = 99.)
> >>
> >>
> >> I[17]: basic.data['Air_Temp'].data =
> np.ones(len(basic.data['Air_Temp']))*30
> >>
> ---
> >> AttributeErrorTraceback (most recent call
> last)
> >>
> >> >  1
> >>2
> >>3
> >>4
> >>5
> >>
> >> AttributeError: can't set attribute
> >>
> >> Why this assignment fails? I want to set each element in the original
> >> basic.data['Air_Temp'].data to another value. (Because the main
> instrument
> >> was forgotten to turn on for that day, and I am using a secondary
> >> measurement data for Air Temperature for my another calculation. However
> it
> >> fails. Although single assignment works:
> >>
> >> I[13]: basic.data['Air_Temp'].data[0] = 30
> >>
> >> Shouldn't this be working like the regular NumPy arrays do?
> >
> > Based on the traceback, I'd say it's because you're trying to replace
> > the object pointed to by the .data attribute. Instead, try to just
> > change the bits contained in .data:
> >
> > basic.data['Air_Temp'].data[:] = np.ones(len(basic.data['Air_Temp']))*30
>
> Also, you since you are setting all elements to a single value, you
> don't need to generate an array on the right-hand side.  And, you don't
> need to manipulate ".data" directly--I think it is best to avoid doing
> so.  Consider:
>
> In [1]:x = np.ma.array([1,2,3], mask=[True, True, True], dtype=float)
>
> In [2]:x
> Out[2]:
> masked_array(data = [-- -- --],
>  mask = [ True  True  True],
>fill_value = 1e+20)
>
>
> In [3]:x[:] = 30
>
> In [4]:x
> Out[4]:
> masked_array(data = [30.0 30.0 30.0],
>  mask = [False False False],
>fill_value = 1e+20)
>
>
> In [5]:x[:] = np.ma.masked
>
> In [6]:x
> Out[6]:
> masked_array(data = [-- -- --],
>  mask = [ True  True  True],
>fill_value = 1e+20)
>
>
> In [7]:x.data
> Out[7]:array([ 30.,  30.,  30.])
>
>
> Eric
>
> >
> > Ryan
> >
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

Good to see this :)

I[45]: x = np.ma.array([1,2,3], mask=[True, True, True], dtype=float)

I[46]: x
O[46]:
masked_array(data = [-- -- --],
 mask = [ True  True  True],
   fill_value = 1e+20)


I[47]: x.data[:] = 25

I[48]: x
O[48]:
masked_array(data = [-- -- --],
 mask = [ True  True  True],
   fill_value = 1e+20)


I[49]: x[:] = 25

I[50]: x
O[50]:
masked_array(data = [25.0 25.0 25.0],
 mask = [False False False],
   fill_value = 1e+20)


I was also updating mask values after updating data attribute. Now setting
the masked array itself to a number automatically flips the masks for me
which is very useful. I check if a valid temperature exists, otherwise
assign my calculation to another missing value.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Another masked array question

[Gökhan Sever]

On Sat, May 8, 2010 at 9:16 PM, Ryan May  wrote:

> On Sat, May 8, 2010 at 7:52 PM, Gökhan Sever 
> wrote:
> > Hello,
> >
> > Consider my masked arrays:
> >
> > I[28]: type basic.data['Air_Temp']
> > -> type(basic.data['Air_Temp'])
> > O[28]: numpy.ma.core.MaskedArray
> >
> > I[29]: basic.data['Air_Temp']
> > O[29]:
> > masked_array(data = [-- -- -- ..., -- -- --],
> >  mask = [ True  True  True ...,  True  True  True],
> >fill_value = 99.)
> >
> >
> > I[17]: basic.data['Air_Temp'].data =
> np.ones(len(basic.data['Air_Temp']))*30
> >
> ---
> > AttributeErrorTraceback (most recent call
> last)
> >
> > > 1
> >   2
> >   3
> >   4
> >   5
> >
> > AttributeError: can't set attribute
> >
> > Why this assignment fails? I want to set each element in the original
> > basic.data['Air_Temp'].data to another value. (Because the main
> instrument
> > was forgotten to turn on for that day, and I am using a secondary
> > measurement data for Air Temperature for my another calculation. However
> it
> > fails. Although single assignment works:
> >
> > I[13]: basic.data['Air_Temp'].data[0] = 30
> >
> > Shouldn't this be working like the regular NumPy arrays do?
>
> Based on the traceback, I'd say it's because you're trying to replace
> the object pointed to by the .data attribute. Instead, try to just
> change the bits contained in .data:
>
> basic.data['Air_Temp'].data[:] = np.ones(len(basic.data['Air_Temp']))*30
>
> Ryan
>
> --
> Ryan May
> Graduate Research Assistant
> School of Meteorology
> University of Oklahoma
>

Thanks for the pointer Ryan. Now it works as it is supposed to be.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Another masked array question

[Gökhan Sever]

Hello,

Consider my masked arrays:

I[28]: type basic.data['Air_Temp']
-> type(basic.data['Air_Temp'])
O[28]: numpy.ma.core.MaskedArray

I[29]: basic.data['Air_Temp']
O[29]:
masked_array(data = [-- -- -- ..., -- -- --],
 mask = [ True  True  True ...,  True  True  True],
   fill_value = 99.)


I[17]: basic.data['Air_Temp'].data = np.ones(len(basic.data['Air_Temp']))*30
---
AttributeErrorTraceback (most recent call last)

> 1
  2
  3
  4
  5

AttributeError: can't set attribute

Why this assignment fails? I want to set each element in the original
basic.data['Air_Temp'].data to another value. (Because the main instrument
was forgotten to turn on for that day, and I am using a secondary
measurement data for Air Temperature for my another calculation. However it
fails. Although single assignment works:

I[13]: basic.data['Air_Temp'].data[0] = 30

Shouldn't this be working like the regular NumPy arrays do?

Thanks.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Question about numpy.ma masking

[Gökhan Sever]

Hello,

I have the following arrays read as masked array.

I[10]: basic.data['Air_Temp'].mask
O[10]: array([ True, False, False, ..., False, False, False], dtype=bool)

[12]: basic.data['Press_Alt'].mask
O[12]: False

I[13]: len basic.data['Air_Temp']
-> len(basic.data['Air_Temp'])
O[13]: 1758


The first item data['Air_Temp'] has only the first element masked and this
result with mask attribute being created an equal data length bool array. On
the other hand data['Press_Alt'] has no elements to mask yielding a 'False'
scalar. Is this a documented behavior or intentionally designed this way?
This is the only case out of 20 that breaks my code as following: :)

IndexErrorTraceback (most recent call last)

130 for k in range(len(shorter)):
131 if (serialh.data['dccnTempSF'][k] != 0) \
--> 132and (basic.data['Air_Temp'].mask[k+diff] == False):
133 dccnConAmb[k] = serialc.data['dccnConc'][k] * \
134 physical.data['STATIC_PR'][k+diff] * \

IndexError: invalid index to scalar variable.

since mask is a scalar in this case, nothing to loop terminating with an
IndexError.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about numpy.arange()

[Gökhan Sever]

On Sat, May 1, 2010 at 3:36 PM, Gökhan Sever  wrote:

> Hello,
>
> Is "b" an expected value? I am suspecting another floating point arithmetic
> issue.
>
> I[1]: a = np.arange(1.6, 1.8, 0.1, dtype='float32')
>
> I[2]: a
> O[2]: array([ 1.6002,  1.7005], dtype=float32)
>
> I[3]: b = np.arange(1.7, 1.8, 0.1, dtype='float32')
>
> I[4]: b
> O[4]: array([ 1.7005,  1.7995], dtype=float32)
>
> A bit conflicting with the np.arange docstring:
>
> "*   Values are generated within the half-open interval ``[start, stop)``
> (in other words, the interval including `start` but excluding `stop`).
> *"
>
> Thanks.
>
> --
> Gökhan
>

Fair enough explanations to use np.linspace instead. What was confusing me
above while a[1] and b[0] shows 1.7005 only "b" steps up to 1.7995
which "a" can't.

The following is another surprise output:

I[5]: c = np.arange(0.4, 0.5, 0.1, dtype='float32')

I[6]: c
O[6]: array([ 0.4001], dtype=float32)

Anyways, a Slashdotter might have seen me asking these questions. I will go
read What Every Programmer Should Know About Floating-Point
Arithmetic<http://developers.slashdot.org/story/10/05/02/1427214/What-Every-Programmer-Should-Know-About-Floating-Point-Arithmetic>article.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Question about numpy.arange()

[Gökhan Sever]

Hello,

Is "b" an expected value? I am suspecting another floating point arithmetic
issue.

I[1]: a = np.arange(1.6, 1.8, 0.1, dtype='float32')

I[2]: a
O[2]: array([ 1.6002,  1.7005], dtype=float32)

I[3]: b = np.arange(1.7, 1.8, 0.1, dtype='float32')

I[4]: b
O[4]: array([ 1.7005,  1.7995], dtype=float32)

A bit conflicting with the np.arange docstring:

"*   Values are generated within the half-open interval ``[start, stop)``
(in other words, the interval including `start` but excluding `stop`). *
"

Thanks.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Find indices of largest elements

[Gökhan Sever]

On Wed, Apr 14, 2010 at 10:16 AM, Nikolaus Rath  wrote:

> Hello,
>
> How do I best find out the indices of the largest x elements in an
> array?
>
> Example:
>
> a = [ [1,8,2], [2,1,3] ]
> magic_function(a, 2) == [ (0,1), (1,2) ]
>
> Since the largest 2 elements are at positions (0,1) and (1,2).
>
> I[1]: a = np.array([ [1,8,2], [2,1,3] ])

I[2]: b = a.max(axis=1)[:,np.newaxis]

I[3]: b
O[3]:
array([[8],
   [3]])

I[4]: np.where(a==b)
O[4]: (array([0, 1]), array([1, 2]))



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] how to tally the values seen

[Gökhan Sever]

On Wed, Apr 14, 2010 at 1:34 AM, Warren Weckesser <
warren.weckes...@enthought.com> wrote:

> Gökhan Sever wrote:
> >
> >
> > On Wed, Apr 14, 2010 at 1:10 AM, Peter Shinners  > <mailto:p...@shinners.org>> wrote:
> >
> > I have an array that represents the number of times a value has been
> > given. I'm trying to find a direct numpy way to add into these sums
> > without requiring a Python loop.
> >
> > For example, say there are 10 possible values. I start with an
> > array of
> > zeros.
> >
> >  >>> counts = numpy.zeros(10, numpy.int <http://numpy.int>)
> >
> > Now I get an array with several values in them, I want to add into
> > counts. All I can think of is a for loop that will give my the
> > results I
> > want.
> >
> >
> >  >>> values = numpy.array((2, 8, 1))
> >  >>> for v in values:
> > ...counts[v] += 1
> >  >>> print counts
> > [0 1 1 0 0 0 0 0 1 0]
> >
> >
> > This is easy:
> >
> > I[3]: a
> > O[3]: array([ 0.,  1.,  1.,  0.,  0.,  0.,  0.,  0.,  1.,  0.])
> >
> > I[4]: a = np.zeros(10)
> >
> > I[5]: b = np.array((2,8,1))
> >
> > I[6]: a[b] = 1
> >
> > I[7]: a
> > O[7]: array([ 0.,  1.,  1.,  0.,  0.,  0.,  0.,  0.,  1.,  0.])
> >
> > Let me think about the other case :)
> >
> >
> > I also need to handle the case where a value is listed more than
> once.
> > So if values is (2, 8, 1, 2) then count[2] would equal 2.
> >
>
>
> numpy.bincount():
>
>
> In [1]: import numpy as np
>
> In [2]: x = np.array([2,8,1,2,7,7,2,7,0,2])
>
> In [3]: np.bincount(x)
> Out[3]: array([1, 1, 4, 0, 0, 0, 0, 3, 1])
>
>
>
I knew a function exists in numpy for this case too :)

This is also safer way to handle the given situation to prevent index out of
bounds errors.



>
> Warren
>
> > What is the most efficient way to do this?
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org>
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> >
> >
> > --
> > Gökhan
> > 
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] how to tally the values seen

[Gökhan Sever]

On Wed, Apr 14, 2010 at 1:10 AM, Peter Shinners  wrote:

> I have an array that represents the number of times a value has been
> given. I'm trying to find a direct numpy way to add into these sums
> without requiring a Python loop.
>
> For example, say there are 10 possible values. I start with an array of
> zeros.
>
>  >>> counts = numpy.zeros(10, numpy.int)
>
> Now I get an array with several values in them, I want to add into
> counts. All I can think of is a for loop that will give my the results I
> want.
>
>
>  >>> values = numpy.array((2, 8, 1))
>  >>> for v in values:
> ...counts[v] += 1
>  >>> print counts
> [0 1 1 0 0 0 0 0 1 0]
>

This is easy:

I[3]: a
O[3]: array([ 0.,  1.,  1.,  0.,  0.,  0.,  0.,  0.,  1.,  0.])

I[4]: a = np.zeros(10)

I[5]: b = np.array((2,8,1))

I[6]: a[b] = 1

I[7]: a
O[7]: array([ 0.,  1.,  1.,  0.,  0.,  0.,  0.,  0.,  1.,  0.])

Let me think about the other case :)


> I also need to handle the case where a value is listed more than once.
> So if values is (2, 8, 1, 2) then count[2] would equal 2.
>
> What is the most efficient way to do this?
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] indexing to sort with argsort(..., axis=1)

[Gökhan Sever]

On Mon, Apr 12, 2010 at 9:41 PM, Angus McMorland  wrote:

> Hi all,
>
> I want to sort a 2d array along one dimension, with the indices returned by
> argsort, but the subsequent indexing syntax to get the sorted array is not
> obvious.
>
> The following works, but I wonder if there is a simpler way:
>
> a = np.random.random(size=(5,3))
> s = np.argsort(a, axis=1)
> sorted = a[:,s][np.eye(5,5, dtype=bool)] # it looks like this line could be
> simpler
>
> What's the correct, concise way to do this?
>

Why not just:

b = np.sort(a)

What advantage does argsort provides in this case?



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Sum over array elements

[Gökhan Sever]

On Mon, Apr 12, 2010 at 9:21 AM, Nicola Creati  wrote:

> Hello,
> I want to calculate, given a one dimension array, the sum over every two
> elements of the array.
> I found this working solution:
>
> a = N.arange(10)
> b = a.reshape(a, (5, 2))
> c = b.sum(axis=1)
>
> Is there any better solution?
>
> Thanks,
>
> Nicola Creati
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

Sum odds and evens:

I[3]: d = a[1::2] + a[::2]

I[4]: d
O[4]: array([ 1,  5,  9, 13, 17])

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Simple way to shift array elements

[Gökhan Sever]

On Sat, Apr 10, 2010 at 7:31 PM, Charles R Harris  wrote:

>
>
> On Sat, Apr 10, 2010 at 6:17 PM, Gökhan Sever wrote:
>
>> Hello,
>>
>> Is there a simpler way to get "c" from "a"
>>
>> I[1]: a = np.arange(10)
>>
>> I[2]: b = a[3:]
>>
>> I[3]: b
>> O[3]: array([3, 4, 5, 6, 7, 8, 9])
>>
>> I[4]: c = np.insert(b, [7]*3, 0)
>> O[5]: array([3, 4, 5, 6, 7, 8, 9, 0, 0, 0])
>>
>> a and c have to be same in length and the left shift must be balanced with
>> equal number of 0's
>>
>>
> Maybe something like
>
> In [1]: a = np.arange(10)
>
> In [2]: b = zeros_like(a)
>
> In [3]: b[:-3] = a[3:]
>
> In [4]: b
> Out[4]: array([3, 4, 5, 6, 7, 8, 9, 0, 0, 0])
>
> Chuck
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
Thanks,

Your ways are more obvious than my first approach. With a bit more playing I
get a one-liner:

c=np.append(a[3:], [0]*3)

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Simple way to shift array elements

[Gökhan Sever]

Hello,

Is there a simpler way to get "c" from "a"

I[1]: a = np.arange(10)

I[2]: b = a[3:]

I[3]: b
O[3]: array([3, 4, 5, 6, 7, 8, 9])

I[4]: c = np.insert(b, [7]*3, 0)
O[5]: array([3, 4, 5, 6, 7, 8, 9, 0, 0, 0])

a and c have to be same in length and the left shift must be balanced with
equal number of 0's

Thanks.


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] lists of zeros and ones

[Gökhan Sever]

On Fri, Mar 19, 2010 at 10:17 AM, Joe Kington  wrote:

> See itertools.permutations (python standard library)
>
> e.g.
> In [3]: list(itertools.permutations([1,1,0,0]))
> Out[3]:
> [(1, 1, 0, 0),
>  (1, 1, 0, 0),
>  (1, 0, 1, 0),
>  (1, 0, 0, 1),
>  (1, 0, 1, 0),
>  (1, 0, 0, 1),
>  (1, 1, 0, 0),
>  (1, 1, 0, 0),
>  (1, 0, 1, 0),
>  (1, 0, 0, 1),
>  (1, 0, 1, 0),
>  (1, 0, 0, 1),
>  (0, 1, 1, 0),
>  (0, 1, 0, 1),
>  (0, 1, 1, 0),
>  (0, 1, 0, 1),
>  (0, 0, 1, 1),
>  (0, 0, 1, 1),
>  (0, 1, 1, 0),
>  (0, 1, 0, 1),
>  (0, 1, 1, 0),
>  (0, 1, 0, 1),
>
>
>
>  (0, 0, 1, 1),
>  (0, 0, 1, 1)]
>
> Hope that helps,
> -Joe
>
>

If you use "set" you automatically eliminate replicates:

a = set(permutations([0,0,1,1]))

a
set([(0, 0, 1, 1),
 (0, 1, 0, 1),
 (0, 1, 1, 0),
 (1, 0, 0, 1),
 (1, 0, 1, 0),
 (1, 1, 0, 0)])

Converting back to a list:

b = list(a)
-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ask.scipy.org is online

[Gökhan Sever]

Hello,

Please tolerate my impatience for being the first announcing the new
discussion platform :) and my cross-posting over the lists.

The new site is beating at ask.scipy.org . David Warde-Farley is moving the
questions from the old-site at advice.mechanicalkern.com (announced at
SciPy09 by Robert Kern)

We welcome you to join the discussions there. I have kicked-off the new
questions chain by
http://ask.scipy.org/en/topic/11-what-is-your-favorite-numpy-feature
(Also cross-posted at
http://stackoverflow.com/questions/2471872/what-is-your-favorite-numpy-featureto
pull more attention to ask.scipy)

Please share your questions and comments, and have fun with your
discussions.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] printing structured arrays

[Gökhan Sever]

On Fri, Mar 5, 2010 at 8:00 AM, Bruce Schultz wrote:

>  Hi,
>
> I've just started playing with numpy and have noticed that when printing a
> structured array that the output is not nicely formatted. Is there a way to
> make the formatting look the same as it does for an unstructured array?
>
> Here an example of what I mean:
>
> data = [ (1, 2), (3, 4.1) ]
> dtype = [('x', float), ('y', float)]
> print '### ndarray'
> a = numpy.array(data)
> print a
> print '### structured array'
> a = numpy.array(data, dtype=dtype)
> print a
>
> Output is:
> ### ndarray
> [[ 1.   2. ]
>  [ 3.   4.1]]
> ### structured array
> [(1.0, 2.0) (3.0, 4.0996)]
>
>
> Thanks
> Bruce
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
I still couldn't figure out how floating point numbers look nicely on screen
in cases like yours (i.e., trying numpy.array2string()) but you can make
sure by using numpy.savetxt("file", array, fmt="%.1f") you will always have
specified precision in the written file.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] For-less code

[Gökhan Sever]

Hello,

I am working on a code shown at
http://code.google.com/p/ccnworks/source/browse/trunk/thesis/part1/logn-fit.py

I use the code to analyse a couple dataset also placed in the same
directory. In the first part I use for-loops all over, but later decided to
write them without using for loops. The script runs correctly for the two
section, however I have a few question regarding to the for-less structures:

This part is taken after line 371:

gm10 = np.exp((1/Nt10)*(h10*np.log(Dp)).sum(axis=1))
gsd10 =
np.exp(((1/Nt10)*(h10*np.log(Dp/gm10[:,np.newaxis])**2).sum(axis=-1))**0.5)

dN_dDp10 =
(Nt10[:,np.newaxis]/((2*np.pi)**0.5*np.log(gsd10[:,np.newaxis])*d))*np.exp(-(np.log(d)-\

np.log(gm10[:,np.newaxis]))**2/(2*np.log(gsd10[:,np.newaxis])**2))

a10 = (dN_dDp10[0:300,d >= dc10u]*0.001).sum(axis=1)

Shape informations for the arrays as follow:

I[306]: gm10.shape; gsd10.shape, Dp.shape, d.shape, dN_dDp10.shape,
a10.shape
O[306]: (300,), (300,), (11,), (991,), (300, 991), (300,)

1-) In gsd10 line what does axis=-1 really means, and why negative axis
value is allowed?

2-) [:,np.newaxis] decreases the readability of the code. Is there any
alternative for it?

3-) In the last line (dN_dDp10[0:300,d >= dc10u]) could I achieve the same
result with different syntax? I have found it somewhat not in accordance
with the previous lines.

Thanks for your time and comments.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ask.scipy.org

[Gökhan Sever]

On Sun, Feb 21, 2010 at 4:06 PM, Robert Kern  wrote:

> On Sun, Feb 21, 2010 at 16:00, Gökhan Sever  wrote:
> > Hello,
> >
> > Since after Robert Kern showed http://advice.mechanicalkern.com/ on
> SciPy09
> > there are many similar initiatives that uses stackoverflow.com (SO)
> layout.
> > Some smart guys come up with this site http://stackexchange.com/ to
> those
> > who want to have a simple but a paid solution.
>
> Indeed, stackexchange.com is the paid hosting option from the Stack
> Overflow team.
>
> > I don't have an intention of creating controversial discussion. It just
> to
> > my eyes SO has a very appealing and easy to use interface and it's
> getting
> > some number of posts related to scientific Python tools. I usually
> suggest
> > my friends to use the mailing lists first and SO for their questions.
> Some
> > prefer mailing lists some not. Mailing lists require more steps to get in
> > however SO register step is much easier due to OpenID logging.
> >
> > Without belabouring further, It would be good to link R. Kern's advice
> site
> > to either ask.scipy or advice.scipy or another alternative to attract
> > new-comers easily. I am more in favor of the ask.scipy.org option. Thus
> I
> > can refer the people (hear I mean mostly non-programmers or
> > students/programmers without Python experience), simply to go
> ask.scipy.org
> > for their first questions instead of telling them to search answers at
> many
> > different mediums.
> >
> > What do you think?
>
> I spent some time on Friday getting Plurk's Solace tweaked for our use
> (for various reasons, it's much better code to deal with than the
> CNPROG software currently running advice.mechanicalkern.com).
>
>  http://opensource.plurk.com/Solace/
>
> I still need to investigate how to migrate the content from the old
> site over, but ask.scipy.org should be up and running quite soon.
>
>
Thanks for your efforts Robert.

Please let us know when the new site is up.



> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ask.scipy.org

[Gökhan Sever]

Hello,

Since after Robert Kern showed http://advice.mechanicalkern.com/ on SciPy09
there are many similar initiatives that uses stackoverflow.com (SO) layout.
Some smart guys come up with this site http://stackexchange.com/ to those
who want to have a simple but a paid solution.

I don't have an intention of creating controversial discussion. It just to
my eyes SO has a very appealing and easy to use interface and it's getting
some number of posts related to scientific Python tools. I usually suggest
my friends to use the mailing lists first and SO for their questions. Some
prefer mailing lists some not. Mailing lists require more steps to get in
however SO register step is much easier due to OpenID logging.

Without belabouring further, It would be good to link R. Kern's advice site
to either ask.scipy or advice.scipy or another alternative to attract
new-comers easily. I am more in favor of the ask.scipy.org option. Thus I
can refer the people (hear I mean mostly non-programmers or
students/programmers without Python experience), simply to go
ask.scipy.orgfor their first questions instead of telling them to
search answers at many
different mediums.

What do you think?

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Request for testing

[Gökhan Sever]

On Sun, Feb 21, 2010 at 4:30 AM, Charles R Harris  wrote:

> Hi All,
>
> I would be much obliged if some folks would run the attached script and
> report the output, numpy version, and python version. It just runs
> np.isinf(np.inf), which raises an "invalid value" warning with current
> numpy. As far as I can see the function itself hasn't changed since
> numpy1.3, yet numpy1.3 & python2.5 gives no such warning.
>
> Chuck
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
[gse...@ccn ~]$ python isinf.py
True

[gse...@ccn various]$ python sysinfo.py

Platform :
Linux-2.6.31.9-174.fc12.i686.PAE-i686-with-fedora-12-Constantine
Python   : ('CPython', 'tags/r262', '71600')
NumPy   : 1.5.0.dev8038



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Syntax equivalent for np.array()

[Gökhan Sever]

On Wed, Feb 10, 2010 at 10:12 AM, Gökhan Sever wrote:

>
>
> On Wed, Feb 10, 2010 at 10:06 AM, Angus McMorland wrote:
>
>> On 10 February 2010 11:02, Gökhan Sever  wrote:
>> > Hi,
>> >
>> > Simple question:
>> >
>> > I[4]: a = np.arange(10)
>> >
>> > I[5]: b = np.array(5)
>> >
>> > I[8]: a*b.cumsum()
>> > O[8]: array([ 0,  5, 10, 15, 20, 25, 30, 35, 40, 45])
>> >
>> > I[9]: np.array(a*b).cumsum()
>> > O[9]: array([  0,   5,  15,  30,  50,  75, 105, 140, 180, 225])
>> >
>> > Is there a syntactic equivalent for the I[9] --for instance instead of
>> using
>> > "list" keyword I use [ ] while creating a list. Is there a shortcut for
>> > np.array instead of writing np.array(a*b) explicitly?
>>
>> How about just (a*b).cumsum() ?
>>
>> Angus.
>> --
>> AJC McMorland
>> Post-doctoral research fellow
>> Neurobiology, University of Pittsburgh
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> Yep that's it :) I knew that it was a very simple question.
>
> What confused me is I remember somewhere not sure maybe in IPython dev I
> have gotten when I do:
>
> (a*b).cumsum()
>
> AttributeError: 'tuple' object has no attribute 'cumsum' error.
>
> So I was thinking ( ) is a ssugar for tuple and np.array might have
> something special than these.
>
> --
> Gökhan
>

Self-correction:

It works correctly in IPython-dev as well.

And further in Python 2.6.2:

>>> p = ()
>>> p
()
>>> type(p)

>>> type((a*b))


( ) doesn't only works as a tuple operator. It also has its original
parenthesis functionality :)

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Syntax equivalent for np.array()

[Gökhan Sever]

On Wed, Feb 10, 2010 at 10:06 AM, Angus McMorland  wrote:

> On 10 February 2010 11:02, Gökhan Sever  wrote:
> > Hi,
> >
> > Simple question:
> >
> > I[4]: a = np.arange(10)
> >
> > I[5]: b = np.array(5)
> >
> > I[8]: a*b.cumsum()
> > O[8]: array([ 0,  5, 10, 15, 20, 25, 30, 35, 40, 45])
> >
> > I[9]: np.array(a*b).cumsum()
> > O[9]: array([  0,   5,  15,  30,  50,  75, 105, 140, 180, 225])
> >
> > Is there a syntactic equivalent for the I[9] --for instance instead of
> using
> > "list" keyword I use [ ] while creating a list. Is there a shortcut for
> > np.array instead of writing np.array(a*b) explicitly?
>
> How about just (a*b).cumsum() ?
>
> Angus.
> --
> AJC McMorland
> Post-doctoral research fellow
> Neurobiology, University of Pittsburgh
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

Yep that's it :) I knew that it was a very simple question.

What confused me is I remember somewhere not sure maybe in IPython dev I
have gotten when I do:

(a*b).cumsum()

AttributeError: 'tuple' object has no attribute 'cumsum' error.

So I was thinking ( ) is a ssugar for tuple and np.array might have
something special than these.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Syntax equivalent for np.array()

[Gökhan Sever]

Hi,

Simple question:

I[4]: a = np.arange(10)

I[5]: b = np.array(5)

I[8]: a*b.cumsum()
O[8]: array([ 0,  5, 10, 15, 20, 25, 30, 35, 40, 45])

I[9]: np.array(a*b).cumsum()
O[9]: array([  0,   5,  15,  30,  50,  75, 105, 140, 180, 225])

Is there a syntactic equivalent for the I[9] --for instance instead of using
"list" keyword I use [ ] while creating a list. Is there a shortcut for
np.array instead of writing np.array(a*b) explicitly?

Thanks.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Interact with matplotlib in Sage

[Gökhan Sever]

Hello,

I have thought of this might interesting to share. Register at
www.sagenb.org or try on your local Sage-notebook and using the following
code:

# Simple example demonstrating how to interact with matplotlib directly.
# Comment plt.clf() to get the plots overlay in each update.
# Gokhan Sever & Harald Schilly (2010-01-24)

from scipy import stats
import numpy as np
import matplotlib.pyplot as plt

@interact
def plot_norm(loc=(0,(0,10)), scale=(1,(1,10))):
rv = stats.norm(loc, scale)
x = np.linspace(-10,10,1000)
plt.plot(x,rv.pdf(x))
plt.grid(True)
plt.savefig('plt.png')
plt.clf()

A very easy to use example, also well-suited for learning and demonstration
purposes.

Posted at: http://wiki.sagemath.org/interact/graphics#Interactwithmatplotlib

Have fun ;)

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

[Gökhan Sever]

On Mon, Dec 28, 2009 at 12:15 PM, Gökhan Sever wrote:

>
>
> On Mon, Dec 28, 2009 at 11:16 AM, Gökhan Sever wrote:
>
>>
>>
>> On Mon, Dec 28, 2009 at 11:07 AM, Robert Kern wrote:
>>
>>> On Mon, Dec 28, 2009 at 11:00, Gökhan Sever 
>>> wrote:
>>>
>>> > One interesting thing I have noticed while installing the numpy from
>>> the
>>> > source is that numpy dependent libraries must be re-installed and this
>>> must
>>> > be a clean re-install. For instance I can't import some matplotlib and
>>> scipy
>>> > modules without making a fresh installation for these packages. My
>>> attempts
>>> > result with a runtime error.
>>>
>>> Please, please, always copy-and-paste the traceback when reporting an
>>> error. I know you aren't formally reporting a bug here, but it always
>>> helps.
>>>
>>> > Could someone clarify this point? Is this due
>>> > to API change in the numpy core?
>>>
>>> Cython/Pyrex code does a runtime check on the struct sizes of types.
>>> We have carefully added a member to the PyArrayDescr struct; i.e. it
>>> shouldn't cause any actual problems, but Cython does the check
>>> anyways. This affects a few modules in scipy, but shouldn't have
>>> affected anything in matplotlib. The traceback may help us identify
>>> the issue you are seeing.
>>>
>>>
>> It is too late for the tracebacks. I have already removed the problematic
>> packages and did clean installs. However, next time I will be more careful
>> while reporting such issues. If it helps, in both matplotlib (via ipython
>> -pylab) and scipy.stats import cases the runtime errors was raised due to
>> numpy.core.multiarray module import.
>>
>>
>>
>>> --
>>>
>>> Robert Kern
>>>
>>> "I have come to believe that the whole world is an enigma, a harmless
>>> enigma that is made terrible by our own mad attempt to interpret it as
>>> though it had an underlying truth."
>>>  -- Umberto Eco
>>> ___
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>>
>> --
>> Gökhan
>>
>
> Here is another interesting point to consider. To reproduce the runtime
> error I mentioned previously I downgraded numpy (from the latest check-out
> installation) following these steps:
>
> svn co http://svn.scipy.org/svn/numpy/branches/1.3.x/ numpy
> cd numpy
> python setup.py install
> Writing /usr/lib/python2.6/site-packages/numpy-1.3.1.dev8031-py2.6.egg-info
>
> I[2]: import matplotlib.pyplot as plt
> Segmentation fault
>
> I[3]: from scipy import stats
> Segmentation fault
>
> I have installed matplotlib and scipy using the latest numpy dev version. A
> little later I will downgrade matplotlib and scipy to their previous stable
> versions, and compile them using numpy 1.3.x. Afterwards I will update numpy
> and test to see if I can re-produce the runtime error to provide the
> tracebacks. First let me know if any tracebacks needed for these segfaults
> or are these known failures?
>
>
> 
> Platform :
> Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
> Python   : ('CPython', 'tags/r26', '66714')
>
> 
>
>
> --
> Gökhan
>


Since no one has replied, I tried to reproduce the runtime error but ended
up with different errors. Read more for the details:

svn co http://svn.scipy.org/svn/scipy/tags/0.7.1/ scipy
python setup.py install
Writing /usr/lib/python2.6/site-packages/scipy-0.7.1-py2.6.egg-info

svn co
https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/tags/v0_99_0/matplotlib
python setup.py install
Writing /usr/lib/python2.6/site-packages/matplotlib-0.99.0-py2.6.egg-info

install them using

>>> import numpy
>>> numpy.__version__
'1.3.1.dev8031'

scipy import is fine but matplotlib fails with a different import error. I
wonder if the buildbots test the source and releases for against different
numpy versions or it is just might system acting weird.

>>> from scipy import stats

>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
  File "", line 1, in 
  File
"/home/gsever/Desktop/python-repo/matplotlib/lib/matplotlib/

Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

[Gökhan Sever]

On Mon, Dec 28, 2009 at 11:16 AM, Gökhan Sever wrote:

>
>
> On Mon, Dec 28, 2009 at 11:07 AM, Robert Kern wrote:
>
>> On Mon, Dec 28, 2009 at 11:00, Gökhan Sever 
>> wrote:
>>
>> > One interesting thing I have noticed while installing the numpy from the
>> > source is that numpy dependent libraries must be re-installed and this
>> must
>> > be a clean re-install. For instance I can't import some matplotlib and
>> scipy
>> > modules without making a fresh installation for these packages. My
>> attempts
>> > result with a runtime error.
>>
>> Please, please, always copy-and-paste the traceback when reporting an
>> error. I know you aren't formally reporting a bug here, but it always
>> helps.
>>
>> > Could someone clarify this point? Is this due
>> > to API change in the numpy core?
>>
>> Cython/Pyrex code does a runtime check on the struct sizes of types.
>> We have carefully added a member to the PyArrayDescr struct; i.e. it
>> shouldn't cause any actual problems, but Cython does the check
>> anyways. This affects a few modules in scipy, but shouldn't have
>> affected anything in matplotlib. The traceback may help us identify
>> the issue you are seeing.
>>
>>
> It is too late for the tracebacks. I have already removed the problematic
> packages and did clean installs. However, next time I will be more careful
> while reporting such issues. If it helps, in both matplotlib (via ipython
> -pylab) and scipy.stats import cases the runtime errors was raised due to
> numpy.core.multiarray module import.
>
>
>
>> --
>>
>> Robert Kern
>>
>> "I have come to believe that the whole world is an enigma, a harmless
>> enigma that is made terrible by our own mad attempt to interpret it as
>> though it had an underlying truth."
>>  -- Umberto Eco
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
>
> --
> Gökhan
>

Here is another interesting point to consider. To reproduce the runtime
error I mentioned previously I downgraded numpy (from the latest check-out
installation) following these steps:

svn co http://svn.scipy.org/svn/numpy/branches/1.3.x/ numpy
cd numpy
python setup.py install
Writing /usr/lib/python2.6/site-packages/numpy-1.3.1.dev8031-py2.6.egg-info

I[2]: import matplotlib.pyplot as plt
Segmentation fault

I[3]: from scipy import stats
Segmentation fault

I have installed matplotlib and scipy using the latest numpy dev version. A
little later I will downgrade matplotlib and scipy to their previous stable
versions, and compile them using numpy 1.3.x. Afterwards I will update numpy
and test to see if I can re-produce the runtime error to provide the
tracebacks. First let me know if any tracebacks needed for these segfaults
or are these known failures?


Platform :
Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
Python   : ('CPython', 'tags/r26', '66714')



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

[Gökhan Sever]

On Mon, Dec 28, 2009 at 11:07 AM, Robert Kern  wrote:

> On Mon, Dec 28, 2009 at 11:00, Gökhan Sever  wrote:
>
> > One interesting thing I have noticed while installing the numpy from the
> > source is that numpy dependent libraries must be re-installed and this
> must
> > be a clean re-install. For instance I can't import some matplotlib and
> scipy
> > modules without making a fresh installation for these packages. My
> attempts
> > result with a runtime error.
>
> Please, please, always copy-and-paste the traceback when reporting an
> error. I know you aren't formally reporting a bug here, but it always
> helps.
>
> > Could someone clarify this point? Is this due
> > to API change in the numpy core?
>
> Cython/Pyrex code does a runtime check on the struct sizes of types.
> We have carefully added a member to the PyArrayDescr struct; i.e. it
> shouldn't cause any actual problems, but Cython does the check
> anyways. This affects a few modules in scipy, but shouldn't have
> affected anything in matplotlib. The traceback may help us identify
> the issue you are seeing.
>
>
It is too late for the tracebacks. I have already removed the problematic
packages and did clean installs. However, next time I will be more careful
while reporting such issues. If it helps, in both matplotlib (via ipython
-pylab) and scipy.stats import cases the runtime errors was raised due to
numpy.core.multiarray module import.



> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

[Gökhan Sever]

On Mon, Dec 28, 2009 at 10:31 AM, Gökhan Sever wrote:

>
>
> On Sat, Dec 26, 2009 at 6:09 PM, David Cournapeau wrote:
>
>> On Sun, Dec 27, 2009 at 6:19 AM, Gökhan Sever 
>> wrote:
>>
>> >
>> > For the develop, it is one of easiest ways to catch up the bug-fixes
>> even
>> > though I don't work on the source directly. So far besides a few
>> glitches it
>> > was always working. I also install scipy, ipython, matplotlib, sympy and
>> all
>> > other available packages using develop. Keep the checkouts in the
>> directory
>> > on my desktop and if/when necessary do svn up or whichever command it
>> > corresponds to their respective vcs.
>>
>> If you do that, you have to be ready to look into the corresponding
>> issues it brings.
>>
>> > I wonder how other people keep up the
>> > changes easily without using develop option.
>>
>> I just install things, and avoid relying on too many developed
>> versions of packages.
>>
>> David
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> Fix comes following your suggestion. Use python install for the time being
> and remove the the check-out after installation. This prevents the funny
> import error even if I don't try to import the numpy from within I made the
> installation.
>
> --
> Gökhan
>

One interesting thing I have noticed while installing the numpy from the
source is that numpy dependent libraries must be re-installed and this must
be a clean re-install. For instance I can't import some matplotlib and scipy
modules without making a fresh installation for these packages. My attempts
result with a runtime error. Could someone clarify this point? Is this due
to API change in the numpy core?



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

[Gökhan Sever]

On Sat, Dec 26, 2009 at 6:09 PM, David Cournapeau wrote:

> On Sun, Dec 27, 2009 at 6:19 AM, Gökhan Sever 
> wrote:
>
> >
> > For the develop, it is one of easiest ways to catch up the bug-fixes even
> > though I don't work on the source directly. So far besides a few glitches
> it
> > was always working. I also install scipy, ipython, matplotlib, sympy and
> all
> > other available packages using develop. Keep the checkouts in the
> directory
> > on my desktop and if/when necessary do svn up or whichever command it
> > corresponds to their respective vcs.
>
> If you do that, you have to be ready to look into the corresponding
> issues it brings.
>
> > I wonder how other people keep up the
> > changes easily without using develop option.
>
> I just install things, and avoid relying on too many developed
> versions of packages.
>
> David
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

Fix comes following your suggestion. Use python install for the time being
and remove the the check-out after installation. This prevents the funny
import error even if I don't try to import the numpy from within I made the
installation.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

[Gökhan Sever]

On Sat, Dec 26, 2009 at 4:15 PM, Charles R Harris  wrote:

>
>
> On Sat, Dec 26, 2009 at 2:19 PM, Gökhan Sever wrote:
>
>>
>>
>> On Thu, Dec 24, 2009 at 4:57 PM, David Cournapeau wrote:
>>
>>> On Wed, Dec 23, 2009 at 1:41 AM, Gökhan Sever 
>>> wrote:
>>> >
>>> >
>>> > On Tue, Dec 22, 2009 at 9:05 AM, David Cournapeau 
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> I have just released the 2nd release candidate for numpy 1.4.0, which
>>> >> fixes a few critical bugs founds since the RC1. Tarballs and binary
>>> >> installers for numpy/scipy may be found on
>>> >> https://sourceforge.net/projects/numpy.
>>> >>
>>> >> cheers,
>>> >>
>>> >> David
>>> >> ___
>>> >> NumPy-Discussion mailing list
>>> >> NumPy-Discussion@scipy.org
>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >
>>> > This release results with the same import error on my system that I
>>> posted
>>> > on
>>>
>>> Don't use develop, and install numpy normally, but from scratch.
>>> Develop mode has some quircks, and it does not worth it unless you
>>> want to work on numpy code yourself IMHO,
>>>
>>
>> OK, a clean svn check-out and python setup.py install I get another
>> interesting import error:
>>
>> [gse...@ccn ~]$ pwd
>> /home/gsever
>> [gse...@ccn ~]$ python
>>
>> Python 2.6 (r26:66714, Jun  8 2009, 16:07:26)
>> [GCC 4.4.0 20090506 (Red Hat 4.4.0-4)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import numpy
>> Traceback (most recent call last):
>>   File "", line 1, in 
>>   File "/home/gsever/Desktop/python-repo/numpy/numpy/__init__.py", line
>> 123, in 
>> raise ImportError(msg)
>> ImportError: Error importing numpy: you should not try to import numpy
>> from
>> its source directory; please exit the numpy source tree, and
>> relaunch
>> your python intepreter from there.
>> >>>
>>
>> I launch the interpreter from a different directory than where the sources
>> located, however it still complains.
>>
>> For the develop, it is one of easiest ways to catch up the bug-fixes even
>> though I don't work on the source directly. So far besides a few glitches it
>> was always working. I also install scipy, ipython, matplotlib, sympy and all
>> other available packages using develop. Keep the checkouts in the directory
>> on my desktop and if/when necessary do svn up or whichever command it
>> corresponds to their respective vcs. I wonder how other people keep up the
>> changes easily without using develop option.
>>
>>
> I never see any of these problems and apparently no one else does either.
> There is something unique about your system. What does os.getcwd() return?
>
> Chuck
>
>
I removed numpy.egg-link (a remnant from setupegg.py develop) file under
/usr/lib/python2.6/site-packages. Still I get the same error. Your query
returns the same as pwd command:

>>> import os
>>> os.getcwd()
'/home/gsever/Desktop'
>>> exit()
[gse...@ccn Desktop]$ pwd
/home/gsever/Desktop




>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

[Gökhan Sever]

On Thu, Dec 24, 2009 at 4:57 PM, David Cournapeau wrote:

> On Wed, Dec 23, 2009 at 1:41 AM, Gökhan Sever 
> wrote:
> >
> >
> > On Tue, Dec 22, 2009 at 9:05 AM, David Cournapeau 
> > wrote:
> >>
> >> Hi,
> >>
> >> I have just released the 2nd release candidate for numpy 1.4.0, which
> >> fixes a few critical bugs founds since the RC1. Tarballs and binary
> >> installers for numpy/scipy may be found on
> >> https://sourceforge.net/projects/numpy.
> >>
> >> cheers,
> >>
> >> David
> >> ___
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion@scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> > This release results with the same import error on my system that I
> posted
> > on
>
> Don't use develop, and install numpy normally, but from scratch.
> Develop mode has some quircks, and it does not worth it unless you
> want to work on numpy code yourself IMHO,
>

OK, a clean svn check-out and python setup.py install I get another
interesting import error:

[gse...@ccn ~]$ pwd
/home/gsever
[gse...@ccn ~]$ python
Python 2.6 (r26:66714, Jun  8 2009, 16:07:26)
[GCC 4.4.0 20090506 (Red Hat 4.4.0-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/gsever/Desktop/python-repo/numpy/numpy/__init__.py", line 123,
in 
raise ImportError(msg)
ImportError: Error importing numpy: you should not try to import numpy from
its source directory; please exit the numpy source tree, and
relaunch
your python intepreter from there.
>>>

I launch the interpreter from a different directory than where the sources
located, however it still complains.

For the develop, it is one of easiest ways to catch up the bug-fixes even
though I don't work on the source directly. So far besides a few glitches it
was always working. I also install scipy, ipython, matplotlib, sympy and all
other available packages using develop. Keep the checkouts in the directory
on my desktop and if/when necessary do svn up or whichever command it
corresponds to their respective vcs. I wonder how other people keep up the
changes easily without using develop option.




>
> cheers,
>
> David
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

[Gökhan Sever]

On Tue, Dec 22, 2009 at 9:05 AM, David Cournapeau wrote:

> Hi,
>
> I have just released the 2nd release candidate for numpy 1.4.0, which
> fixes a few critical bugs founds since the RC1. Tarballs and binary
> installers for numpy/scipy may be found on
> https://sourceforge.net/projects/numpy.
>
> cheers,
>
> David
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

This release results with the same import error on my system that I posted
on
http://old.nabble.com/Another-numpy-svn-installation-error-td26878029.html

[gse...@ccn Desktop]$ python
Python 2.6 (r26:66714, Jun  8 2009, 16:07:26)
[GCC 4.4.0 20090506 (Red Hat 4.4.0-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/gsever/Desktop/python-repo/numpy/numpy/__init__.py", line 132,
in 
import add_newdocs
  File "/home/gsever/Desktop/python-repo/numpy/numpy/add_newdocs.py", line
9, in 
from lib import add_newdoc
  File "/home/gsever/Desktop/python-repo/numpy/numpy/lib/__init__.py", line
4, in 
from type_check import *
  File "/home/gsever/Desktop/python-repo/numpy/numpy/lib/type_check.py",
line 8, in 
import numpy.core.numeric as _nx
  File "/home/gsever/Desktop/python-repo/numpy/numpy/core/__init__.py", line
6, in 
import umath
ImportError: /home/gsever/Desktop/python-repo/numpy/numpy/core/umath.so:
undefined symbol: npy_spacing

Is there any remedy for this error?


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Another numpy svn installation error

[Gökhan Sever]

Hello,

Here are the steps that I went through to install the numpy from the
svn-repo:

svn co http://svn.scipy.org/svn/numpy/trunk numpy
Be "su" and type: python setupegg.py develop
Successful installation so far, but import fails with the given error:

[gse...@ccn Desktop]$ python
Python 2.6 (r26:66714, Jun  8 2009, 16:07:26)
[GCC 4.4.0 20090506 (Red Hat 4.4.0-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/gsever/Desktop/python-repo/numpy/numpy/__init__.py", line 132,
in 
import add_newdocs
  File "/home/gsever/Desktop/python-repo/numpy/numpy/add_newdocs.py", line
9, in 
from numpy.lib import add_newdoc
  File "/home/gsever/Desktop/python-repo/numpy/numpy/lib/__init__.py", line
4, in 
from type_check import *
  File "/home/gsever/Desktop/python-repo/numpy/numpy/lib/type_check.py",
line 8, in 
import numpy.core.numeric as _nx
  File "/home/gsever/Desktop/python-repo/numpy/numpy/core/__init__.py", line
6, in 
import umath
ImportError: /home/gsever/Desktop/python-repo/numpy/numpy/core/umath.so:
undefined symbol: npy_spacing

My platform:
Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas


Any ideas what might be wrong here?

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Producing a Histogram When Bins Are Known

[Gökhan Sever]

On Fri, Nov 27, 2009 at 8:05 PM, Sebastian  wrote:

> Hi Chris, yeah there should, try the following:
> import numpy
> import matplotlib.pyplot as pylab
> regards
>
>
Shouldn't the 2nd import be:

import matplotlib.pyplot as plt

?


>
> On Fri, Nov 27, 2009 at 8:47 PM, Wayne Watson <
> sierra_mtnv...@sbcglobal.net> wrote:
>
>> I tried this and it put ranges on y from 0 to 0.45 and x from 5 to 50.
>>
>> import numpy as np
>> import pylab
>>
>> v = np.array([20, 15,10,30, 50, 30, 20, 25, 10])
>> #Plot a normalized histogram
>> print np.linspace(0,50,10)
>> pylab.hist(v, normed=1, bins=np.linspace(0,9,10), range=(0,100))
>> pylab.show()
>>
>> I  added the two imports. I got a fig error on the first line.
>> import pylab
>> import numpy
>>
>> Shouldn't there by a pylab.Show in there?
>>
>> ax = fig.add_subplot(1,1,1)
>> pylab.title(r'\Large  BCG NO radio distribution $ \rm{TITLE}$')
>> n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)),
>>
>> range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)),
>> normed=1, facecolor='y', alpha=0.5)
>> ax.set_xlabel(r'\Large$ \rm{values}$')
>> ax.set_ylabel(r'\Large Delatavalue/Value')
>>
>>
>> gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1)
>>
>> gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value))
>> pylab.plot(gausx,gaus, color='red', lw=2)
>> ax.set_xlim(-1.5, 1.5)
>> ax.grid(True)
>>
>> Sebastian wrote:
>> > Did you try using the parameter range?
>> > I do something like this.
>> > regards
>> >
>> > ax = fig.add_subplot(1,1,1)
>> > pylab.title(r'\Large  BCG NO radio distribution $ \rm{TITLE}$')
>> > n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)),
>> >
>> range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)),
>> > normed=1, facecolor='y', alpha=0.5)
>> > ax.set_xlabel(r'\Large$ \rm{values}$')
>> > ax.set_ylabel(r'\Large Delatavalue/Value')
>> >
>> >
>> gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1)
>> >
>> gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value))
>> > pylab.plot(gausx,gaus, color='red', lw=2)
>> > ax.set_xlim(-1.5, 1.5)
>> > ax.grid(True)
>> >
>> >
>> > On Fri, Nov 27, 2009 at 4:38 PM, Christopher Barker
>> > mailto:chris.bar...@noaa.gov>> wrote:
>> >
>> > josef.p...@gmail.com  wrote:
>> > > On Fri, Nov 27, 2009 at 12:57 PM, Skipper Seabold
>> > mailto:jsseab...@gmail.com>> wrote:
>> >
>> > >>  This kind of info might be useful to other newcomers
>> > >> somewhere...  ?  Thoughts
>> on
>> > >> posting this on the wiki here?
>> > >
>> > > I also agree. It will improve with the newly redesigned website
>> > for scipy.org 
>> > > However, I cannot find the link right now for the development
>> > version of
>> > > the new website.
>> >
>> > Feel free to crib whatever you want from my post for that -- or
>> > suggest
>> > a place for me to put it, and I'll do it. I'm just not sure where it
>> > should go at this point.
>> >
>> > -Chris
>> >
>> >
>> > --
>> > Christopher Barker, Ph.D.
>> > Oceanographer
>> >
>> > Emergency Response Division
>> > NOAA/NOS/OR&R(206) 526-6959   voice
>> > 7600 Sand Point Way NE   (206) 526-6329   fax
>> > Seattle, WA  98115   (206) 526-6317   main reception
>> >
>> > chris.bar...@noaa.gov 
>> > ___
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion@scipy.org 
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> >
>> > 
>> >
>> > ___
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion@scipy.org
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>>
>> --
>>   Wayne Watson (Watson Adventures, Prop., Nevada City, CA)
>>
>> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time)
>>  Obz Site:  39° 15' 7" N, 121° 2' 32" W, 2700 feet
>>
>>   350 350 350 350 350 350 350 350 350 350
>> Make the number famous. See 350.org
>>The major event has passed, but keep the number alive.
>>
>>Web Page: 
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _

Re: [Numpy-discussion] using numpy.savetxt to save columns of numerical values and columns of text values

[Gökhan Sever]

On Thu, Nov 26, 2009 at 11:30 PM, Richared Beare <
richard.be...@sci.monash.edu.au> wrote:

> I have been unable to find a way of doing a very simple thing: saving
> data that contains both arrays of numerical values and arrays of string
> values, using savetxt in numpy.
>
> As a very simple example, suppose a is a numpy array of integers and b
> is one containing strings, e.g.:
>
>a = np.array([1,2,3])
>
>b = np.array(['one', 'two', 'three'])
>
> The following call to savetxt:
>
>savetxt('test.txt', transpose((a,b)), fmt='%i %s')
>
> produces the following error:
>
>float argument required, not numpy.string_
>
> I don't have any problems if I mix integers and floats in the format
> string, or save all values as strings, but I am completely unable to
> save formatted numerical values and strings in different columns using
> savetxt.
>
> There must be an easy solution!
>

Seemingly savetxt('test.txt', transpose((a,b)), fmt='%s %s')  ; replacing
the %i with %s works. However you can't control the output precision while
saving the data into a file.


>
> Thank you.
>
> Richard Beare
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Python 2.6, NumPy on CentOS 5.3

[Gökhan Sever]

On Mon, Nov 23, 2009 at 7:07 PM,  wrote:

> An application package that I have requires Python 2.6 and NumPy.
>
> I've installed Python 2.6 in a parallel manner as follows:
>
> NO modification of the core Python2.4 in /usr/bin has been done. Rather, I
> installed Python 2.6 under /opt/Python_2.6.4 and modified my user (not root)
> environment variables appropriately. The directory /opt/Python_2.6.4 was
> modified with chown to give me rwx access.
>
> To install NumPy, I've downloaded the latest .tgz sources (v1.3.0) to
> build. When I attempt to configure/build I receive various errors related to
> blas and lapack. The NumPy configuration is searching /usr/lib, /usr/lib64,
> /usr/local/lib, and /usr/local/lib64 for various blas, lapack, and atlas
> libraries. Within /usr/lib64 I do find a few lapack and blas and lapack
> libraries installed (libblas.so.3.1.1 and liblapack.so.3.1.1), but configure
> is not finding them. No atlas libraries are found, but my understanding is
> that these are deprecated anyway.
>
> As an alternative, I tried to install NumPy using the standard Python 2.4.3
> using yum install NumPy but I receive an error saying that NumPy is
> obsoleted by PyNumeric. What?? PyNumeric is the precursor to NumPy. So even
> in the most basic instance, I cannot install NumPy because a deprecated
> library is seen as higher priority? Even given the generally out of date
> nature of CentOS this is unrealistic.
>
> Finally, I could try to build blas and lapack myself, but this seems to
> border on insanity.
>
> Any help is appreciated.
>
> -Kirk
>

I had success installing NumPy 1.3 to two of our RHEL 5.x (x86) machines
using the pre-built packages from this repository:
http://download.opensuse.org/repositories/home:/ashigabou/CentOS_5/

RHEL 5.x serie has Python 2.4.3 by default as you mentioned. Installation of
NumPy et. al is far easier in Fedora (mine is still at version 11) from the
source check-out plus manual installation of dependencies. I haven't tried
installing a Python 2.6.x to RHEL machines. I would like to see an update
from if you succeed with the mentioned repository.




> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fitting a curve on a log-normal distributed data

[Gökhan Sever]

One more update on this subject. I have been looking through some of the
papers on this topic, and I have finally found exactly what I need in this
paper:

Hussein, T., Dal Maso, M., Petaja, T., Koponen, I. K., Paatero, P., Aalto,
P. P., Hameri, K., and Kulmala, M.: Evaluation of an automatic algorithm for
fitting the particle number size distributions, Boreal Environ. Res., 10,
337–355, 2005.

Here is the abstract:

"The multi log-normal distribution function is widely in use to parameterize
the aerosol particle size distributions. The main purpose of such a
parameterization is to quantitatively describe size distributions and to
allow straightforward comparisons between different aerosol particle data
sets. In this study, we developed and evaluated an algorithm to parameterize
aerosol particle number size distributions with the multi log-normal
distribution function. The current algorithm is automatic and does not need
a user decision for the initial input parameters; it requires only the
maximum number of possible modes and then it reduces this number, if
possible, without affecting the fitting quality. The reduction of the number
of modes is based on an overlapping test between adjacent modes. The
algorithm was evaluated against a previous algorithm that can be considered
as a standard procedure. It was also evaluated against a long-term data set
and different types of measured aerosol particle size distributions in the
ambient atmosphere. The evaluation of the current algorithm showed the
following advantages: (I) it is suitable for different types of aerosol
particles observed in different environments and conditions, (2) it showed
agreement with the previous standard algorithm in about 90% of long-term
data set, (3) it is not time-consuming, particularly when long-term data
sets are analyzed, and (4) it is a useful tool in the studies of atmospheric
aerosol particle formation and transformation."

The full-text is freely available at:
http://www.borenv.net/BER/pdfs/ber10/ber10-337.pdf


On Mon, Nov 16, 2009 at 11:44 PM, Gökhan Sever wrote:

> Hello,
>
> I have a data which represents aerosol size distribution in between 0.1 to
> 3.0 micrometer ranges. I would like extrapolate the lower size down to 10
> nm. The data in this context is log-normally distributed. Therefore I am
> looking a way to fit a log-normal curve onto my data. Could you please give
> me some pointers to solve this problem?
>
> Thank you.
>
> --
> Gökhan
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fitting a curve on a log-normal distributed data

[Gökhan Sever]

On Thu, Nov 19, 2009 at 9:12 PM, Ian Mallett  wrote:

> Hello,
>
> My analysis shows that the exponential regression gives the best result
> (r^2=87%)--power regression gives worse results (r^2=77%).  Untransformed
> data gives r^2=76%.
>
> I don't think you want lognorm.  If I'm not mistaken, that fits the data to
> a log(normal distribution random variable).
>

Lognormal fitting is the motivation behind my study since aerosol in the
atmosphere typically log-normally size distributed. See for an example case
:
http://webmathematica.seas.harvard.edu/webMathematica/AerosolCalculator.jsp

Of course this is just a simplification. There are other approaches to
represent the size-distribution besides the lognormal. So my intention is
not actually find the best fit but represent the actuality as much as I can.



>
> So, take the logarithm (to any base) of all the "conc" values.  Then do a
> linear regression on those values versus "sizes".
>
> Try (semi-pseudocode):
> slope, intercept, p, error = scipy.stats.linregress(sizes,log(conc))
>

linregress also returns the r_value which I am not sure if the documentation
from the web-based editor checked-in completely to the scipy trunk yet.


>
> Ian
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fitting a curve on a log-normal distributed data

[Gökhan Sever]

On Tue, Nov 17, 2009 at 12:13 AM, Ian Mallett  wrote:

> Theory wise:
> -Do a linear regression on your data.
> -Apply a logrithmic transform to your data's dependent variable, and do
> another linear regression.
> -Apply a logrithmic transform to your data's independent variable, and do
> another linear regression.
> -Take the best regression (highest r^2 value) and execute a back transform.
>
> Then, to get your desired extrapolation, simply substitute in the size for
> the independent variable to get the expected value.
>
> If, however, you're looking for how to implement this in NumPy or SciPy, I
> can't really help :-P
> Ian
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
OK, before applying your suggestions. I have a few more questions. Here is 1
real-sample data that I will use as a part of the log-normal fitting. There
is 15 elements in this arrays each being a concentration for corresponding
0.1 - 3.0 um size ranges.

I[74]: conc
O[74]:
array([ 119.7681,  118.546 ,  146.6548,   96.5478,  109.9911,   32.9974,
 20.7762,6.1107,   12.2212,3.6664,3.6664,1.2221,
  2.4443,2.4443,3.6664])

For now not calibrated size range I just assume a linear array:

I[78]: sizes = linspace(0.1, 3.0, 15)

I[79]: sizes
O[79]:
array([ 0.1   ,  0.30714286,  0.51428571,  0.72142857,  0.92857143,
1.13571429,  1.34285714,  1.55  ,  1.75714286,  1.96428571,
2.17142857,  2.37857143,  2.58571429,  2.79285714,  3.])


Not a very ideal looking log-normal, but so far I don't know what else
besides a log-normal fit would give me a better estimate:
I[80]: figure(); plot(sizes, conc)
http://img406.imageshack.us/img406/156/sizeconc.png

scipy.stats has the lognorm.fit

lognorm.fit(data,s,loc=0,scale=1)
- Parameter estimates for lognorm data

and applying this to my data. However not sure the right way of calling it,
and not sure if this could be applied to my case?

I[81]: stats.lognorm.fit(conc)
O[81]: array([ 2.31386066,  1.19126064,  9.5748391 ])

Lastly, what is the way to create a ideal log-normal sample using the
stats.lognorm.rvs?

Thanks


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Fitting a curve on a log-normal distributed data

[Gökhan Sever]

Hello,

I have a data which represents aerosol size distribution in between 0.1 to
3.0 micrometer ranges. I would like extrapolate the lower size down to 10
nm. The data in this context is log-normally distributed. Therefore I am
looking a way to fit a log-normal curve onto my data. Could you please give
me some pointers to solve this problem?

Thank you.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy distutils and distribute

[Gökhan Sever]

On Sat, Nov 14, 2009 at 9:29 AM, Darren Dale  wrote:

> Please excuse the cross-post. I have installed distribute-0.6.8 and
> numpy-svn into my ~/.local/lib/python2.6/site-packages (using "python
> setup.py install --user"). I am now trying to install Enthought's
> Enable from a fresh svn checkout on ubuntu karmic:
>
> $ python setup.py develop --user
> [...]
> building library "agg24_src" sources
> building library "kiva_src" sources
> building extension "enthought.kiva.agg._agg" sources
> building extension "enthought.kiva.agg._plat_support" sources
> building data_files sources
> build_src: building npy-pkg config files
> running build_clib
> customize UnixCCompiler
> customize UnixCCompiler using build_clib
> running build_ext
> build_clib already run, it is too late to ensure in-place build of
> build_clib
> Traceback (most recent call last):
>  File "setup.py", line 327, in 
>**config
>  File
> "/home/darren/.local/lib/python2.6/site-packages/numpy/distutils/core.py",
> line 186, in setup
>return old_setup(**new_attr)
>  File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
>dist.run_commands()
>  File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
>self.run_command(cmd)
>  File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
>cmd_obj.run()
>  File
> "/home/darren/.local/lib/python2.6/site-packages/numpy/distutils/command/build_ext.py",
> line 74, in run
>self.library_dirs.append(build_clib.build_clib)
> UnboundLocalError: local variable 'build_clib' referenced before assignment
>
>
Darren,

I had a similar installation error. Could you try the solution that was
given in this thread?

http://www.mail-archive.com/numpy-discussion@scipy.org/msg19798.html



>
> I am able to run "python setup.py install --user". Incidentally,
> "python setup.py develop --user" worked for TraitsGui, EnthoughtBase,
> TraitsBackendQt4.
>
> I have been (sort of) following the discussion on distutils-sig. Thank
> you Robert, David, Pauli, for all your effort.
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Multiple Regression

[Gökhan Sever]

On Thu, Nov 12, 2009 at 9:14 PM, Sturla Molden  wrote:
> Alexey Tigarev skrev:
>> I have implemented multiple regression in a following way:
>>
>>
> You should be using QR or SVD for this.
>
> Sturla
>

Seeing this QR and SVD terms I recalled the answer to the "I am the very
model for a student mathematical" poem. I am just quoting the answer part
for the full poem see Google :) My apologies for the noise in the subject...

"When you have learnt just what is meant by 'Jacobian' and 'Abelian';
When you at sight can estimate, for the modal, mean and median;
When describing normal subgroups is much more than recitation;
When you understand precisely what is 'quantum excitation';

When you know enough statistics that you can recognise RV;
When you have learnt all advances that have been made in SVD;
And when you can spot the transform that solves some tricky PDE,
You will feel no better student has ever sat for a degree.

Your accumulated knowledge, whilst extensive and exemplary,
Will have only been brought down to the beginning of last century,
But still in matters rational, and logical and practical,
You'll be the very model of a student mathematical."

*
*K. F. Riley, with apologies to W.S. Gilbert *




-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] parsing tab separated files into dictionaries - alternative to genfromtxt?

[Gökhan Sever]

On Wed, Nov 11, 2009 at 11:53 AM, per freem  wrote:
> hi all,
>
> i've been using genfromtxt to parse tab separated files for plotting
> purposes in matplotlib. the problem is that genfromtxt seems to give
> only two ways to access the contents of the file: one is by column,
> where you can use:
>
> d = genfromtxt(...)
>
> and then do d['header_name1'] to access the column named by
> 'header_name1', d['header_name2'] to access the column named by
> 'header_name2', etc.  Or it will allow you to traverse the file line
> by line, and then access each header by number, i.e.
>
> for line in d:
>  field1 = d[0]
>  field2 = d[1]
>  # etc.
>
> the problem is that the second method relies on knowing the order of
> the fields rather than just their name, and the first method does not
> allow line by line iteration.
> ideally what i would like is to be able to traverse each line of the
> parsed file, and then refer to each of its fields by header name, so
> that if the column order in the file changes my program will be
> unaffected:
>
> for line in d:
>  field1 = ['header_name1']
>  field2 = ['header_name2']
>
> is there a way to do this using standard matplotlib/numpy/scipy
> utilities? i could write my own code to do this but it seems like
> something somebody probably already thought of a good representation
> for and has implemented a more optimized version than i could write on
> my own. does such a thing exist?
>
> thanks very much

I have a constructor class to read space-delimited ASCII files.

class NasaFile(object):
   def __init__(self, filename):
...
# Reading data
_data = np.loadtxt(filename, dtype='float', skiprows=self.NLHEAD).T
# Read using data['Time'] syntax
self.data = dict(zip(self.VDESC, _data))
...

There is a meta-header in this type of data and NLHEAD is the variable
telling me how many lines to skip to reach the actual data. VDESC
tells me what each columns are (starting with Time variable and many
other different measurement results.)

There is not any column dependence in this case, and generically read
any length specifically formatted data. For instance:

from nasafile import NasaFile

c = NasaFile("mydata")

c.data['Time'] gets me the whole Time column as an ndarray . Why do
you think dictionaries are not sufficient for your case? I was using
locals() to create automatic names but that was not a very wise
approach.



> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy Array of dtype=object with strings and floats question

[Gökhan Sever]

On Tue, Nov 10, 2009 at 12:09 PM, Darryl Wallace
 wrote:
> Hello again,
> The best way so far that's come to my attention is to use:
> numpy.ma.masked_object
> The problem with this is that it's looking for a specific instance of an
> object.  So if the user had some elements of their array that were, for
> example, "randomString" , then it would not be picked up
> e.g.
> ---
> from numpy import *
> mixedArray=array([1,2, '', 3, 4, 'randomString'], dtype=object)
> mixedArrayMask = ma.masked_object(mixedArray, 'randomString').mask
> ---
> then mixedArrayMask will yield:
>
> array([ False, False, False, False, False, True])
> Can anyone help me so that all strings are found in the array without having
> to explicitly loop through them in Python?
> Thanks,
> Darryl

Why not stick to a same Missing-Value-Code or for all the non-valid
data? I don't know how MA module would handle mixed MVCs in a same
array without modifying the existing code. Otherwise looping over the
array an masking the str instances as NaN would be my alternative
solution.


>
> On Fri, Nov 6, 2009 at 3:56 PM, Darryl Wallace 
> wrote:
>>
>> What I'm doing is importing some data from excel and sometimes there are
>> strings in the worksheet.  Often times a user will use an empty cell or a
>> string to represent data that is missing.
>> e.g.
>> from numpy import *
>> mixedArray=array([1, 2, '', 3, 4, 'String'], dtype=object)
>> Two questions:
>> 1) Is there a quick way to find the elements in the array that are the
>> strings without iterating over each element in the array?
>> or
>> 2) Could I quickly turn it into a masked array of type float where all
>> string elements are set as missing points?
>> I've been struggling with this for a while and can't come across a method
>> that will all me to do it without iterating over each element.
>> Any help or pointers in the right direction would be greatly appreciated.
>> Thanks,
>> Darryl
>
>
>
> --
> __
> Darryl Wallace: Project Leader
> ProSensus Inc.
> McMaster Innovation Park
> 175 Longwood Road South, Suite 301
> Hamilton, Ontario, L8P 0A1
> Canada        (GMT -05:00)
>
> Tel:       1-905-528-9136
> Fax:       1-905-546-1372
>
> Web site:  http://www.prosensus.ca/
> __
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Comparing variable time-shifted two measurements

[Gökhan Sever]

Hello,

I have two aircraft based aerosol measurements. The first one is dccnConSTP
(blue), and the latter is CPCConc (red) as shown in this screen capture. (
http://img513.imageshack.us/img513/7498/ccncpclag.png). My goal is to
compare these two measurements. It is expected to see that they must have a
positive correlation throughout the flight. However, the instrument that
gives CPCConc was experiencing a sampling issue and therefore making a
varying time-shifted measurements with respect to the first instrument.
(From the first box it is about 20 seconds, 24 from the seconds before the
dccnConSTP measurements shows up.) In other words in different altitude
levels, I have varying time differences in between these two measurements in
terms of their shapes. So, my goal turns to addressing this variable
shifting issue before I start doing the comparisons.

Is there a known automated approach to correct this mentioned varying-lag
issue? If so, how?

Thank you.

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Using matplotlib's prctile on masked arrays

[Gökhan Sever]

On Tue, Oct 27, 2009 at 12:23 PM, Pierre GM  wrote:

>
> On Oct 27, 2009, at 7:56 AM, Gökhan Sever wrote:
> >
> >
> > Unfortunately, matplotlib.mlab's prctile cannot handle this division:
>
> Actually, the division's OK, it's mlab.prctile which is borked. It
> uses the length of the input array instead of its count to compute the
> nb of valid data. The easiest workaround in your case is probably to
> use:
>  >>> prctile((am/bm).compressed(), p=[5,25,50,75,95])
> HIH
> P.
>

Great. Exact solution. I should have asked this last week :)

One simple method solves all the riddle. I had manually masked the MVCs
using NaN's.

My guess is using compressed() masked arrays could be used with any of
regularly defined numpy and scipy functions, right?

Thanks for the tip.


> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion