Re: [Numpy-discussion] Optimize speed of for loop using numpy
Trond, See if the attached file contains something close to what you need. It has no loops at all; I have not timed it, but it should be quite quick. I have given it only a cursory check, so I don't guarantee it works correctly. Depending on how your particular NetCDF interface works, you might need to copy arrays from NetCDF to ensure they are genuine ndarray objects. For plotting, you might want to try matplotlib. I think you will find it easier to use than GMT, especially if you are accustomed to matlab. Eric Trond Kristiansen wrote: Hi again. I have attached the function that the FOR loop is part of as a python file. What I am trying to do is to create a set of functions that will read the output files (NetCDF) from running the ROMS model (ocean model). The output file is organized in xi (x-direction), eta (y-direction), and s (z-direction) where the s-values are vertical layers and not depth. This particular function (z_slice) will find the closest upper and lower s-layer for a given depth in meters (e.g. -100). Then values from the two selcted layers will be interpolated to create a new layer at the selected depth (-100). The problem is that the s-layers follow the bathymetry and a particular s-layer will therefore sometimes be above and sometimes below the selected depth that we want to interpolate to. That's why I need a quick script that searches all of the layers and find the upper and lower layers for a given depth value (which is negative). The z_r is a 3D array (s,eta,xi) that is created using the netcdf module. The main goal of these set of functions is to move away from using matlab, but also to speed things up. The sliced data array will be plotted using GMT or pyNGL. Thanks for helping me. Cheers, Trond ''' Suppose you have a 3-D array of (negative) depths with indices level, x, and y. You also have a field, say T, indexed the same way, and you want to interpolate it to a given depth. Here is a quick example of how to do it with no loops. This is not carefully tested, may be buggy or incorrect, needs docstrings, etc. It is intended to work for arrays of any dimension greater than 1 provided the first dimension is the one indexed by level. ''' import numpy as np def brackets(z, z0): below = (z > z0).astype(np.int16).sum(axis=0) above = below - 1 badmask = ((below == z.shape[0]) | (above < 0)).astype(np.bool) above[badmask] = 0 below[badmask] = z.shape[0] - 1 return below, above, badmask def interp(T, z, z0): below, above, badmask = brackets(z, z0) ind = np.indices(z.shape[1:]) ibelow = tuple([below] + list(ind)) iabove = tuple([above] + list(ind)) Tb = T[ibelow] zb = z[ibelow] dT = T[iabove] - Tb dz = z[iabove] - zb Ti = Tb + (dT/dz) * (z0 - zb) return Ti, badmask #Example use: # Make depth levels varying linearly from the bottom to the surface: nlevels = 5 # Use larger values to verify convergence. z0 = np.linspace(0.0, -5000.0, num=nlevels) z0.shape = nlevels,1,1 z1 = np.linspace(0.5, 1.0, num=4) z1.shape = 1,4,1 z2 = np.linspace(0.1, 1.0, num=3) z2.shape = 1,1,3 z = z0*z1*z2 # Temperature increases quadratically from 0 at -5000 m to 20 at the surface: T = 20.0 * (1 + z/5000.0)**2 T1500, T1500badmask = interp(T, z, -1500) # Now you might want to use a masked array to keep track of # the valid values: T1500m = np.ma.array(T1500, mask=T1500badmask) print T1500m ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FORTRAN compiler detection
Travis E. Oliphant wrote: > You need to use it in your case because you are linking against lapack > and blas that were built with a Fortran compiler. The Fortran compiler > is only used in the link step of lapack_lite.so > If for some reasons, Christopher does not want to use fortran compiler at all, I just want to mention that on solaris (and linux), it should be possible to link with BLAS/LAPACK without a fortran compiler, if using sun performance libraries, since they do not depend on fortran at all (but do provide fortran calling convention). But it is certainly easier to use a fortran compiler, since this configuration is supported out of the box. cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)
Manos Pontikakis wrote:
> Hello,
>
> I am trying to install numpy 1.0.4 to the following machine:
>
> $ uname -a
> Linux myhostname 2.6.9-55.ELsmp #1 SMP Sat Apr 21 11:16:24 EDT 2007
> x86_64 x86_64 x86_64 GNU/Linux
>
>
> and I am getting the error that appears in the subject of this email.
> It appears that BLAS and LAPACK libraries cannot be found as well, but
> I think numpy still shouldn't have problem with that. There must be
> some problem with 64 bit. Please see the complete log.
>
> ---
> Running from numpy source directory.
> F2PY Version 2_4422
> blas_opt_info:
> blas_mkl_info:
> libraries mkl,vml,guide not found in /usr/local/lib
> libraries mkl,vml,guide not found in /usr/lib
> NOT AVAILABLE
>
> atlas_blas_threads_info:
> Setting PTATLAS=ATLAS
> libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
> libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
> libraries ptf77blas,ptcblas,atlas not found in /usr/lib
> NOT AVAILABLE
>
> atlas_blas_info:
> libraries f77blas,cblas,atlas not found in /usr/local/lib
> libraries f77blas,cblas,atlas not found in /usr/lib/sse2
> libraries f77blas,cblas,atlas not found in /usr/lib
> NOT AVAILABLE
>
> /.../numpy-1.0.4/numpy/distutils/system_info.py:1340: UserWarning:
> Atlas (http://math-atlas.sourceforge.net/) libraries not found.
> Directories to search for the libraries can be specified in the
> numpy/distutils/site.cfg file (section [atlas]) or by setting
> the ATLAS environment variable.
> warnings.warn(AtlasNotFoundError.__doc__)
> blas_info:
> libraries blas not found in /usr/local/lib
> FOUND:
> libraries = ['blas']
> library_dirs = ['/usr/lib']
> language = f77
>
> FOUND:
> libraries = ['blas']
> library_dirs = ['/usr/lib']
> define_macros = [('NO_ATLAS_INFO', 1)]
> language = f77
>
> lapack_opt_info:
> lapack_mkl_info:
> mkl_info:
> libraries mkl,vml,guide not found in /usr/local/lib
> libraries mkl,vml,guide not found in /usr/lib
> NOT AVAILABLE
>
> NOT AVAILABLE
>
> atlas_threads_info:
> Setting PTATLAS=ATLAS
> libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
> libraries lapack_atlas not found in /usr/local/lib
> libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
> libraries lapack_atlas not found in /usr/lib/sse2
> libraries ptf77blas,ptcblas,atlas not found in /usr/lib
> libraries lapack_atlas not found in /usr/lib
> numpy.distutils.system_info.atlas_threads_info
> NOT AVAILABLE
>
> atlas_info:
> libraries f77blas,cblas,atlas not found in /usr/local/lib
> libraries lapack_atlas not found in /usr/local/lib
> libraries f77blas,cblas,atlas not found in /usr/lib/sse2
> libraries lapack_atlas not found in /usr/lib/sse2
> libraries f77blas,cblas,atlas not found in /usr/lib
> libraries lapack_atlas not found in /usr/lib
> numpy.distutils.system_info.atlas_info
> NOT AVAILABLE
>
> /.../numpy-1.0.4/numpy/distutils/system_info.py:1247: UserWarning:
> Atlas (http://math-atlas.sourceforge.net/) libraries not found.
> Directories to search for the libraries can be specified in the
> numpy/distutils/site.cfg file (section [atlas]) or by setting
> the ATLAS environment variable.
> warnings.warn(AtlasNotFoundError.__doc__)
> lapack_info:
> libraries lapack not found in /usr/local/lib
> FOUND:
> libraries = ['lapack']
> library_dirs = ['/usr/lib']
> language = f77
>
> FOUND:
> libraries = ['lapack', 'blas']
> library_dirs = ['/usr/lib']
> define_macros = [('NO_ATLAS_INFO', 1)]
> language = f77
>
> running install
> running build
> running config_cc
> unifing config_cc, config, build_clib, build_ext, build commands
> --compiler options
> running config_fc
> unifing config_fc, config, build_clib, build_ext, build commands
> --fcompiler options
> running build_src
> building py_modules sources
> building extension "numpy.core.multiarray" sources
> Generating build/src.linux-x86_64-2.5/numpy/core/config.h
> customize GnuFCompiler
> Could not locate executable g77
> Could not locate executable f77
> customize IntelFCompiler
> Could not locate executable ifort
> Could not locate executable ifc
> customize LaheyFCompiler
> Could not locate executable lf95
> customize PGroupFCompiler
> Could not locate executable pgf90
> Could not locate executable pgf77
> customize AbsoftFCompiler
> Could not locate executable f90
> customize NAGFCompiler
> Could not locate executable f95
> customize VastFCompiler
> customize GnuFCompiler
> customize CompaqFCompiler
> Could not locate executable fort
> customize IntelItaniumFCompiler
> Could not locate executable efort
> Could not locate executable efc
> customize IntelEM64TFCompiler
> customize Gnu95FCompiler
> Found executable /usr/bin/gfortran
>
> customize G95FCompiler
> Could not locate executable g95
> don't know how to compile Fortran code o
Re: [Numpy-discussion] masked_array/matplotlib issue with memmaps
On Tue, Feb 26, 2008 at 5:26 PM, Christopher Burns <[EMAIL PROTECTED]> wrote: > If I initialize an AxesImage using a np.zeros array and then set the > axes data later to a np.memmap array, I get a RuntimeError when > matplotlib tries to autoscale the image. The errors continue to fill > my console and I'm forced to close the shell. This bug was introduced > when I switched from numpy v1.0.3.1 to the trunk v1.0.5.dev4815 > > The two hacks to get around this are: > 1) Setting any array element to something other than zero fixes the error: > zdata[0,0] = 1 > 2) Specify the extent and max/min values when creating the image: > imgaxes = pylab.imshow(zdata, extent=(0, data_shape[1], > data_shape[0], 0), vmin=0, vmax=1) > > Unfortunately, due to the way this errors I'm having a difficult time > debugging it. I'm hoping someone with in-depth knowledge of > masked_arrays will have some insight. > Exception exceptions.AttributeError: "'memmap' object has no attribute > '_mmap'" in 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., > 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., > 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., > 0., 0.])> ignored Some operations on numpy.memmap objects create new arrays, but unfortunately, the new array objects, which should be ndarrays, are created as numpy.memmap instances even though they aren't. When they go to clean up after themselves, they fail. A workaround would be to make numpy.memmap.__del__ more robust and do nothing if ._mmap isn't present. A real fix would be to figure out how to make sure that "memmap+memmap", etc., make ndarray instances rather than memmap instances. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] masked_array/matplotlib issue with memmaps
If I initialize an AxesImage using a np.zeros array and then set the
axes data later to a np.memmap array, I get a RuntimeError when
matplotlib tries to autoscale the image. The errors continue to fill
my console and I'm forced to close the shell. This bug was introduced
when I switched from numpy v1.0.3.1 to the trunk v1.0.5.dev4815
The two hacks to get around this are:
1) Setting any array element to something other than zero fixes the error:
zdata[0,0] = 1
2) Specify the extent and max/min values when creating the image:
imgaxes = pylab.imshow(zdata, extent=(0, data_shape[1],
data_shape[0], 0), vmin=0, vmax=1)
Unfortunately, due to the way this errors I'm having a difficult time
debugging it. I'm hoping someone with in-depth knowledge of
masked_arrays will have some insight.
Code and output are below.
Thanks!
Chris
script to reproduce the bug
import pylab
import numpy as np
def printinfo(imgaxes):
a = imgaxes.get_array()
print '\nimgaxes array info:'
print 'type', type(a)
print 'shape', a.shape
print 'dtype', a.dtype
print 'has _mmap', hasattr(a, '_mmap')
data_type = 'float32'
data_shape = (30, 40)
zdata = np.zeros(data_shape, dtype=data_type)
#zdata[0,0] = 1# No exception raised if this line is executed
imgaxes = pylab.imshow(zdata)
printinfo(imgaxes)
mmdata = np.memmap('foo.dat', dtype=zdata.dtype, shape=zdata.shape, mode='w+')
imgaxes.set_data(mmdata)
printinfo(imgaxes)# imgaxes array now has a _mmap
pylab.show()
version info
In [2]: pylab.matplotlib.__version__
Out[2]: '0.91.2'
In [4]: numpy.version.version
Out[4]: '1.0.5.dev4817'
error
In [26]: run memmap_reassign.py
imgaxes array info:
type
shape (30, 40)
dtype float32
has _mmap False
imgaxes array info:
type
shape (30, 40)
dtype float32
has _mmap True
Exception exceptions.RuntimeError: 'maximum recursion depth exceeded'
in ignored
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (10, 0))
---
RuntimeError Traceback (most recent call last)
/Users/cburns/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.pyc
in _onPaint(self, evt)
1079 self.realize()
1080 # Render to the bitmap
-> 1081 self.draw(repaint=False)
1082 # Update the display using a PaintDC
1083 self.gui_repaint(drawDC=wx.PaintDC(self))
/Users/cburns/local/lib/python2.5/site-packages/matplotlib/backends/backend_wxagg.pyc
in draw(self, repaint)
59 """
60 DEBUG_MSG("draw()", 1, self)
---> 61 FigureCanvasAgg.draw(self)
62
63 self.bitmap =
_convert_agg_to_wx_bitmap(self.get_renderer(), None)
/Users/cburns/local/lib/python2.5/site-packages/matplotlib/backends/backend_agg.pyc
in draw(self)
356
357 self.renderer = self.get_renderer()
--> 358 self.figure.draw(self.renderer)
359
360 def get_renderer(self):
/Users/cburns/local/lib/python2.5/site-packages/matplotlib/figure.pyc
in draw(self, renderer)
622
623 # render the axes
--> 624 for a in self.axes: a.draw(renderer)
625
626 # render the figure text
/Users/cburns/local/lib/python2.5/site-packages/matplotlib/axes.pyc in
draw(self, renderer, inframe)
1303 mag = renderer.get_image_magnification()
1304 ims = [(im.make_image(mag),0,0)
-> 1305for im in self.images if im.get_visible()]
1306
1307
/Users/cburns/local/lib/python2.5/site-packages/matplotlib/image.pyc
in make_image(self, magnification)
129 im.is_grayscale = False
130 else:
--> 131 x = self.to_rgba(self._A, self._alpha)
132 im = _image.fromarray(x, 0)
133 if len(self._A.shape) == 2:
/Users/cburns/local/lib/python2.5/site-packages/matplotlib/cm.pyc in
to_rgba(self, x, alpha, bytes)
74 x = ma.asarray(x)
75 x = self.norm(x)
---> 76 x = self.cmap(x, alpha=alpha, bytes=bytes)
77 return x
78
/Users/cburns/local/lib/python2.5/site-packages/matplotlib/colors.pyc
in __call__(self, X, alpha, bytes)
431 vtype = 'array'
432 xma = ma.asarray(X)
--> 433 xa = xma.filled(0)
434 mask_bad = ma.getmask(xma)
435 if xa.dtype.char in npy.typecodes['Float']:
/Users/cburns/local/lib/python2.5/site-packages/numpy/ma/core.pyc in
filled(self, fill_value)
1542 m = self._mask
1543 if m is nomask or not m.any():
-> 1544 return self._data
1545 #
1546 if fill_value is None:
/Users/cburns/local/lib/python2.5/site-packages/numpy/ma/core.pyc in
_get_data(self)
1472
1473 """
-> 1474
Re: [Numpy-discussion] FORTRAN compiler detection
Christopher Hanley wrote: > Robert Kern wrote: > >> On Tue, Feb 26, 2008 at 12:09 PM, Christopher Hanley <[EMAIL PROTECTED]> >> wrote: >> >>> Robert Kern wrote: >>> > On Tue, Feb 26, 2008 at 11:50 AM, Christopher Hanley <[EMAIL PROTECTED]> >>> wrote: >>> >> Greetings, >>> >> >>> >> I was wondering if within the last 8 - 10 weeks anyone has made changes >>> >> to the way FORTRAN compilers are detected. In the past I was able to >>> >> specify which compiler was used by the F77 system variable. However, I >>> >> am now having a f90 compiler that exists on my Solaris system detected >>> >> regardless of the F77 value. Even unsetting the F77 variable leads to >>> >> the use of the f90 compiler. >>> >> >>> >> I was going to look though the distutil change logs but I was hoping >>> >> that someone might remember changing something off the top of their >>> heads. >>> > >>> > Which FORTRAN compilers do you have installed? What --fcompiler flag >>> > are you using? >>> > >>> Hi Robert, >>> >>> Thank you for your help. >>> >>> I am not using the --fcompiler flag. >>> >> Okay, use the --fcompiler flag. That is the way to tell numpy to use a >> particular compiler if you have multiple ones installed. >> >> > > What do you do if you have FORTRAN compilers installed but don't want to > use any of the compilers to build numpy? > You need to use it in your case because you are linking against lapack and blas that were built with a Fortran compiler. The Fortran compiler is only used in the link step of lapack_lite.so -Travis O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] loadtxt broken if file does not end in newline
On Tue, 26 Feb 2008, Lisandro Dalcin apparently wrote: > I believe the current 'loadtxt' function is broken I agree: http://projects.scipy.org/pipermail/numpy-discussion/2007-November/030057.html> Cheers, Alan Isaac ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] loadtxt broken if file does not end in newline
Dear all, I believe the current 'loadtxt' function is broken if file does not end in newline. The problem is at the last line of this fragment: for i,line in enumerate(fh): if ihttp://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Trouble With MaskedArray and Shared Masks
Alexander, The rationale behind the current behavior is to avoid an accidental propagation of the mask. Consider the following example: >>>m = numpy.array([1,0,0,1,0], dtype=bool_) >>>x = numpy.array([1,2,3,4,5]) >>>y = numpy.sqrt([5,4,3,2,1]) >>>mx = masked_array(x,mask=m) >>>my = masked_array(y,mask=m) >>>mx[0] = 0 >>>print mx,my, m [0 2 3 -- 5] [-- 4 3 -- 1] [ True False False True False] At the creation, mx._sharedmask and my._sharedmask are both True. Setting mx[0]=0 forces mx._mask to be copied, so that we don't affect the mask of my. Now, >>>m = numpy.array([1,0,0,1,0], dtype=bool_) >>>x = numpy.array([1,2,3,4,5]) >>>y = numpy.sqrt([5,4,3,2,1]) >>>mx = masked_array(x,mask=m) >>>my = masked_array(y,mask=m) >>>mx._sharedmask = False >>>mx[0] = 0 >>>print mx,my, m [0 2 3 -- 5] [5 4 3 -- 1] [False False False True False] By mx._sharedmask=False, we deceived numpy.ma into thinking that it's OK to update the mask of mx (that is, m), and my gets updated. Sometimes it's what you want (your case for example), often it is not: I've been bitten more than once before reintroducing the _sharedmask flag. As you've observed, setting a private flag isn't a very good idea: you should use the .unshare_mask() function instead, that copies the mask and set the _sharedmask to False. OK, in your example, copying the mask is not needed, but in more general cases, it is. At the initialization, self._sharedmask is set to (not copy). That is, if you didn't specify copy=True at the creation (the default being copy=False), self._sharedmask is True. Now, I recognize it's not obvious, and perhaps we could introduce yet another parameter to masked_array/array/MaskedArray, share_mask, that would take a default value of True and set self._sharedmask=(not copy)&share_mask So: should we introduce this extra parameter ? In any case, I hope it helps. P. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FORTRAN compiler detection
On Tue, Feb 26, 2008 at 12:46 PM, Christopher Hanley <[EMAIL PROTECTED]> wrote: > Robert Kern wrote: > > On Tue, Feb 26, 2008 at 12:09 PM, Christopher Hanley <[EMAIL PROTECTED]> > wrote: > >> Robert Kern wrote: > >> > On Tue, Feb 26, 2008 at 11:50 AM, Christopher Hanley <[EMAIL > PROTECTED]> wrote: > >> >> Greetings, > >> >> > >> >> I was wondering if within the last 8 - 10 weeks anyone has made > changes > >> >> to the way FORTRAN compilers are detected. In the past I was able to > >> >> specify which compiler was used by the F77 system variable. > However, I > >> >> am now having a f90 compiler that exists on my Solaris system > detected > >> >> regardless of the F77 value. Even unsetting the F77 variable leads > to > >> >> the use of the f90 compiler. > >> >> > >> >> I was going to look though the distutil change logs but I was hoping > >> >> that someone might remember changing something off the top of their > heads. > >> > > >> > Which FORTRAN compilers do you have installed? What --fcompiler flag > >> > are you using? > >> > > >> Hi Robert, > >> > >> Thank you for your help. > >> > >> I am not using the --fcompiler flag. > > > > Okay, use the --fcompiler flag. That is the way to tell numpy to use a > > particular compiler if you have multiple ones installed. > > > > What do you do if you have FORTRAN compilers installed but don't want to > use any of the compilers to build numpy? Unless if you are linking to a FORTRAN LAPACK or BLAS, no FORTRAN compiler should be used to build numpy. If you find that one is being used, please show us the full output log. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FORTRAN compiler detection
Robert Kern wrote: > On Tue, Feb 26, 2008 at 12:09 PM, Christopher Hanley <[EMAIL PROTECTED]> > wrote: >> Robert Kern wrote: >> > On Tue, Feb 26, 2008 at 11:50 AM, Christopher Hanley <[EMAIL PROTECTED]> >> wrote: >> >> Greetings, >> >> >> >> I was wondering if within the last 8 - 10 weeks anyone has made changes >> >> to the way FORTRAN compilers are detected. In the past I was able to >> >> specify which compiler was used by the F77 system variable. However, I >> >> am now having a f90 compiler that exists on my Solaris system detected >> >> regardless of the F77 value. Even unsetting the F77 variable leads to >> >> the use of the f90 compiler. >> >> >> >> I was going to look though the distutil change logs but I was hoping >> >> that someone might remember changing something off the top of their >> heads. >> > >> > Which FORTRAN compilers do you have installed? What --fcompiler flag >> > are you using? >> > >> Hi Robert, >> >> Thank you for your help. >> >> I am not using the --fcompiler flag. > > Okay, use the --fcompiler flag. That is the way to tell numpy to use a > particular compiler if you have multiple ones installed. > What do you do if you have FORTRAN compilers installed but don't want to use any of the compilers to build numpy? -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FORTRAN compiler detection
On Tue, Feb 26, 2008 at 12:09 PM, Christopher Hanley <[EMAIL PROTECTED]> wrote: > Robert Kern wrote: > > On Tue, Feb 26, 2008 at 11:50 AM, Christopher Hanley <[EMAIL PROTECTED]> > wrote: > >> Greetings, > >> > >> I was wondering if within the last 8 - 10 weeks anyone has made changes > >> to the way FORTRAN compilers are detected. In the past I was able to > >> specify which compiler was used by the F77 system variable. However, I > >> am now having a f90 compiler that exists on my Solaris system detected > >> regardless of the F77 value. Even unsetting the F77 variable leads to > >> the use of the f90 compiler. > >> > >> I was going to look though the distutil change logs but I was hoping > >> that someone might remember changing something off the top of their > heads. > > > > Which FORTRAN compilers do you have installed? What --fcompiler flag > > are you using? > > > Hi Robert, > > Thank you for your help. > > I am not using the --fcompiler flag. Okay, use the --fcompiler flag. That is the way to tell numpy to use a particular compiler if you have multiple ones installed. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FORTRAN compiler detection
Robert Kern wrote: > On Tue, Feb 26, 2008 at 11:50 AM, Christopher Hanley <[EMAIL PROTECTED]> > wrote: >> Greetings, >> >> I was wondering if within the last 8 - 10 weeks anyone has made changes >> to the way FORTRAN compilers are detected. In the past I was able to >> specify which compiler was used by the F77 system variable. However, I >> am now having a f90 compiler that exists on my Solaris system detected >> regardless of the F77 value. Even unsetting the F77 variable leads to >> the use of the f90 compiler. >> >> I was going to look though the distutil change logs but I was hoping >> that someone might remember changing something off the top of their heads. > > Which FORTRAN compilers do you have installed? What --fcompiler flag > are you using? > Hi Robert, Thank you for your help. I am not using the --fcompiler flag. The FORTRAN compilers I have installed are: f77: Sun WorkShop 6 update 2 FORTRAN 77 5.3 Patch 111691-07 2004/04/23 Usage: f77 [ options ] files. Use 'f77 -flags' for details basil> f90 -V f90: Sun WorkShop 6 update 2 Fortran 95 6.2 Patch 111690-10 2003/08/28 Usage: f90 [ options ] files. Use 'f90 -flags' for details basil> Chris -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FORTRAN compiler detection
On Tue, Feb 26, 2008 at 11:50 AM, Christopher Hanley <[EMAIL PROTECTED]> wrote: > Greetings, > > I was wondering if within the last 8 - 10 weeks anyone has made changes > to the way FORTRAN compilers are detected. In the past I was able to > specify which compiler was used by the F77 system variable. However, I > am now having a f90 compiler that exists on my Solaris system detected > regardless of the F77 value. Even unsetting the F77 variable leads to > the use of the f90 compiler. > > I was going to look though the distutil change logs but I was hoping > that someone might remember changing something off the top of their heads. Which FORTRAN compilers do you have installed? What --fcompiler flag are you using? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] FORTRAN compiler detection
Greetings, I was wondering if within the last 8 - 10 weeks anyone has made changes to the way FORTRAN compilers are detected. In the past I was able to specify which compiler was used by the F77 system variable. However, I am now having a f90 compiler that exists on my Solaris system detected regardless of the F77 value. Even unsetting the F77 variable leads to the use of the f90 compiler. I was going to look though the distutil change logs but I was hoping that someone might remember changing something off the top of their heads. Thank you for your time and help, Chris -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)
Hello,
I am trying to install numpy 1.0.4 to the following machine:
$ uname -a
Linux myhostname 2.6.9-55.ELsmp #1 SMP Sat Apr 21 11:16:24 EDT 2007 x86_64
x86_64 x86_64 GNU/Linux
and I am getting the error that appears in the subject of this email. It
appears that BLAS and LAPACK libraries cannot be found as well, but I think
numpy still shouldn't have problem with that. There must be some problem
with 64 bit. Please see the complete log.
---
Running from numpy source directory.
F2PY Version 2_4422
blas_opt_info:
blas_mkl_info:
libraries mkl,vml,guide not found in /usr/local/lib
libraries mkl,vml,guide not found in /usr/lib
NOT AVAILABLE
atlas_blas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
libraries ptf77blas,ptcblas,atlas not found in /usr/lib
NOT AVAILABLE
atlas_blas_info:
libraries f77blas,cblas,atlas not found in /usr/local/lib
libraries f77blas,cblas,atlas not found in /usr/lib/sse2
libraries f77blas,cblas,atlas not found in /usr/lib
NOT AVAILABLE
/.../numpy-1.0.4/numpy/distutils/system_info.py:1340: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.__doc__)
blas_info:
libraries blas not found in /usr/local/lib
FOUND:
libraries = ['blas']
library_dirs = ['/usr/lib']
language = f77
FOUND:
libraries = ['blas']
library_dirs = ['/usr/lib']
define_macros = [('NO_ATLAS_INFO', 1)]
language = f77
lapack_opt_info:
lapack_mkl_info:
mkl_info:
libraries mkl,vml,guide not found in /usr/local/lib
libraries mkl,vml,guide not found in /usr/lib
NOT AVAILABLE
NOT AVAILABLE
atlas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/local/lib
libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
libraries lapack_atlas not found in /usr/lib/sse2
libraries ptf77blas,ptcblas,atlas not found in /usr/lib
libraries lapack_atlas not found in /usr/lib
numpy.distutils.system_info.atlas_threads_info
NOT AVAILABLE
atlas_info:
libraries f77blas,cblas,atlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/local/lib
libraries f77blas,cblas,atlas not found in /usr/lib/sse2
libraries lapack_atlas not found in /usr/lib/sse2
libraries f77blas,cblas,atlas not found in /usr/lib
libraries lapack_atlas not found in /usr/lib
numpy.distutils.system_info.atlas_info
NOT AVAILABLE
/.../numpy-1.0.4/numpy/distutils/system_info.py:1247: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.__doc__)
lapack_info:
libraries lapack not found in /usr/local/lib
FOUND:
libraries = ['lapack']
library_dirs = ['/usr/lib']
language = f77
FOUND:
libraries = ['lapack', 'blas']
library_dirs = ['/usr/lib']
define_macros = [('NO_ATLAS_INFO', 1)]
language = f77
running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler
options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler
options
running build_src
building py_modules sources
building extension "numpy.core.multiarray" sources
Generating build/src.linux-x86_64-2.5/numpy/core/config.h
customize GnuFCompiler
Could not locate executable g77
Could not locate executable f77
customize IntelFCompiler
Could not locate executable ifort
Could not locate executable ifc
customize LaheyFCompiler
Could not locate executable lf95
customize PGroupFCompiler
Could not locate executable pgf90
Could not locate executable pgf77
customize AbsoftFCompiler
Could not locate executable f90
customize NAGFCompiler
Could not locate executable f95
customize VastFCompiler
customize GnuFCompiler
customize CompaqFCompiler
Could not locate executable fort
customize IntelItaniumFCompiler
Could not locate executable efort
Could not locate executable efc
customize IntelEM64TFCompiler
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize G95FCompiler
Could not locate executable g95
don't know how to compile Fortran code on platform 'posix'
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
-Wstrict-prototypes -fPIC
compile options: '-I/.../python2.5 -Inumpy/core/src -Inumpy/core/include
-I/.../python2.5 -c'
gcc: _configtest.c
In file included from /.../python2.5/Python.h:57,
from _con
Re: [Numpy-discussion] A little help please?
Neal Becker wrote: > My user-defined type project has mostly gone well, but I'm stuck on > mixed-type arithmetic. > > I have 2 types: cmplx_int32 and cmplx_int64. I have added basic arithmetic > for those types, and for mix of those arrays and their respective scalars. > But mixed arithmetic only partly works This is an area that needs testing and possible fixes. The relevant code is in ufuncobject.c (select_types) and in multiarraymodule.c (PyArray_CanCoerceScalar).If you can go through that code you may be able to see what the problem is and let us know. I tried to support this kind of thing you are doing, but I'm not sure how well I succeeded because I didn't have time or the code to test it with. Thus, there is still some work to do. The fact that radd is not called is because ufuncs try to handle everything (the ufunc is more general than just the functions with "r" prefixes. I think one problem may be due to the fact that the first argument to a ufunc is the one that defines the search for the correctly registered function and there may be no code to allow other arguments to direct the search should that one fail. I'm actually pleased you've gotten this far. I'll keep trying to help as I get time. -Travis O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Trouble With MaskedArray and Shared Masks
I'm having trouble with MaskedArray's _sharedmask flag. I would like to create a sub-view of a MaskedArray, fill it, and have the modifications reflected in the original array. This works with regular ndarrays, but only works with MaskedArrays if _sharedmask is set to False. Here's an example: >>> a = numpy.ma.MaskedArray( ... data=numpy.zeros((4,5), dtype=float), ... mask=numpy.ones((4,5), dtype=numpy.ma.MaskType), ... fill_value=0.0 ... ) >>> sub_a = a[:2,:3] >>> sub_a[0,0] = 1.0 >>> print sub_a [[1.0 -- --] [-- -- --]] >>> print a [[-- -- -- -- --] [-- -- -- -- --] [-- -- -- -- --] [-- -- -- -- --]] >>> print a.data [[ 1. 0. 0. 0. 0.] [ 0. 0. 0. 0. 0.] [ 0. 0. 0. 0. 0.] [ 0. 0. 0. 0. 0.]] The data array receives the new value, but the mask array does not. >>> a._sharedmask = False >>> sub_a = a[:2,:3] >>> sub_a[0,0] = 1.0 >>> print sub_a [[1.0 -- --] [-- -- --]] >>> print a [[1.0 -- -- -- --] [-- -- -- -- --] [-- -- -- -- --] [-- -- -- -- --]] This sort of (for me) unexpected behavior extends to other ways I've been using numpy arrays as well: a[:] = 1.0 (set to constant); a[:] = b (copy into); a[:5] = a[-5:] (rotating copy), etc. I wasn't seeing this behavior before because I was working on an array that had already been sliced and therefore "unshared", which caused a good deal of confusion for me when I started working on an array that wasn't the product of slicing. All of this leads me to some questions. What is the rational for initializing a new MaskedArray with _sharedmask=True when its mask isn't (actively) being shared yet? Is there a better way to say: "a=MaskedArray(...); a._sharedmask=False" that does not require touching a "private" attribute? Or am I going about this all wrong? What's the correct MaskedArray idioms for these actions that doesn't cause a new mask to be created? Thanks! Alex ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] A little help please?
My user-defined type project has mostly gone well, but I'm stuck on mixed-type arithmetic. I have 2 types: cmplx_int32 and cmplx_int64. I have added basic arithmetic for those types, and for mix of those arrays and their respective scalars. But mixed arithmetic only partly works. In [2]: a Out[2]: array([(0,0), (1,0), (2,0), (3,0), (4,0), (5,0), (6,0), (7,0), (8,0), (9,0)], dtype=cmplx_int32) In [3]: b Out[3]: array([(0,0), (1,0), (2,0), (3,0), (4,0), (5,0), (6,0), (7,0), (8,0), (9,0)], dtype=cmplx_int64) In [4]: a+b --- TypeError Traceback (most recent call last) /home/nbecker/numpy/ in () TypeError: function not supported for these types, and can't coerce safely to supported types In [5]: b+a Out[5]: array([(0,0), (2,0), (4,0), (6,0), (8,0), (10,0), (12,0), (14,0), (16,0), (18,0)], dtype=cmplx_int64) What I did: d1 is dtype cmplx_int32, d2 is dtype cmplx_int64. 1. PyArray_RegisterCastFunc (d1, d2->type_num, &cmplx_to_cmplx); That registers a conversion from cmplx_int32->cmplx_int64. (Docs never explain when this conversion is used, BTW) 2. PyArray_RegisterCanCast (d1, d2->type_num, PyArray_NOSCALAR); 3. d1->f->castdict = PyDict_New(); PyObject *key = PyInt_FromLong (d2->type_num); PyObject *cobj = PyCObject_FromVoidPtr ((void*)(void(* (void*,void*,npy_intp,void*,void*))&cmplx_to_cmplx, 0); PyDict_SetItem (d1->f->castdict, key, cobj); So now add (cmplx_int64, cmplx_int32) is OK, the 2nd arg is converted, but no attempt is made at radd. Obviously, I don't want to convert 64->32. Any clues what's missing here? ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.random.randint() inconsistent with plain random.randint()
Robert Kern wrote: > On Mon, Feb 25, 2008 at 2:58 PM, Christopher Kerr <[EMAIL PROTECTED]> > wrote: >> I don't know if this is the right place to report bugs, but I couldn't >> find >> anywhere else on the website... >> >> random.randint(min,max) from python core returns an integer between min >> and max inclusive. The documentation on the website says that >> numpy.random.randint(min,max [,size]) does this too, but it in fact only >> ever returns numbers strictly less than the max, and gives an error if >> min is equal to max > > The documentation on what website? It needs to be fixed. > numpy.random.randint() is behaving correctly. numpy.random is not > intended to replace the standard library's random module. > It looks like I might have misread the documentation the first time round. It still seems rather silly to have a function with the same name as one in the core distribution but with different semantics, though. (On the other hand, you could argue that randint() in the core distribution is inconsistent with the general policy in Python of describing ranges as half-open intervals.) Perhaps the easiest way to deal with this without breaking things add something in BIG BOLD LETTERS to the API docs saying that numpy.random.randint behaves differently to the core random.randint, so that other newbies don't hit the same problems as me (i.e. my Monte Carlo never touching one side of my array). ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Optimize speed of for loop using numpy
On 25/02/2008, Trond Kristiansen <[EMAIL PROTECTED]> wrote: > I have attached the function that the FOR loop is part of as a python file. > What I am trying to do is to create a set of functions that will read the > output files (NetCDF) from running the ROMS model (ocean model). The output > file is organized in xi (x-direction), eta (y-direction), and s > (z-direction) where the s-values are vertical layers and not depth. This > particular function (z_slice) will find the closest upper and lower s-layer > for a given depth in meters (e.g. -100). Then values from the two selcted > layers will be interpolated to create a new layer at the selected depth > (-100). The problem is that the s-layers follow the bathymetry and a > particular s-layer will therefore sometimes be above and sometimes below the > selected depth that we want to interpolate to. That's why I need a quick > script that searches all of the layers and find the upper and lower layers > for a given depth value (which is negative). The z_r is a 3D array > (s,eta,xi) that is created using the netcdf module. If I understand what you're doing correctly, you have a 3D array of depths, indexed by two unproblematic coordinates (eta and xi), and by a third coordinate whose values are peculiar. For each pair (eta, xi), you want to find the value of the third coordinate that occurs at a depth closest to some given depth. Are you looking to do this for many values of depth? It seems like what you're trying to do is (approximately) invert a function - you have z as a function of s, and you want s as a function of z. Is the mapping between depths and s values monotonic? First of all, numpy.searchsorted() is the right tool for finding where depth occurs in a list of z values. It gives you the position of the next lower value; it's pretty straightforward to write down a linear interpolation (you should be able to do it without any for loop at all) and more sophisticated interpolation schemes may be straightforward as well. If you want a smoother interpolation scheme, you may want to look at scipy.interpolate. It is not always as vectorial as you might wish, but it implements splines (optionally with smoothing) quite efficiently. I believe scipy.ndimage may well have some suitable interpolation code as well. In my opinion, the value of numpy/scipy comes from the tools that allow you to express major parts of your algorithm as a line or two of code. But it is often difficult to take advantage of this by "trying to accelerate for loops". I find it really pays to step back and look at my algorithm and think how to express it as uniform operations on arrays. (Of course it helps to have a rough idea of what operations are available; the numpy documentation is sometimes sadly lacking in terms of letting you know what tools are there.) You might find useful the new http://www.scipy.org/Numpy_Functions_by_Category Good luck, Anne ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
