Re: PythonMagick on Windows
Adam Endicott wrote: > Does anyone know anything about PythonMagick? I've got a project that > could use ImageMagick, so I went looking around for PythonMagick, and > I'm a bit lost. > > I was able to download the PythonMagick source from the ImageMagick > site, but I'm on Windows and don't have the ability to compile it. I > found reference to a windows installer for PythonMagick on the wxPython > wiki, but the links take me to procoders.net, which seems to have > abandoned the project (at least I couldn't find anything to download). > > So what's the state of this project? > > Actually, I'm wondering if I would gain anything by using PythonMagick > anyway. I'm going to be distributing a py2exe created executable that > needs to do some image manipulation. Would I still need ImageMagick > seperately installed on the client machine to use PythonMagick anyway? > Or could it all be self contained? If I'd need to install ImageMagick > on client machines anyway, I guess I can just use the commandline > interface to ImageMagick from Python. > There is a boost-less interface to imagemagick on the pylab project page at Sourceforge. Last I checked it compiled for windows. But, I haven't really played with it for a while. You might find the old binaries there useful for at least some version of Image Magick. http://sourceforge.net/project/showfiles.php?group_id=1315 -Travis Oliphant -- http://mail.python.org/mailman/listinfo/python-list
Re: Best library for basic stats
MKoool wrote: > Is there any decent library for basic stats? I am just looking for the > basics, like quartiles, median, standard deviation, mean, min, max, > regression, etc. > > I was going to use SciPy, but I use Python 2.4, and it seems to be > supported primarily for 2.3... > > What other statistical packages have any level of public support at > this point? > > thanks I would definitely look at SciPy. It works fine with Python 2.4. The new scipy_core has basic stats (mean, min, max, standard deviation) built in as methods to the array object. A new scipy_core release will be made in a few days. But, you can get started with the release already available. Look for the download link at http://numeric.scipy.org For *much* more stats look at full scipy (you can just check out the stats sub-package if you want). The full scipy can be checked out of SVN anonymously (or browsed through the web) at http://svn.scipy.org/svn/scipy/trunk -Travis Oliphant -- http://mail.python.org/mailman/listinfo/python-list
Re: csrss.exe & Numeric
jelle wrote: > I have a function that uses the Numeric module. When I launch the > function csrss.exe consumes 60 / 70 % cpu power rather than having > python / Numeric run at full speed. Has anyone encountered this problem > before? It seriously messes up my Numeric performance. > Are you memory-limited so that the process is swapping memory to disk? We'll need more details to offer a better suggestion. Best, -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: Hypergeometric distribution
Raven wrote: > Thanks Steven for your very interesting post. > > This was a critical instance from my problem: > > from scipy import comb >>> comb(14354,174) > > inf > > The scipy.stats.distributions.hypergeom function uses the scipy.comb > function, so it returned nan since it tries to divide an infinite. I > did not tried to write a self-made function using standard python as I > supposed that the scipy functions reached python's limits but I was > wrong, what a fool :-) Notice the keyword for the comb function (in scipy) lets you use it to compute exact values. SciPy does not just automatically use the long integer because this will always slow you down. comb(N, k, exact=0) Combinations of N things taken k at a time. If exact==0, then floating point precision is used, otherwise exact long integer is computed. Notes: - Array arguments accepted only for exact=0 case. - If k > N, N < 0, or k < 0, then a 0 is returned. -Travis Oliphant -- http://mail.python.org/mailman/listinfo/python-list
Re: void * C array to a Numpy array using Swig
Philip Austin wrote: > "Travis E. Oliphant" <[EMAIL PROTECTED]> writes: > > >>Krish wrote: > > >>Yes, you are right that you need to use typemaps. It's been awhile >>since I did this kind of thing, but here are some pointers. > > > Also, there's http://geosci.uchicago.edu/csc/numptr > This is interesting. I've noticed his getpointerX functions seem to be implemented already by PyArray_AsCArray in the new NumPy. -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: how do "real" python programmers work?
bblais wrote: > In Python, there seems to be a couple ways of doing things. I could > write it in one window, and from a Unix shell call >python myscript.py > and be like C++, but then I lose the interactiveness which makes > prototyping easier. If I use the python shell, I can use import (and > reload), or execfile perhaps. Try IPython. It makes the process of executing live code very productive. http://ipython.scipy.org -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: void * C array to a Numpy array using Swig
Krish wrote:
> Hello People
>
> I hope I am On Topic. Anyways, here is my problem. Any insights would
> be really appreciated.
Posting to the [EMAIL PROTECTED] list would help
generate more responses, I think.
>
> I have wrapped a C IO module using SWIG -> Python Module. Suppose the
> name of the module is "imageio" and the reader function from the file
> is image_read() which returns an object ( "filled" C structure) with a
> lot of members.
>
> Let
>
> a = imageio.image_read("testfile.image")
>
> Now a is the object which contains pointer to data of the image. ( void
> * pdata). ( a also contains the datatype which I need to typecast to
> get the data pointed by pdata). My question is how to access the data
> pointed by pdata in *Python*.
Yes, you are right that you need to use typemaps. It's been awhile
since I did this kind of thing, but here are some pointers.
1) Look at Michael Sanner's typemaps at
http://www.scripps.edu/mb/olson/people/sanner/html/Python/NumericTypemaps/
Except for the CHAR version, these should work for NumPy by replacing
Numeric/arrayobject.h with numpy/arrayobject.h
2) In full scipy there are typemaps for numpy arrays in
cluster/src/swig_num.i
Look here...
http://projects.scipy.org/scipy/scipy/file/trunk/Lib/cluster/src/swig_num.i
This should help you get started with some examples. Typemaps can be a
little confusing at first, but they do make your interface a bit nicer.
-Travis
--
http://mail.python.org/mailman/listinfo/python-list
Re: Numarray, numeric, NumPy, scpy_core ??!!
J wrote: > I will just jump in an use NumPy. I hope this one will stick and evolve > into the mother of array packages. > How stable is it ? For now I really just need basic linear algebra. > i.e. matrix multiplication, dot, cross etc > There is a new release coming out this weekend. It's closer to 1.0 and so should be more stable.It also has some speed improvements in matrix-vector operations (if you have ATLAS BLAS --- or if you download a binary version with ATLAS BLAS compiled in). I would wait for it. -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: Possible memory leak?
Tuvas wrote: > I have a function in a program that works something like this. > > def load_pic_data(width,heigth,inpdat, filt=TRUE): > data='' > total=0 > tnum=0 > size=100 > for y in range(0,heigth): > row='' > for x in range(0,width): > index=2*(x+y*width) > num=ord(inpdat[index+1])*256+ord(inpdat[index]) > if(vfilter.get() and d_filter and filt): > num=round((num-(d_filter[index/2]))) > if(num<0): > num=0 > if(num>255*64): > num=255*64 > row=row+chr(num/64) > data=data+row > > > The purpose of this part of a program is to take a 14 bit numerical > representation and convert it to an 8 bit representation. This is exactly the kind of thing NumPy (http://numeric.scipy.org) is ideal for. If that's of interest to you, then I can offer particular suggestions... -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a way to profile underlying C++ code?
Bo Peng wrote: > Dear list, > > I have a C++-SWIG-wrapped python module. The running time for one of the > functions is pretty unpredictable so I would like to profile it. > However, the python profiler does not seem to enter the compiled module > (.so file). Is there a way to profile the C++ functions? > On Linux you can use oprofile (which is pretty nice and easy to use --- no recompiling. Just start the profiler, run your code, and stop the profiler). http://oprofile.sourceforge.net/about/ -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: writing arrays to binary file
Sheldon wrote:
> Hi everyone,
>
> I have a short program the writes the output data to an acsii file:
>
> import sys
> import os
> import string
> import gisdata
> from Numeric import *
>
> def latlon(area_id, i, j):
> lon_file = "lon_%s.dat" % area_id
> flon= open(lon_file, 'wa')
> lat_file = "lat_%s.dat" % area_id
> flat = open(lat_file, 'wa')
> for x in range(i):
> for y in range(j):
> flat.write("%f\n" % gisdata.xy2lonlat(area_id,x,y)[1])
> flon.write("%f\n" % gisdata.xy2lonlat(area_id,x,y)[0])
> flat.close()
> flon.close()
> #
> if __name__ == '__main__':
> tile_info ={"ibla_35n13e":[1215,1215],
>"ibla_46n16e":[1215,1215],
>"ibla_57n40w":[1215,1215],
>}
> for t in tile_info.keys():
> xsize = tile_info[t][0]
> ysize = tile_info[t][1]
> result = latlon_(t, xsize, ysize)
> Now this works but the output is in ascii form. Whenever I try to write
> it binary form by creating the 2D array and then opening the file with
> open("filename", 'wb'), I lose the decimals that are vital. The input
> data is float with about 4 decimal places.
You need to use the tostring() method and write that out in binary form.
flon = open(lon_file,'wb')
flon.write(your2darray.tostring())
of course you will need to manage byte-order issues if you will be
transferring this file to a different kind of processor.
Alternatively, with new NumPy there is a tofile method that you can use
to write in either ascii or binary form.
-Travis Oliphant
--
http://mail.python.org/mailman/listinfo/python-list
Re: Scientific Computing with NumPy
mclaugb wrote: > Is Scipy the same thing as ScientificPython? No. They are two separate projects. Scientific Python is still Numeric-only. SciPy 0.3.x is Numeric-based and SciPy 0.4.x is NumPy-based. The developers for NumPy are also the developers for SciPy (for the most part). There is a mailing list [EMAIL PROTECTED] where all of your questions can be answered. -- http://mail.python.org/mailman/listinfo/python-list
Re: Scientific Computing with NumPy
Tim Hochberg wrote: > mclaugb wrote: > > > No, Scientific Python is "a collection of Python modules that are useful > for scientific computing" written by Konrad Hinsen. I'm not a user, but > you can find information here: > > http://starship.python.net/~hinsen/ScientificPython/ > > SciPy is, I believe, a more recent and more ambitious project. In any > event it is not directly related to ScientficPython. Information here: > SciPy's been around since 2001. ScientificPython since about 1998, I think. > Numeric: This is the origingal array package. > > Numarray: This was written as a replacement for Numeric. It has improved > performance for large arrays. The internals were also simplified and > many other improvements were made (arrays were subclassable, numeric > signal handling vastly improved, etc). Unfortunately, a side effect of > the changes was that small array performance got worse. There was a > signifigant chunk of the numeric community for whom this was a deal > breaker and as a result there ended up being a split between the Numeric > and Numarray communities. Just to be clear, small array performance was only 1 of the "deal-breaker" problems with numarray. Anothber big problem for SciPy was that the Numeric C-API was never fully supported (in particular the Ufunc C-API). This made it more difficult to convert to numarray. Thus, porting never happened. The small-array speed issue just made porting that much less enticing. > > Numpy: This is a rewrite of Numeric that incorporates most of the > improvements in Numarray. The goal is to incorporate *all* of the improvements (unless being written in Python is one of the improvements). If there are missing improvements we need to know about them. Thanks to Tim for spreading some light on the issue. There will no doubt be continued confusion for new users over the coming months. Hopefully, with time the confusion will fade as more people use NumPy and any remainging issues get resolved. -Travis -- http://mail.python.org/mailman/listinfo/python-list
ANN: Release of NumPy 0.9.5
NumPy is the successor to both Numeric and Numarray. It builds from and uses code from both. More information can be found at the following links http://numeric.scipy.org http://www.scipy.org http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 Highlights of Release: - Unicode arrays are now always UCS4 even on narrow Python builds - Many improvements and bugfixes to dtype objects - Improvements to numpy.distutils - Optimized dot function improved so copies are no longer made for transposed arrays - Many more bug-fixes Full release notes at http://sourceforge.net/project/shownotes.php?release_id=394206&group_id=1369 Best regards, -Travis Oliphant -- http://mail.python.org/mailman/listinfo/python-list
Re: calculating on matrix indices
Brian Blais wrote: > Hello, > > In my attempt to learn python, migrating from matlab, I have the following > problem. > Here is what I want to do, (with the wrong syntax): > > from numpy import * > > t=arange(0,20,.1) > x=zeros(len(t),'f') > > idx=(t>5) > tau=5 > x[idx]=exp(-t[idx]/tau) # <---this line is wrong (gives a TypeError) Hi Brian, Just to let you know, you are more likely to get good answers by mailing [EMAIL PROTECTED] then posting to this list. The indexing technique you are using is actually fine. The trouble you are having is that t is a double-precision array (and thus so is exp(-t[idx]/tau) while x is a single-precision array. Compare: x.dtype.name t.dtype.name Because you are trying to store something with greater precision into an array with less precision you get a TypeError. So, either change x to be double precision: x = zeros(len(t), float) #the default Python float is doubleprecision or change t to be single precision: t = arange(0,20,.1,dtype=single) Good luck, -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: ANN: Release of NumPy 0.9.5
Thomas Gellekum wrote: > "Travis E. Oliphant" <[EMAIL PROTECTED]> writes: > > >> - Improvements to numpy.distutils > > > Stupid questions: is it really necessary to keep your own copy of > distutils and even install it? What's wrong with the version in the > Python distribution? How can I disable the colorized output to get > something more readable than yellow on white (well, seashell1)? > Yes --- distutils does not provide enough functionality. Besides, it's not a *copy* of distutils. It's enhancements to distutils. It builds on top of standard distutils. I don't know the answer to the colorized output question. Please post to [EMAIL PROTECTED] to get more help and/or make suggestions. -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: ANN: Release of NumPy 0.9.5
vinjvinj wrote: > I read some of the earlier threads which essentially said that numpy is > about 3-4 times slower then Numeric for smaller arrays. I'm assuming > that applies only to operations that apply to the whole arrays. > > I was curious how the performance of the following operations would > compare though: > 1. Copying arrays > 2. Creating arrays > 3. Slicing Arrays > It is really hard to say unless you show specific applications: But, here are some facts. 1) NumPy is in C just like Numeric 2) NumPy is more flexible than Numeric and that means a few more if-statements and checks which can slow-down array creation and slicing. 3) Some optimizations have been done to remove unnecessary slowness (e.g. unnecessary checking but more could be done). 4) Ufuncs have more overhead primarily because of a) the ability to alter how errors are obtained using local / global variables (these must be looked-up in a dictionary which is most of the source of any performance penalty), b) the ability for other objects that can be converted to arrays to interact with ufuncs so that the original object is returned by the ufuncs c) the fact that excessive copying is not done in order to type-cast from one array to annother, this requires some "set-up" overhead My understanding is that most of the "slow-downs" reported for small arrays can be attributed to ufuncs, but there will be slower paths through the C-code for other cases as well because of the increased flexibility. Our goal is to be as fast as we can be so give it a spin and tell us how to improve... Thanks, -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
[EMAIL PROTECTED] wrote: > I'm running operations large arrays of floats, approx 25,000 x 80. > Python (scipy) does not seem to come close to using 4GB of wired mem, > but segments at around a gig. Everything works fine on smaller batches > of data around 10,000 x 80 and uses a max of ~600mb of mem. Any Ideas? > Is this just too much data for scipy? > > Thanks Conor > > Traceback (most recent call last): > File "C:\Temp\CR_2\run.py", line 68, in ? >net.rProp(1.2, .5, .01, 50.0, input, output, 1) > File "/Users/conorrob/Desktop/CR_2/Network.py", line 230, in rProp >print scipy.trace(error*scipy.transpose(error)) > File "D:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line > 149, in > __mul__ >return N.dot(self, other) > MemoryError You should ask this question on the numpy-discussion list for better feedback. Does it actually segfault or give you this Memory Error? Temporary arrays that need to be created could be the source of the extra memory. Generally, you should be able to use all the memory on your system (unless you are on a 64-bit system and are not using Python 2.5). -Travis -- http://mail.python.org/mailman/listinfo/python-list
ANN: NumPy 0.9.8 released
NumPy 0.9.8 has been released. It can be downloaded from
http://numeric.scipy.org
The release notes are attached.
Best regards,
-Travis Oliphant
NumPy 0.9.8 is a bug-fix and optimization release with a
few new features. The C-API was changed so that extensions compiled
against NumPy 0.9.6 will need re-compilation to avoid errors.
The C-API should be stabilizing. The next release will be 1.0 which
will come out in a series of release-candidates during Summer 2006.
There were many users and developers who contributed to the fixes for
this release. They deserve much praise and thanks. For details see
the Trac pages where bugs are reported and fixed.
http://projects.scipy.org/scipy/numpy/
New features (and changes):
* string and unicode comparisons now work on array objects
without having to go through the chararray.
* string and unicode scalars selected from an array have
NULL bytes removed so that comparisons work correctly.
* changed fortran= keywords to order= keywords
The order= keyword accepts 'C', 'FORTRAN', or None
as arguments. The order= keyword also accepts True for
'FORTRAN' and False for 'C' for backwards compatibility.
* The error-lookup for math functions was changed to work on a
per-thread basis instead of a local, module (global), bulitin
name-space basis.
* PyArray_CHAR now works as does the 'c' code for specifying a
1-element string. This improves compatibility with Numeric
when PyArray_CHAR and typecode='c' are used.
Now array("mystr", 'c') works the same as it did in Numeric.
* where(condition) and condition.nonzero() always return
tuples. nonzero(condition) is for backwards compatibility
with Numeric and only works with 1-d arrays.
* overflow checking is nolonger done an array multiplication for
consistency with addition and subtraction.
* math module added to numpy namespace as it used to be in the Numeric
name-space. numpy.emath has extended math functions
* matrices return correctly shaped matrices for reduction-like methods
and scalars for reduction over the entire array (i.e. A.argmax()
returns a scalar.)
* numpy should install now with easy_install from setuptools
* masked array improvements including more methods added.
Speed ups:
* scalarmath module added to speed up math on array scalars (the
objects returned when indexing into arrays).
* a.flags is now a builtin object
* copying code was sped up significantly for well-behaved cases.
* indexing a 1-d array by an integer has been sped-up.
Other fixes:
* fixed problem with bad arguments to .transpose() not being caught.
* fix problem with .argmax(axis) and .argmin(axis) for multi-dimensional
arrays.
* cov and corrcoef fixed to work correctly and not
in-place modify the input.
* several fixes for numpy.distutils were applied
* errors involving reshape and fortran-order arrays fixed
* fixed several errors in optimized blasdot function
* several segfaults fixed
* record array pickling and byteswapped pickling fixed.
* fix sorting on byteswapped arrays
* fancy-indexing no longer alters the index array.
* divbyzero should work now on optimizing compilers.
* vectorize segfaults should be fixed.
* b.shape = now fails as it should
for non-contiguous arrays
* fixed errors apperaing in use of flattened and byteswapped
arrays
* several memory leaks closed and other Valgrind-discovered
errors fixed.
* fixed attribute access for record arrays and their sub-classes
--
http://mail.python.org/mailman/listinfo/python-list
Re: Scipy: vectorized function does not take scalars as arguments
ago wrote: > Once I vectorize a function it does not acccept scalars anymore. Es > > def f(x): return x*2 > vf = vectorize(f) > print vf(2) > > AttributeError: 'int' object has no attribute 'astype' > > Is this the intended behaviour? > Vectorize handles scalars in recent versions of NumPy. Which version do you have? -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: Looking for triangulator/interpolator
Grant Edwards wrote: > I need to interpolate an irregularly spaced set of sampled > points: Given a set of x,y,z points, I need to interpolate z > values for a much finer x,y grid. How many x,y,z points do you have? Did you try the fitpack function bisplrep in scipy? It can work well as long as you don't have too many starting points. Here is an example of how to use it from numpy import rand, exp, ogrid from scipy import interpolate x = 2*rand(20)-1 y = 2*rand(20)-1 z = (x+y)*exp(-6.0*(x*x+y*y)) tck = interpolate.bisplrep(x,y,z,s=0,xb=-1,xe=1,yb=-1,ye=1) xnew = r_[-1:1:70j] ynew = r_[-1:1:70j] znew = interpolate.bisplev(xnew,ynew,tck) There is a buglet that is fixed in SVN scipy that means you need to enter xb, xe, yb, and ye manually. -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: Looking for triangulator/interpolator
Grant Edwards wrote: > On 2006-05-27, Travis E. Oliphant <[EMAIL PROTECTED]> wrote: > >>> I need to interpolate an irregularly spaced set of sampled >>> points: Given a set of x,y,z points, I need to interpolate z >>> values for a much finer x,y grid. >> How many x,y,z points do you have? > > I've got about 700 data points. They're arranged irregularly > along arcs arranged in sort of truncated fan-shape. > >> Did you try the fitpack function bisplrep in scipy? > > Not recently. I spent a month or so wrestling with bisplrep a > while back, but was unable to get stable results (either with > real world data or with the example from the tutorial): > > http://www.visi.com/~grante/scipy/interpolate.html > http://www.scipy.net/pipermail/scipy-user/2004-December/003921.html > > I switched to a trianguation scheme shortly after that posting, > and haven't tired bisplrep since then. > Not that you made a bad choice. I do wonder, how much of your difficulty was with the interface to the underlying fitpack routines. The interface is pretty involved as there are lots of parameter choices to the underlying routine. It's very possible the interface is broken in some strange way. Given how many people want to do interpolation. I'm surprised nobody has looked into fixing up the fitpack routines that do it. I suppose it's possible that the underlying fitpack routines are faulty in 2-d. Your examples will help in that study. Pearu made some nice class-based interfaces to fitpack in 2003. They are in SciPy, but, it appears that people aren't making much use of them. Did you try them? They are in fitpack2.py There are new wrappers for it -- http://mail.python.org/mailman/listinfo/python-list
Re: matplotlib and numpy installation
cesco wrote: > Hi, > > I wanted to install python, numpy and matplotlib on Linux Ubuntu. > I installed python with the following commands > ./configure --enable-unicode=ucs4 > [snip] > running build_ext > building 'matplotlib.backends._ns_backend_agg' extension > gcc options: '-pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall > -Wstrict-prot\ > otypes -fPIC' > compile options: > '-I/usr/local/lib/python2.4/site-packages/numpy/core/include -\ > I/usr/include -I. -Isrc -Iswig -Iagg23/include -I. -I/usr/include -I. > -I/usr/lo\ > cal/lib/python2.4/site-packages/numpy/core/include/freetype2 > -I/usr/include/fre\ > etype2 -I./freetype2 -Isrc/freetype2 -Iswig/freetype2 > -Iagg23/include/freetype2\ > -I./freetype2 -I/usr/include/freetype2 -I./freetype2 > -I/usr/local/include/pyth\ > on2.4 -c' > extra options: '-DSCIPY=1' > gcc: src/_image.cpp > src/_image.cpp:5:17: png.h: No such file or directory This is the source of the remaining errors On my system the "png.h" file comes from a libpng3-devel rpm Perhaps you need to use apt-get to obtain the development package for png. -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: Looking for triangulator/interpolator
Grant Edwards wrote: > On 2006-05-27, Travis E. Oliphant <[EMAIL PROTECTED]> wrote: > >> Not that you made a bad choice. I do wonder, how much of your >> difficulty was with the interface to the underlying fitpack >> routines. > > I've no idea. I had never done anything with splines before, > so it's quite possible I just wasn't doing things right. I > never got it to work at all for non-gridded data (which is what > I needed). Since it wasn't stable even for gridded data, I > more or less gave up. > >> The interface is pretty involved as there are lots of >> parameter choices to the underlying routine. It's very >> possible the interface is broken in some strange way. > > I took a look at underlying Fortran code, but that made my > dizzy. Just to finish this thread, I should mention I spent some time yesterday looking into the details of the underlying surfit fucntion used by bisplrep. It appears to have difficulties when s=0 is used. It seems to be geared toward smoothing applications where you are trying to fit "noisy" scattered data to a smooth function.There are many warnings about choosing s to be too low. Thus, trying to use bisplrep for interpolation (instead of data smoothing) is probably going to be difficult with bisplrep. We still need a good N-d re-gridding algorithm in SciPy. -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: reordering elements of a list
greenflame wrote:
> I am trying to reorder elements of a list and I am stuck as to what
> might be the best way to approach this. I have a (main) list of
> elements and another (ordering) list (which is may shorter, but not
> longer than the main list) which contains the order in which I want the
> elements of the main list but only as far along as the length of the
> ordering list. This may be confusing so I will try to give an example.
>
> Suppose the main list is: mainlist = list('qwertyuiop')
>
> Suppose the ordering list is: orderinglist = [3, 4, 2, 1]
>
> Then I am looking for a function that will take mainlist and
> orderinglist as arguments and return the following list:
>
> ['e', 'r', 'w', 'q', 't', 'y', 'u', 'i', 'o', 'p']
>
> Also by the way the main list is always going to be a list of strings
> and the ordering list will be a list of numbers. Also the largest
> number in orderinglist will always be equal to the length of
> orderinglist. I hope this makes any sense. Thanks for your help.
>
NumPy ( http://numeric.scipy.org ) can do this using element-based
indexing on an array of strings.
Your example:
import numpy
a = numpy.array('qwertyuiop','c')
newlist = a[[2,3,1,0]+range(4,10)].tolist()
print newlist
Returns:
['e', 'r', 'w', 'q', 't', 'y', 'u', 'i', 'o', 'p']
But you can also do it with list comprehension pretty easily, so this is
probably just a shameless plug for NumPy :-)
-Travis
--
http://mail.python.org/mailman/listinfo/python-list
Re: Scientific Computing with NumPy
[EMAIL PROTECTED] wrote: > Terry Reedy wrote: > >>"kpp9c" <[EMAIL PROTECTED]> wrote in message >>news:[EMAIL PROTECTED] >> >>>Numeric, Numarray, & Numpy... some one stick a screwdriver in my >>>forehead and end the madness that Numeric has become. >> >>>For crying all night! Numpy was Numeric's nickname >> >>Given that NumPy *is* an updated version of Numeric, with the same basic >>interface, I think the name quite appropriate. It also works well as a >>partner for SciPy. Exactly! I suppose people must vent, but the truth is that satisfying the diversity of scientific Python users is *very* hard. NumPy *is* trying to do something about the confusion between Numeric and Numarray. I'm sorry you don't like the name, but it was properly discussed and a host of other names were suggested before settling on NumPy as the *least bad*. Yes, it will be confusing for a few months, but then it will probably be *less* confusing as people realize that NumPy *is* Numeric 3K The interface is not that different. For example, you can usually compile extensions simply by replacing Numeric/arrayobject.h with numpy/arrayobject.h If you don't like things then start suggesting specific improvements at [EMAIL PROTECTED] instead of just ranting on the python list which most of us don't have the time to read anyway (notice how late I'm responding to this...) -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: Trouble with numpy-0.9.4 and numpy-0.9.5
drife wrote: > Hello, > > I use the Python Numeric package extensively, and had been an > avid user of the "old" scipy. In my view, both pieces of software > are truly first rate, and have greatly improved my productivity in > the area of scientific analysis. Thus, I was excited to make the > transition to the new scipy core (numpy). > > Unfortunately, I am experiencing a problem that I cannot sort > out. I am running Python 2.4.2 on a Debian box (V3.1), using > gcc version 3.3.5, and ATLAS, BLAS, and LAPACK libraries > built from scratch. When building numpy everything seems to > go AOK. But trouble soon follows. > > As a sanity check, I run the diagnostic tests that come as > part of the numpy package: > > from numpy import * > from scipy import * > > test(level=N) # N can vary from 1 to 10. : > > > No matter which test I run, Python crashes hard with only > the following output: "Floating exception". > This could be related to the Debian glibc bug that has been discussed before. Aparently the Debian version of glibc had some issues with SSE. This is the same behavior experienced by others on the Debian platform. There is a patch, but I'm not sure what it is. See, for example: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/2207861 Come over to the [EMAIL PROTECTED] and/or [EMAIL PROTECTED] lists for more help. -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: installing numpy
cesco wrote: > Hi, > > I'm trying to install the numpy library (precisely > numpy-0.9.6-py2.4-linux-i686) on Linux but I encounter several This is a dumb pre-built binary package (useful perhaps because it links against ATLAS already) built using distutils. You don't build it and install it using setup.py you just un-tar it in the / directory and start using it. You only run python setup.py install when you have a source file. You can download the sources if you want and build it yourself using: python setup.py install > Finally I tried to move the folder numpy to the location where all the > third-party modules are installed, that in my case is > /usr/lib/python2.4/site-packages/ and run the command from there like: > python numpy/setup.py install Presumably, you can copy the files directly there and then start using it (don't run setup.py though, there is no need). After copying the files over to your site-packages directory, just start python and do import numpy -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: switching to numpy and failing, a user story
[EMAIL PROTECTED] wrote: > After using numeric for almost ten years, I decided to attempt to > switch a large codebase (python and C++) to using numpy. Here's are > some comments about how that went. > > - The code to automatically switch python stuff over just kind of > works. But it was a 90% solution, I could do the rest by hand. Of > course, the problem is that then the code is still using the old > numeric API, so it's not a long term solution. Unfortunately, to switch > to the numpy API one needs documentation, which is a problem; see > below. I'm glad to hear of your experiences (good and bad). We need feedback exactly from users like you in order to improve things. Yep, the code converter is a 80% solution, but it does work. Improvements are always welcome. > > - Well, ok, the automatic switching code doesn't really work all that > well... my uses of RandomArray still work, but they generate different > numbers. The underlying random-number generator must have changed. I'm > sure that it's "better" now, but it's different. This is a major pain > for my regression tests that rely on seeding the random number > generator and getting particular results. But that's ok, I can update > the regressions for the new RNG. > You definitely can't expect the same random number generator. The new one (thanks to Robert Kern) is quite good. > - My extension modules just won't build because the new numpy stuff > lives in a different location from where Numeric used to live. This is an easy one and is documented in lots of places on the new http://www.scipy.org site. Plus, people are always willing to help out if you just ask. The numpy-discussion list is quite active. Don't be shy. > - I guess I should just buy the documentation. I don't like this idea, > because I think it's counter-productive to the project to have payware > docs (would Python be successful if you had to buy the documentation? I > don't think so), but that's the way this project goes. It's probably better to call it "complete documentation." Normal open-source documentation is available from http://www.scipy.org. There are lots of people who have helped it. I had to do something to at least pretend to justify the time NumPy took me to the people who care about how I spend my time (including my family). This was the best I could come up with. I'm doubly > unhappy about it because they payment system is using Paypal and I > don't like Paypal at all, but I guess that's just the way it goes. Oh, > wait, I *can't* buy the docs because I'm not in the US and the payment > page requires a US address. Is that really true? A lot of people not living in the U.S. have used Paypal successfully. When difficulties arise, communicating with me your difficulty is usually productive. I give up; I guess NumPy is only for people living in the US. Definitely not true. People in Singapore, Japan, Ghana, South Africa, France, Germany, New Zealand, Australia, and many other countries are using NumPy successfully. Gratefully, a few have contributed by buying the book, but a lot more have downloaded and are successfully using it. I'm sorry about your experience, you definitely don't *have* to buy the book to use NumPy. Just like you don't *have* to buy any Python book to use Python. The amount of documentation for NumPy is growing and I expect that trend to continue. There is a lot of information in the source file. > > I guess I'll come back to NumPy in 2010, when the docs are available. > Or just ask on the mailing lists, use the numpy.oldnumeric interface (the differences are all documented in the first few pages of my book which is available for free now). Thanks, -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: How to convert this list to string?
Jia Lu wrote: > Hi all > > I have a list like: > list > [1, 2, 3] list[1:] > [2, 3] > > I want to get a string "2 3" > str(list[1:]) > '[2, 3]' > > How can I do that ? > " ".join(str(x) for x in list) -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: using mmap on large (> 2 Gig) files
Martin v. Löwis wrote: > [EMAIL PROTECTED] schrieb: >> Anyone ever done this? It looks like Python2.4 won't take a length arg >>> 2 Gig since its not seen as an int. > > What architecture are you on? On a 32-bit architecture, it's likely > impossible to map in 2GiB, anyway (since it likely won't fit into the > available address space). > > On a 64-bit architecture, this is a known limitation of Python 2.4: > you can't have containers with more than 2Gi items. This limitation > was removed in Python 2.5, so I recommend to upgrade. Notice that > the code has seen little testing, due to lack of proper hardware, NumPy uses the mmap object and I saw a paper at SciPy 2006 that used Python 2.5 + mmap + numpy to do some pretty nice and relatively fast manipulations of very large data sets. So, the very useful changes by Martin have seen more testing than he is probably aware of. -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: numpy error
Juergen Kareta wrote: > Hello, > > this is my first try to get wxmpl-1.2.8 running. Therefor I installed: > > python 2.5 > matplotlib-0.87.6.win32-py2.5.exe > numpy-1.0rc3.win32-py2.5.exe > > on WinXP SP2 > > The result is a version mismatch (see below). > > Numpy version 102 seems to be numpy-1.0b5 which is not downloadable > anymore. Any hints ? > You need to re-compile matplotlib (or wait until they release a new binary that is compiled against numpy 1.0) Or install the numpy-1.0rc2 binary which is now again available on sourceforge. -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: numbers to string
David Isaac wrote:
y
> [116, 114, 121, 32, 116, 104, 105, 115]
z=''.join(chr(yi) for yi in y)
z
> 'try this'
>
> What is an efficient way to do this if y is much longer?
> (A numpy solution is fine.)
Here's another numpy solution just for fun:
import numpy
z = numpy.array(y,dtype='u1').view('S%d' % len(y))[0]
-Travis
--
http://mail.python.org/mailman/listinfo/python-list
Re: unsigned 32 bit arithmetic type?
Robin Becker wrote: > Hi, just trying to avoid wheel reinvention. I have need of an unsigned 32 bit > arithmetic type to carry out a checksum operation and wondered if anyone had > already defined such a beast. > > Our current code works with 32 bit cpu's, but is failing with 64 bit > comparisons; it's clearly wrong as we are comparing a number with a negated > number; the bits might drop off in 32 bits, but not in 64. NumPy defines an array and an array scalar for all signed and unsigned integers defined in C. import numpy as nx type(nx.array([1,2,3],dtype=nx.uint32)[0]) -- http://mail.python.org/mailman/listinfo/python-list
Re: problem with fft periodogram
[EMAIL PROTECTED] wrote: > Hello, > > I am ploting a fft periodogram of my data using a script (found in > internet: http://linuxgazette.net/115/andreasen.html ) that gave me > good results before. There should be a periodicity of >160 in the data > as can be seen by eye. However, this script now says that there is > periodicity of ~9. Can somebody explain me what's wrong? Thank you in > advance. The units are wrong on your period axis. Right now you have them in units of "sample-spacing". So, the plot is telling you that you have a periodicity of ~9 sample spacings. To get it in unites of minutes you need to multiply period by the difference in minutes period_in_minutes = period * (minutes[1] - minutes[0]) Then, plot period_in_minutes versus power. I see a peak around 180 minutes in your data. -Travis -- http://mail.python.org/mailman/listinfo/python-list
[ANN] NumPy 1.0 release
We are very pleased to announce the release of NumPy 1.0 available for download at http://www.numpy.org This release is the culmination of over 18 months of effort to allow unification of the Numeric and Numarray communities. NumPy provides the features of both packages as well as comparable speeds in the domains where both were considered fast --- often beating both packages on certain problems. If there is an area where we can speed up NumPy then we are interested in hearing about the solution. NumPy is essentially a re-write of Numeric to include the features of Numarray plus more. NumPy is a C-based extension module to Python that provides an N-dimensional array object (ndarray), a collection of fast math functions, basic linear algebra, array-producing random number generators, and basic Fourier transform capabilities. Also included with NumPy are: 1) A data-type object. The data-type of all NumPy arrays are defined by a data-type object that describes how the block of memory that makes up an element of the array is to be interpreted. Supported are all basic C-types, structures containing C-types, arrays of C-types, and structures containing structures of C-types. Data-types can also be in big or little-endian order. NumPy arrays can therefore be constructed from any regularly-sized chunk of data. A chunk of data can also be a pointer to a Python object and therefore Object arrays can be constructed (including record arrays with object members). 2) Array scalars: there is a Python scalar object (inheriting from the standard object where possible) defined for every basic data-type that an array can have. 2) A matrix object so that '*' is re-defined as matrix-multiplication and '**' as matrix-power. 3) A character array object that can replace Numarray's similarly-named object. It is basically an array of strings (or unicode) with methods matching the string and unicode methods. 4) A record array that builds on the advanced data-type support of the basic array object to allow field access using attribute look-up as well as to provide more ways to build-up a record-array. 5) A memory-map object that makes it easier to use memory-mapped areas as the memory for an array object. 6) A basic container class that uses the ndarray as a member. This often facilitates multiple-inheritance. 7) A large collection of basic functions on the array. 8) Compatibility layer for Numeric including code to help in the conversion to NumPy and full C-API support. 9) Compatibility layer for NumPy including code to help in the conversion to NumPy and full C-API support. NumPy can work with Numeric and Numarray installed and while the three array objects are different to Python, they can all share each other's data through the use of the array interface. As the developers for Numeric we can definitively say development of Numeric has ceased as has effective support. You may still find an answer to a question or two and Numeric will be available for download as long as Sourceforge is around so and code written to Numeric will still work, but there will not be "official" releases of Numeric for future versions of Python (including Python2.5). The development of NumPy has been supported by the people at STScI who created Numarray and support it. They have started to port their applications to NumPy and have indicated that support for Numarray will be phased out over the next year. You are strongly encouraged to move to NumPy. The whole point of NumPy is to unite the Numeric/Numarray development and user communities. We have done our part in releasing NumPy 1.0 and doing our best to make the transistion as easy as possible. Please support us by adopting NumPy. If you have trouble with that, please let us know why so that we can address the problems you identify. Even better, help us in fixing the problems. New users should download NumPy first unless they need an older package to work with third party code. Third-party package writers should migrate to use NumPy. Though it is not difficult, there are some things that have to be altered. Several people are available to help with that process, just ask (we will do it free for open source code and as work-for-hire for commercial code). This release would not have been possible without the work of many people. Thanks go to (if we have missed your contribution please let us know): * Travis Oliphant for the majority of the code adaptation (blame him for code problems :-) ) * Jim Hugunin, Paul Dubois, Konrad Hinsen, David Ascher, Jim Fulton and many others for Numeric on which the code is based. * Perry Greenfield, J Todd Miller, Rick White, Paul Barrett for Numarray which gave much inspiration and showed the way forward. * Paul Dubois for Masked Arrays * Pearu Peterson for f2py and numpy.distutils and help with code organization * Robert Kern for mtrand, bug fixes, help
Re: latest numpy & scipy - incompatible ?
robert wrote: > I'm using latest numpy & scipy. What is this problem ? : > import scipy.stats > RuntimeError: module compiled against version 102 of C-API but this > version of numpy is 109 > Traceback (most recent call last): > File "", line 1, in ? > File "C:\PYTHON23\Lib\site-packages\scipy\stats\__init__.py", line 7, in ? > from stats import * > File "C:\PYTHON23\Lib\site-packages\scipy\stats\stats.py", line 191, in ? > import scipy.special as special > File "C:\PYTHON23\Lib\site-packages\scipy\special\__init__.py", line 8, in ? > from basic import * > File "C:\PYTHON23\Lib\site-packages\scipy\special\basic.py", line 8, in ? > from _cephes import * > ImportError: numpy.core.multiarray failed to import import numpy.core.multiarray import scipy scipy.__version__ > '0.5.1' numpy.__version__ > '1.0' Download the 1.0rc2 binary until SciPy 0.5.2 which should be out in a few days. If you can recompile, SciPy yourself, then it will work fine with 1.0 -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenating numpy arrays
Rolf Wester wrote: > Hi, > > I want to concatenate two numpy arrays with shape (n1,n2) and (n1,n3) > into a single array with shape (n1,n2+n3). I guess there is an elegant > way to do this but I couldn't figure it out. So any help is very much > appreciated. > Suppose a1.shape is (n1,n2) and a2.shape is (n1,n3) Then you want to do a3 = concatenate((a1,a2),axis=1) or equivalently a3 = hstack([a1,a2]) a3 = r_['1',a1,a2] -Travis -- http://mail.python.org/mailman/listinfo/python-list
ANN: NumPy 1.0.2
We are pleased to announce the release of NumPy 1.0.2 NumPy is a Python extension that provides a general-purpose multi-dimensional array that can act as a container of arbitrary raw-data formats (including bools, integers, floats, complex, Python objects, string, unicode, and general C-structures). NumPy also provides fast array processing and mathematics on these multi-dimensional arrays. In addition, NumPy provides tools that allows these arrays to form the foundation for basic image and signal processing, linear algebra and Fourier transforms, and random number generation. NumPy also includes easy integration with ctypes and Fortran allowing seamless use of NumPy arrays with extensions written in either C/C++ or Fortran. The new support in NumPy for completely general (including nested structure) data-types allows it to be used as a container object for any number of applications where memory efficiency and speed are critical. For example, the ability to handle general data-formats means that NumPy could be used as an interface to general databases (see PyTables for one example). Get the latest version of NumPy by clicking on the Download link at http://numpy.scipy.org Enjoy!! -NumPy Developers -- http://mail.python.org/mailman/listinfo/python-list
Re: numpy migration (also posted to numpy-discussion)
Duncan Smith wrote: > Hello, > Since moving to numpy I've had a few problems with my existing > code. It basically revolves around the numpy scalar types. e.g. > You will probably get more help on the numpy discussion list: [EMAIL PROTECTED] You are encountering problems because numpy scalar types don't raise errors (unless you have set the appropriate hardware flag using numpy.seterr). You can get Python scalars out of NumPy arrays if you really want them using (for example...) a.item(0,0) > > An additional problem involves classes that have e.g. __rmul__ methods > defined and are sufficiently similar to numpy arrays that my classes' > __rmul__ methods are not invoked when using numpy scalars. > Could you please post an example showing the problem? > > I might (I hope) be missing something obvious; but it seems like, to be > safe, I'm going to have to do a lot of explicit conversions to Python > types (or abandon catching zero division errors, and documenting some of > my classes to highlight that whether scalar * a equals a * scalar > depends on whether a.__rmul__ is called, which depends on the type of > scalar). > numpy scalars are try a lot more things before giving up on multiplication and letting the other class have a stab at it. Post your problems to the numpy discussion list for better help and more discussion. -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: Sorting Multidimesional array(newbie)
Tartifola wrote:
> Hi,
> how can I sort an array like
>
> array([[5, 2],
>[1, 3]])
>
> along the first column to obtain
>
> array([[1, 3],
>[5, 2]])
> i.e. keeping track of the ordered couples?
>
> Thanks,
> A
Just to add one more solution to this question that works with numpy.
You can also view the array using fields and then sort (which will do
lexicographic ordering) and convert the result back to the original view.
Something like this:
a = array([[5,2],[1,3]])
dt = a.dtype
g = a.view([('',dt),('',dt)])
g.sort(0)
a = g.view(dt)
-Travis
--
http://mail.python.org/mailman/listinfo/python-list
[ANN] NumPy 1.0b4 now available
The 4th beta release of NumPy 1.0 has just been made available. NumPy 1.0 represents the culmination of over 18 months of work to unify the Numeric and Numarray array packages into a single best-of-breed array package for Python. NumPy supports all the features of Numeric and Numarray with a healthy dose of it's own improved features. It's time to start porting your applications to use NumPy as Numeric is no longer maintained and Numarray will only be maintained for a few more months. Porting is not difficult especially using the compatibility layers numpy.oldnumeric and numpy.numarray and the alter_code1.py modules in those packages. The full C-API of Numeric is supported as is the C-API of Numarray. More information is available at http://numpy.scipy.org NumPy Developers -- http://mail.python.org/mailman/listinfo/python-list
Re: [ANN] NumPy 1.0b4 now available
Bruce Who wrote: > Hi, Travis > > I can pack my scripts into an executable with py2exe, but errors occur > once it runs: I suspect you need to force-include the numpy/core/_internal.py file by specifying it in your setup.py file as explained on the py2exe site. That module is only imported by the multiarraymodule.c file which I suspect py2exe can't automatically discern. In 1.0 we removed the package-loader issues which are probably giving the scipy-style subpackage errors. So, very likely you might be O.K. with the beta releases of 1.0 as long as you tell py2exe about numpy/core/_internal.py so that it gets included in the distribution. Please post any successes. Best, -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: Random Drawing Simulation -- performance issue
Brendon Towle wrote: > I need to simulate scenarios like the following: "You have a deck of > 3 orange cards, 5 yellow cards, and 2 blue cards. You draw a card, > replace it, and repeat N times." > Thinking about the problem as drawing sample froms a discrete distribution defined by the population might help. For example, in SciPy you can define your own discrete random variable: var = scipy.stats.rv_discrete(name='sample', values=([0,1,2],[3/10.,5/10.,2/10.])) Then var.rvs(size=1) would quickly return an array of "draws" If you didn't want to install SciPy, but were willing to install NumPy, then the crux of the algorithm is to generate an entire array of uniform random variables: numpy.random.rand(count) and then map them through the inverse cumulative distribution function to generate your samples. The inverse cumulative distribution function is just a series of steps whose width depends on the probablity distribution. Thus, the population defines the draw boundaries. To make it easy, just multiply the uniform random number by the total number of cards and then the boundaries are on the integers of the number of each kind of card. Here is an implementation. I find this version to be 2x - 5x faster depending on how many draws are used. import numpy as N def randomDrawing_N(count, population): probs = [x[0] for x in population] res = [[0, item[1]] for item in population] nums = N.random.rand(count) cprobs = N.cumsum([0]+probs) nums *= cprobs[-1] for k, val in enumerate(probs): res[k][0] += ((nums > cprobs[k]) & (nums < cprobs[k+1])).sum() return res -Travis Oliphant -- http://mail.python.org/mailman/listinfo/python-list
Re: Trouble Passing Array of Strings (using Numeric) to C Extension Module
goetzie wrote:
> I am using Python 2.4.1 and Numeric 23.8 and running on Windows XP. I
> am passing a Numeric array of strings (objects) to a C Extension module
> using the following python code:
Numeric 23.8 is *very* old and unsupported. Unless you absolutely have
to use Numeric (then use 24.2), NumPy is a better choice for new code.
With that advice aside. Let's see...
>
> And here is the relevent code from my C Extension module:
>
>static PyObject * _StringArrayIn( PyObject *self, PyObject *args )
>{
> PyObject *pObject; // input array
> PyArrayObject *pArray; // contiguous array
> int iCount;
> int iStride;
> BOOL bString;
>
> if ( !PyArg_ParseTuple( args, "O", &pObject ) ) return NULL;
>
> if ( ( pArray = ( PyArrayObject * )PyArray_ContiguousFromObject(
> pObject, PyArray_OBJECT, 1, 1 ) ) == NULL ) return NULL;
>
> iCount = pArray->dimensions[0];
> iStride = pArray->strides[0];
>
> bString = PyString_Check( ( PyObject * )( pArray->data ) );
>
This is the problem.:
pArray->data should be interpreted as (PyObject **) -- an array of
PyObject *'s, and then de-referenced to get the PyObject * present at
the first address
So. this should work to check that the first entry in the array is a
string:
PyString_Check( *( ( PyObject ** )( pArray->data ) ) );
By the way, NumPy has support for true string (and unicode) arrays (as
well as object arrays like this)...
-Travis
--
http://mail.python.org/mailman/listinfo/python-list
Re: Newbie - converting csv files to arrays in NumPy - Matlab vs. Numpy comparison
oyekomova wrote:
> Thanks for your help. I compared the following code in NumPy with the
> csvread in Matlab for a very large csv file. Matlab read the file in
> 577 seconds. On the other hand, this code below kept running for over 2
> hours. Can this program be made more efficient? FYI - The csv file was
> a simple 6 column file with a header row and more than a million
> records.
>
>
There is some facility to read simply-formatted files directly into NumPy.
You might try something like this.
numpy.fromfile('somename.csv', sep=',')
and then reshape the array.
-Travis
--
http://mail.python.org/mailman/listinfo/python-list
Re: Newbie - converting csv files to arrays in NumPy - Matlab vs. Numpy comparison
oyekomova wrote:
> Thanks for your help. I compared the following code in NumPy with the
> csvread in Matlab for a very large csv file. Matlab read the file in
> 577 seconds. On the other hand, this code below kept running for over 2
> hours. Can this program be made more efficient? FYI - The csv file was
> a simple 6 column file with a header row and more than a million
> records.
>
>
> import csv
> from numpy import array
> import time
> t1=time.clock()
> file_to_read = file('somename.csv','r')
> read_from = csv.reader(file_to_read)
> read_from.next()
>
> datalist = [ map(float, row[:]) for row in read_from ]
>
> # now the real data
> data = array(datalist, dtype = float)
>
> elapsed=time.clock()-t1
> print elapsed
>
If you use numpy.fromfile, you need to skip past the initial header row
yourself. Something like this:
fid = open('somename.csv')
data = numpy.fromfile(fid, sep=',').reshape(-1,6)
# for 6-column data.
-Travis
--
http://mail.python.org/mailman/listinfo/python-list
Re: Newbie - converting csv files to arrays in NumPy - Matlab vs. Numpy comparison
oyekomova wrote: > Thanks to everyone for their excellent suggestions. I was able to > acheive the following results with all your suggestions. However, I am > unable to cross file size of 6 million rows. I would appreciate any > helpful suggestions on avoiding memory errors. None of the solutions > posted was able to cross this limit. Did you try using numpy.fromfile ? This will not require you to allocate more memory than needed. If you specify a count, it will also not have to re-allocate memory in blocks as the array size grows. It's limitation is that it is not a very sophisticated csv reader (it only understands a single separator (plus line-feeds are typically seen as a separator). -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: numpy or _numpy or Numeric?
auditory wrote:
> I am a newbie here
>
> I am trying to read "space separated floating point data" from file
>
> I read about csv module by searching this group,
> but I couldn't read space separated values with csv.
> (which may be matter of course..)
>
> I also read about numpy.fromfile(file, sep=' ') which i can use.
> but on my machine(ubuntu linux) numpy is unknown module,
> which I didn't install by myself.
You will need to install NumPy.
>
> While trying to install numpy accroding to its homepage.
> (http://numpy.scipy.org/numpydoc/numdoc.htm).
> i am quite confused.
You are reading old documentation for Numeric and so any installation
description is how to install the Numeric module (not its newer
replacement which is called NumPy).
So:
1) Yes, you need NumPy
2) This *is different* from Numeric
3) You get it by either installing a pre-built package for your system
or by
a) downloading the source tar-file from
http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103
(get the numpy-.tar.gz file
b) tar zxvf numpy-.tar.gz
c) cd numpy-
d) sudo python setup.py install
e) If you want to link against high-performance libraries on your
system, then either put them in standard locations or edit the site.cfg
file appropriately (Optional).
>
> 4. Or what is general way to read 'space separated values' from file?
You can easily read space-separated values from a file by reading in a
line at a time and using the split method of strings:
fid = open('filename')
linedata = fid.readlines()
new = [[float(x) for x in line.split()] for line in linedata]
new will be a nested sequence of floats. You can convert it to an array
(if you want to do math on it) using
anew = numpy.array(new)
-Travis
--
http://mail.python.org/mailman/listinfo/python-list
Re: numpy or _numpy or Numeric?
auditory wrote: >>> While trying to install numpy accroding to its homepage. >>> (http://numpy.scipy.org/numpydoc/numdoc.htm). >>> i am quite confused. >> You are reading old documentation for Numeric and so any installation >> description is how to install the Numeric module (not its newer >> replacement which is called NumPy). > > I found the above website from googling with "numpy" keyword, > and supprised at that the top matching page is old one. > The website is not the old one. http://numpy.scipy.org is correct. But, the documentation is old. But, it is still useful (except for Numeric specific information like how to install it). Therefore we still make a link to it. -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: numpy : argmin in multidimensional arrays
TG wrote: > Hi there. > > I am working with multi-dimensional arrays and I need to get > coordinates of the min value in it. > > using myarray.argmin() returns the index in the flatten array, which is > a first step, but I wonder if it is possible to get the coordinates > directly as an array, rather than calculating them myself by using this > flat index and the shape of the array. > > well, in fact i'm not sure to understand how argmin(myarray) works, > when myarray is multidimensional. By default, the argmin method flattens the array and returns the flat index. You can get the corresponding element using myarray.flat[index] Alternatively, you can use the function unravel_index unravel_index(flat_index, myarray.shape) to return an N-dimensional index. If you give an axis argument, then the minimum is found along the specified dimension and you get an N-1 dimensional array of indices that will all be between 1 and myarray.shape[axis] -Travis -- http://mail.python.org/mailman/listinfo/python-list
