Re: LinearAlgebra incredibly slow for eigenvalue problems
drife [EMAIL PROTECTED] writes: Hello, I need to calculate the eigenvectors and eigenvalues for a 3600 X 3600 covariance matrix. The LinearAlgebra package in Python is incredibly slow to perform the above calculations (about 1.5 hours). This in spite of the fact that I have installed Numeric with the full ATLAS and LAPACK libraries. Also note that my computer has dual Pentium IV (3.1 GHz) processors with 2Gb ram. Every Web discussion I have seen about such issues indicates that one can expect huge speed ups if one compiles and installs Numeric linked against the ATLAS and LAPACK libraries. Are you *sure* that Numeric is linked against these? Even more perplexing is that the same calculation takes a mere 7 min in Matlab V6.5. Matlab uses both ATLAS and LAPACK. Moreover, the above calculation takes the same amount of time for Numeric to complete with --and-- without ATLAS and PACK. I am certain that I have done the install correctly. This is good evidence that Numeric *isn't* linked to them. If you're on a Linux system, you can check with ldd: [EMAIL PROTECTED] ldd /usr/lib/python2.3/site-packages/Numeric/lapack_lite.so liblapack.so.3 = /usr/lib/atlas/liblapack.so.3 (0x002a95677000) libblas.so.3 = /usr/lib/atlas/libblas.so.3 (0x002a95e55000) libg2c.so.0 = /usr/lib/libg2c.so.0 (0x002a96721000) libpthread.so.0 = /lib/libpthread.so.0 (0x002a96842000) libc.so.6 = /lib/libc.so.6 (0x002a96957000) libm.so.6 = /lib/libm.so.6 (0x002a96b96000) /lib64/ld-linux-x86-64.so.2 = /lib64/ld-linux-x86-64.so.2 (0x00552000) You can see that lapack and blas (the Atlas versions) are linked to the lapack_lite.so. To install Numeric using Lapack: - remove the build/ directory in your Numeric sources, so you don't any old binaries - edit setup.py and follow the comments on using Lapack (you need to comment out a few lines, and set some directories) Also set use_dotblas to 1. - do the 'python setup.py build', 'python setup.py install' dance. -- ||\/| /--\ |David M. Cooke |cookedm(at)physics(dot)mcmaster(dot)ca -- http://mail.python.org/mailman/listinfo/python-list
Re: LinearAlgebra incredibly slow for eigenvalue problems
drife == drife [EMAIL PROTECTED] writes: drife Hi David, I performed the above check, and sure enough, drife Numeric is --not-- linked to the ATLAS libraries. drife I followed each of your steps outlined above, and Numeric drife still is not linking to the ATLAS libraries. drife My setup.py file is attached below. Are you sure that you 1) 'rm -rf build' before rebuilding 2) are using the python / Numeric that you think you are, eg, might you have one python in /usr/bin and another in /usr/local. root paths can differ from user paths which may effect which python is used at install time versus run time -- http://mail.python.org/mailman/listinfo/python-list
Re: LinearAlgebra incredibly slow for eigenvalue problems
Hi John, I do have more than one version of Python laying around. To do the build and install I am typing: /d2/python/bin/python setup.by build ! build.out /d2/python/bin/python setup.by install ! install.out Should I be doing something different? Daran -- http://mail.python.org/mailman/listinfo/python-list
Re: LinearAlgebra incredibly slow for eigenvalue problems
drife [EMAIL PROTECTED] writes:
Hi David,
I performed the above check, and sure enough, Numeric
is --not-- linked to the ATLAS libraries.
I followed each of your steps outlined above, and Numeric
still is not linking to the ATLAS libraries.
My setup.py file is attached below.
# delete all but the first one in this list if using your own LAPACK/BLAS
sourcelist = [os.path.join('Src', 'lapack_litemodule.c')]
# set these to use your own BLAS;
library_dirs_list = ['/d2/lib/atlas']
libraries_list = ['lapack', 'ptcblas', 'ptf77blas', 'atlas', 'g2c']
# set to true (1), if you also want BLAS optimized
matrixmultiply/dot/innerproduct
use_dotblas = 1
include_dirs = ['/d2/include']
This all look right (assuming you've got the right stuff in /d2).
When it compiles, does it look like it's actually doing the linking?
After doing python setup.py build, you can run ldd on the libraries in
the build directory (something like build/lib.linux-i386-2.3/lapack_lite.so).
If that's linked, then it's not being installed right.
You don't have a previous Numeric installation that's being picked up
instead of the one you're trying to install, do you?
At the interpreter prompt, check that
import Numeric
Numeric.__file__
gives you something you're expecting, and not something else.
--
||\/|
/--\
|David M. Cooke
|cookedm(at)physics(dot)mcmaster(dot)ca
--
http://mail.python.org/mailman/listinfo/python-list
Re: LinearAlgebra incredibly slow for eigenvalue problems
Hi David, Yes, when Numeric compiles it does look like the linking is being done properly. I captured the build to a file and see a few lines similar to: gcc -pthread -shared build/temp.linux-i686-2.4/Src/lapack_litemodule.o -L/d2/lib/atlas -llapack -lptcblas -lptf77blas -latlas -lg2c -o build/lib.linux-i686-2.4/lapack_lite.so I checked the files in build/lib.linux-i686-2.4 and none have dependencies on ATLAS. When I run Python interpreter with: import Numeric Numeric.__file__ I get the answer I expect. I am totall baffled. Any other ideas? Thanks for your help, Daran -- http://mail.python.org/mailman/listinfo/python-list
Re: LinearAlgebra incredibly slow for eigenvalue problems
David, One more thing. I checked to see if the SciPy libraries had dependencies on ATLAS. They do not, however, the eigenvector calculation is still much faster than Numeric? This is very strange. Daran -- http://mail.python.org/mailman/listinfo/python-list
Re: LinearAlgebra incredibly slow for eigenvalue problems
David, I noticed that the libraries that ATLAS builds are not shared objects (e.g., liblapack.a). Should these be shared objects? I see nothing in the ATLAS documentation about building things as shared objects. Wondering if this is why the Numeric install is failing. Daran -- http://mail.python.org/mailman/listinfo/python-list
Re: LinearAlgebra incredibly slow for eigenvalue problems
drife wrote: David, I noticed that the libraries that ATLAS builds are not shared objects (e.g., liblapack.a). Should these be shared objects? I see nothing in the ATLAS documentation about building things as shared objects. Wondering if this is why the Numeric install is failing. No, they should work just fine as static libraries. -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter -- http://mail.python.org/mailman/listinfo/python-list
