Re: [Numpy-discussion] NumPy 1.0.3 release next week
On Fri, 11 May 2007, Travis Oliphant wrote: Thanks for the ticket reviews, Albert. That is really helpful. My pleasure. Found two more issues that look like they could be addressed: http://projects.scipy.org/scipy/numpy/ticket/422 http://projects.scipy.org/scipy/numpy/ticket/450 Cheers, Albert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy 1.0.3 release next week
Hello all On Fri, 11 May 2007, David M. Cooke wrote: I've added a 1.0.3 milestone and set these to them (or to 1.1, according to Travis's comments). I've reviewed some more tickets and filed everything that looks like it can be resolved for this release under 1.0.3. To see which tickets are still outstanding (some need fixes, some can just be closed if appropriate), take a look at the list under 1.0.3 Release Release on this page: http://projects.scipy.org/scipy/numpy/report/3 Tickets marked with the 1.0.3 milestone that aren't going to be fixed, but should be, should have their milestone changed to 1.1 (and maybe we should add a 1.0.4 milestone too). Regards, Albert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] best way of counting time and cputime?
It depends on what you're aiming at. If you want to compare different implementations of some expressions and need to know their average execution times you should use the timeit module. If you want to have the full execution time of a script, time.time (call at the begin and end, compute the difference, gives the elapsed time) and time.clock (under linux the cpu clock time used by the process) are the methods you want. Cheers! Bernhard On May 12, 12:22 am, dmitrey [EMAIL PROTECTED] wrote: hi all, please inform me which way for counting elapsed time and cputime in Python is best? Previously in some Python sources I noticed there are some ones. Currently I use time.time() for time. Thx, D. ___ Numpy-discussion mailing list [EMAIL PROTECTED]://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy 1.0.3 release next week
I've more or less finished my quick triage effort. Issues remaining to be resolved for the 1.0.3 release: http://projects.scipy.org/scipy/numpy/query?status=newstatus=assignedstatus=reopenedmilestone=1.0.3+Release If they can't be fixed for this release, we should move them over to 1.1 or maybe 1.0.4 (when it is created) if they can be fixed soon but not now. There are a few tickets that don't have a milestone yet: http://projects.scipy.org/scipy/numpy/query?status=newstatus=assignedstatus=reopenedmilestone= The roadmap also shows a better picture of what's going on: http://projects.scipy.org/scipy/numpy/roadmap?show=all Cheers, Albert On Thu, 10 May 2007, Travis Oliphant wrote: Hi all, I'd like to relase NumPy 1.0.3 next week (on Tuesday) along with a new release of SciPy. Please let me know of changes that you are planning on making before then. Best, -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Getting started wiki page
Hi all, I would very much link the Getting Started wiki page ( http://scipy.org/Getting_Started ) to the front page. But I am not sure it is of good enough quality so far. Could people please have a look and make comments, or edit the page. Cheers, Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-user] Getting started wiki page
I would very much link the Getting Started wiki page ( http://scipy.org/Getting_Started ) to the front page. But I am not sure it is of good enough quality so far. Could people please have a look and make comments, or edit the page. Thank you for doing this. It's pitched very well. Matthew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy 1.0.3 release next week
On 5/12/07, Albert Strasheim [EMAIL PROTECTED] wrote: I've more or less finished my quick triage effort. Issues remaining to be resolved for the 1.0.3 release: http://projects.scipy.org/scipy/numpy/query?status=newstatus=assignedstatus=reopenedmilestone=1.0.3+Release If they can't be fixed for this release, we should move them over to 1.1 or maybe 1.0.4 (when it is created) if they can be fixed soon but not now. There are a few tickets that don't have a milestone yet: http://projects.scipy.org/scipy/numpy/query?status=newstatus=assignedstatus=reopenedmilestone= The roadmap also shows a better picture of what's going on: http://projects.scipy.org/scipy/numpy/roadmap?show=all Thanks, Albert. The tickets look much better organized now. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] best way of counting time and cputime?
On 5/12/07, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: It depends on what you're aiming at. If you want to compare different implementations of some expressions and need to know their average execution times you should use the timeit module. If you want to have the full execution time of a script, time.time (call at the begin and end, compute the difference, gives the elapsed time) and time.clock (under linux the cpu clock time used by the process) are the methods you want. Don't use time.clock, it has a nasty wraparound problem that will give you negative times after a while, and effectively junk for very long running processes. This is much safer (from IPython.genutils): def clock(): clock() - floating point number Return the *TOTAL USER+SYSTEM* CPU time in seconds since the start of the process. This is done via a call to resource.getrusage, so it avoids the wraparound problems in time.clock(). u,s = resource.getrusage(resource.RUSAGE_SELF)[:2] return u+s The versions for only user or only system time are obvious, and they're already in IPython.genutils as well: http://projects.scipy.org/ipython/ipython/browser/ipython/trunk/IPython/genutils.py#L165 Cheers, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-user] Getting started wiki page
On 5/12/07, Gael Varoquaux [EMAIL PROTECTED] wrote: Hi all, I would very much link the Getting Started wiki page ( http://scipy.org/Getting_Started ) to the front page. But I am not sure it is of good enough quality so far. Could people please have a look and make comments, or edit the page. Great work! Thanks a lot for putting time into this, which is extremely useful to newcomers. One minor nit: I think it would be better to more prominently mention the -pylab switch right at the begginning of your FFT example. The reason is that without it, plotting is really nearly unusable, so rather than 1. show really doesn't-works-well approach 2. show solution later I think it would be best to start with the -pylab approach from the start. You can mention that -pylab is only needed if you want plotting and requires matplotlib. Just my 1e-2, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-user] Getting started wiki page
On Sat, May 12, 2007 at 12:12:21PM -0600, Fernando Perez wrote: Thanks a lot for putting time into this, which is extremely useful to newcomers. I got bored of always explaining the same things to project students :-. I think it would be best to start with the -pylab approach from the start. You can mention that -pylab is only needed if you want plotting and requires matplotlib. I think you are right. The biggest problem is that (at least with the enthon python distribution) there is no icon under windows to start ipython with the -pylab switch, and many people have no clue what we are talking about. I change as you suggested, though. Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] best way of counting time and cputime?
Is the genutils module not included to standard CPython edition? First of all I'm interested in what is the best way for latter, for to users not need installing anything else. Thx, D. Fernando Perez wrote: On 5/12/07, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: It depends on what you're aiming at. If you want to compare different implementations of some expressions and need to know their average execution times you should use the timeit module. If you want to have the full execution time of a script, time.time (call at the begin and end, compute the difference, gives the elapsed time) and time.clock (under linux the cpu clock time used by the process) are the methods you want. Don't use time.clock, it has a nasty wraparound problem that will give you negative times after a while, and effectively junk for very long running processes. This is much safer (from IPython.genutils): def clock(): clock() - floating point number Return the *TOTAL USER+SYSTEM* CPU time in seconds since the start of the process. This is done via a call to resource.getrusage, so it avoids the wraparound problems in time.clock(). u,s = resource.getrusage(resource.RUSAGE_SELF)[:2] return u+s The versions for only user or only system time are obvious, and they're already in IPython.genutils as well: http://projects.scipy.org/ipython/ipython/browser/ipython/trunk/IPython/genutils.py#L165 Cheers, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-user] Getting started wiki page
You can add the -pylab switch to the desktop shortcut under Windows. I had created a Windows IPython installer that automatically creates a second entry under Start All Programs IPython that includes the -pylab -p scipy option. You can download my installer from here: http://www.siue.edu/~rkrauss/ipython-0.7.3.svn.win32.exe but as the name implies it is 0.7.3 which is quite old. Ville said he was going to put this in the standard windows executable from now on. Ryan On 5/12/07, Gael Varoquaux [EMAIL PROTECTED] wrote: On Sat, May 12, 2007 at 12:12:21PM -0600, Fernando Perez wrote: Thanks a lot for putting time into this, which is extremely useful to newcomers. I got bored of always explaining the same things to project students :-. I think it would be best to start with the -pylab approach from the start. You can mention that -pylab is only needed if you want plotting and requires matplotlib. I think you are right. The biggest problem is that (at least with the enthon python distribution) there is no icon under windows to start ipython with the -pylab switch, and many people have no clue what we are talking about. I change as you suggested, though. Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-user] Getting started wiki page
Seems like it might be convenient for IPython to detect if matplotlib is installed and if it is then to use pylab mode by default (unless specified otherwise with a switch like -nopylab). Brian On 5/12/07, Ryan Krauss [EMAIL PROTECTED] wrote: You can add the -pylab switch to the desktop shortcut under Windows. I had created a Windows IPython installer that automatically creates a second entry under Start All Programs IPython that includes the -pylab -p scipy option. You can download my installer from here: http://www.siue.edu/~rkrauss/ipython-0.7.3.svn.win32.exe but as the name implies it is 0.7.3 which is quite old. Ville said he was going to put this in the standard windows executable from now on. Ryan On 5/12/07, Gael Varoquaux [EMAIL PROTECTED] wrote: On Sat, May 12, 2007 at 12:12:21PM -0600, Fernando Perez wrote: Thanks a lot for putting time into this, which is extremely useful to newcomers. I got bored of always explaining the same things to project students :-. I think it would be best to start with the -pylab approach from the start. You can mention that -pylab is only needed if you want plotting and requires matplotlib. I think you are right. The biggest problem is that (at least with the enthon python distribution) there is no icon under windows to start ipython with the -pylab switch, and many people have no clue what we are talking about. I change as you suggested, though. Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-user] Getting started wiki page
Brian Hawthorne wrote: Seems like it might be convenient for IPython to detect if matplotlib is installed and if it is then to use pylab mode by default (unless specified otherwise with a switch like -nopylab). That's a bad idea. IPython has some magic, but it shouldn't be that magical. Just because a package is installed doesn't mean that the user wants it loaded every time with GUI threads and all of the other finicky stuff that comes along with -pylab. IPython is a general purpose tool, not a front end to matplotlib. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] problems with calculating numpy.float64
Hello out there, i try to run this Python-code snippet after I have imported: import numpy as Numeric import numpy as numpy Numeric.Int = Numeric.int32 Numeric.Float = Numeric.float64 Code: if m maxN and n maxN and self.activeWide[m+1, n+1]: try: deltaX = x[m+1] - x[m] except TypeError: print '#' * 70 print 'type x', type(x) type_a, type_b = map(type, (x[m + 1], x[m])) print 'type_a, type_b', type_a, type_b, type_a is type_b print '#' * 70 raise My result for this part looks like this: type 'numpy.ndarray' type 'numpy.float64' type 'numpy.float64' True Inappropriate argument type. unsupported operand type(s) for -: 'numpy.float64' and 'numpy.float64' line 161 in setIDs in file F:\Unstrut_2007\arc_egmo\gw.py line 71 in __init__ in file F:\Unstrut_2007\arc_egmo\mf2000\mf2000exe.py line 380 in makeInactiveFringeMap in file F:\Unstrut_2007\arc_egmo\mf2000\mf2000exe.py This means that Python cant execute the line: deltaX = x[m+1] - x[m] You can see the Types of the values between the --- --- two lines above. I reproduce this problem in my Python 2.4 interpreter on my Windows XP Prof. OS and there are no problems in calculating with these Types, everything is OK. Thanks for your help Greetz from Berlin, Germany Michael ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] problems with calculating numpy.float64
[EMAIL PROTECTED] wrote: Hello out there, i try to run this Python-code snippet after I have imported: Can you try to come up with a small, self-contained example? I can't replicate your problem. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] very large matrices.
Hi, I have test data of about 75000 x 75000 dimensions. I need to do svd, or at least an eigen decomp on this data. from search suggests to me that the linalg functions in scipy and numpy don't work on sparse matrices. I can't even get empty((1,1),dtype=float) to work (memory errors, or too many dims), I'm starting to feel like I'm in a bit of trouble here :) What do people use to do large svd's? I'm not adverse to using another lib or wrapping something. Cheers Dave ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy array sharing between processes?
On 5/12/07, Andrew Straw [EMAIL PROTECTED] wrote: Ray Schumacher wrote: After Googling for examples on this, in the Cookbook http://www.scipy.org/Cookbook/Multithreading MPI and POSH (dead?), I don't think I know the answer... We have a data collection app running on dual core processors; I start one thread collecting/writing new data directly into a numpy circular buffer, another thread does correlation on the newest data and occasional FFTs, both now use 50% CPU, total. The threads never need to access the same buffer slices. I'd prefer to have two processes, forking the FFT process off and utilizing the second core. The processes would only need to share two variables (buffer insert position and a short_integer result from the FFT process, each process would only read or write), in addition to the numpy array itself. Should I pass the numpy address to the second process and just create an identical array there, as in http://projects.scipy.org/pipermail/numpy-discussion/2006-October/023647.html ? Use a file-like object to share the other variables? mmap? http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/413807 I also thought ctypes ctypes.string_at(address[, size]) might do both easily enough, although would mean a copy. We already use it for the collection thread. Does anyone have a lightweight solution to this relatively simple sort of problem? I'll pitch in a few donuts (and my eternal gratitude) for an example of shared memory use using numpy arrays that is cross platform, or at least works in linux, mac, and windows. I wonder if you could mmap a file and use it as common memory? Forking in python under linux leads to copies because anything that accesses an object changes its reference count. Pipes are easy and could be used for synchronization. Would python threading work for you? It might be the easiest way to have the fft going on while doing something else. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy array sharing between processes?
Andrew added: I'll pitch in a few donuts (and my eternal gratitude) for an example of shared memory use using numpy arrays that is cross platform, or at least works in linux, mac, and windows. I thought that getting the address from the buffer() of the array and creating a new one from it in the other process is most likely to work cross-platform, but I don't know enough of Python innards to speak confidently. I have used it with numarray before : N = numarray.zeros((1000,), numarray.Float) N.info() ... data: memory at 009d67b8 with size:8000 held by object 009d6798... I'm also considering trying more mix with ctypes, as in: http://www.scipy.org/Cookbook/Ctypes#head-db3c64e7ade3f0257084b9c03f03d286ff3b1b4f - Ray ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy array sharing between processes?
Charles R Harris wrote: I'll pitch in a few donuts (and my eternal gratitude) for an example of shared memory use using numpy arrays that is cross platform, or at least works in linux, mac, and windows. I wonder if you could mmap a file and use it as common memory? Yes, that's the basic idea. Now for the example that works on those platforms... Forking in python under linux leads to copies because anything that accesses an object changes its reference count. I'm not sure what you're trying to say here. If it's shared memory, it's not copied -- that's the whole point. I don't really care how I spawn the multiple processes, and indeed forking is one way. Pipes are easy and could be used for synchronization. True. But they're not going to be very fast. (I'd like to send streams of realtime images between different processes.) Would python threading work for you? That's what I use now and what I'd like to get away from because 1) the GIL sucks and 2) (bug-free) threading is hard. -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] very large matrices.
On 12/05/07, Dave P. Novakovic [EMAIL PROTECTED] wrote: core 2 duo with 4gb RAM. I've heard about iterative svd functions. I actually need a complete svd, with all eigenvalues (not LSI). I'm actually more interested in the individual eigenvectors. As an example, a single row could probably have about 3000 non zero elements. I think you need to think hard about whether your problem can be done in another way. First of all, the singular values (as returned from the svd) are not eigenvalues - eigenvalue decomposition is a much harder problem, numerically. Second, your full non-sparse matrix will be 8*75000*75000 bytes, or about 42 gibibytes. Put another way, the representation of your data alone is ten times the size of the RAM on the machine you're using. Third, your matrix has 225 000 000 nonzero entries; assuming a perfect sparse representation with no extra bytes (at least two bytes per entry is typical, usually more), that's 1.7 GiB. Recall that basically any matrix operation is at least O(N^3), so you can expect order 10^14 floating-point operations to be required. This is actually the *least* significant constraint; pushing stuff into and out of disk caches will be taking most of your time. Even if you can represent your matrix sparsely (using only a couple of gibibytes), you've said you want the full set of eigenvectors, which is not likely to be a sparse matrix - so your result is back up to 42 GiB. And you should expect an eigenvalue algorithm, if it even survives massive roundoff problems, to require something like that much working space; thus your problem probably has a working size of something like 84 GiB. SVD is a little easier, if that's what you want, but the full solution is twice as large, though if you discard entries corresponding to small values it might be quite reasonable. You'll still need some fairly specialized code, though. Which form are you looking for? Solving your problem in a reasonable amount of time, as described and on the hardware you specify, is going to require some very specialized algorithms; you could try looking for an out-of-core eigenvalue package, but I'd first look to see if there's any way you can simplify your problem - getting just one eigenvector, maybe. Anne ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
