Tracking down memory leaks?
I have an application with one function called compute, which given a filename, goes through that file and performs various statistical analyses. It uses arrays extensively and loops alot. it prints the results of it's statistical significance tests to standard out. Since the compute function returns and I think no variables of global scope are being used, I would think that when it does, all memory returns back to the operating system. Instead, what I see is that every iteration uses several megs more. For example, python uses 52 megs when starting out, it goes through several iterations and I'm suddenly using more than 500 megs of ram. Does anyone have any pointers on how to figure out what I'm doing wrong? Thanks, mohan -- http://mail.python.org/mailman/listinfo/python-list
Re: Tracking down memory leaks?
How big is the file you are reading in? If it is (say) 400 MB, then it is hardly surprising that you will be using 500MB of RAM. If the file is 25K, that's another story. Actually, I am downloading the matrix data from a file on a server on the net using urllib2, and then I am running several basic stats on it using some functions that i get from matplotlib. Most are statistical functions I run on standard vectors, such as standard deviation, mean, median, etc. I do then loop through various matrix items, and then based on a set of criteria, I attempt to perform a sort of linear regression model using a few loops on the vectors. How are you storing your data while you are processing it? I'd be looking for hidden duplicates. I am storing basically everything as a set of vectors. For example, I would have one vector for my X-axis, time. The other variables are the number of units sold and the total aggregate revenue from selling all units. I am wondering if it's actually urllib2 that is messing me up. It could be matplotlib as well, although I doubt it since I do not use matplotlib unless the statistical significance test I produce indicates a high level of strength (very rare), indicating to me that the company has a winning product. -- http://mail.python.org/mailman/listinfo/python-list
Array construction from object members
Hi everyone, I am doing several operations on lists and I am wondering if python has anything built in to get every member of several objects that are in an array, for example, if i have a class like the following: class myClass: a = 0.0 And lets say I populate the a element in an array of objects of myClass. If I want to retrieve all items in this and perhaps give it to a mean function, I would need to make a loop now: mySimpleArray = [] for i in range(0,len(myArray)): mySimpleArray.append(myArray[i].a) There must be some more efficient way to do this, can someone point me to the right direction so that I can review some documentation and get things a little more efficient? thanks! -- http://mail.python.org/mailman/listinfo/python-list
Best library for basic stats
Is there any decent library for basic stats? I am just looking for the basics, like quartiles, median, standard deviation, mean, min, max, regression, etc. I was going to use SciPy, but I use Python 2.4, and it seems to be supported primarily for 2.3... What other statistical packages have any level of public support at this point? thanks -- http://mail.python.org/mailman/listinfo/python-list
Filtering out non-readable characters
I have a file with binary and ascii characters in it. I massage the data and convert it to a more readable format, however it still comes up with some binary characters mixed in. I'd like to write something to just replace all non-printable characters with '' (I want to delete non-printable characters). I am having trouble figuring out an easy python way to do this... is the easiest way to just write some regular expression that does something like replace [^\p] with ''? Or is it better to go through every character and do ord(character), check the ascii values? What's the easiest way to do something like this? thanks -- http://mail.python.org/mailman/listinfo/python-list
stdout and stderr showing output only after script is done
I just installed python 2.4 for windows. I was using the cygwin version previously, but wanted to use the official python 2.4 release to install a module that reads off some registry entries. After installing, I did what I normally do, ssh to my local windows machine, and run the python script I wrote. The script takes about 30 seconds to complete and gives me incremental output. I ran it, and it seems to be buffering standard input and output. It ran my whole script and showed me all the output of stderr and stdout only after the whole script was done. For example, my script can sometimes take like 10 minutes to finish, so I print out 1% complete, 2% complete and so on. Now it shows me nothing, and at the end, it just tells me 1% complete, 2% complete... 100% complete all in one shot. Can I change this behavior somehow? thanks, mohan -- http://mail.python.org/mailman/listinfo/python-list
Re: stdout and stderr showing output only after script is done
sorry, i'm an idoit who just found that i should use the -u option, my bad -- http://mail.python.org/mailman/listinfo/python-list
Re: stdout and stderr showing output only after script is done
sorry i meant i installed python version 2.4.1, thanks MKoool wrote: I just installed python 2.4 for windows. I was using the cygwin version previously, but wanted to use the official python 2.4 release to install a module that reads off some registry entries. After installing, I did what I normally do, ssh to my local windows machine, and run the python script I wrote. The script takes about 30 seconds to complete and gives me incremental output. I ran it, and it seems to be buffering standard input and output. It ran my whole script and showed me all the output of stderr and stdout only after the whole script was done. For example, my script can sometimes take like 10 minutes to finish, so I print out 1% complete, 2% complete and so on. Now it shows me nothing, and at the end, it just tells me 1% complete, 2% complete... 100% complete all in one shot. Can I change this behavior somehow? thanks, mohan -- http://mail.python.org/mailman/listinfo/python-list
