subtle error slows code by 10x (builtin sum()) - replace builtin sum without using import?
First a trap for new players, then a question to developers Code accelerated by numpy can be slowed down by a large factor is you neglect to import numpy.sum . from timeit import Timer frag = 'x=sum(linspace(0,1,1000))' Timer(frag ,setup='from numpy import linspace').timeit(1000) # 0.6 sec Timer(frag, setup='from numpy import sum, linspace').timeit(1000) # difference is I import numpy.sum # 0.04 sec 15x faster! This is obvious of course - but it is very easy to forget to import numpy.sum and pay the price in execution. Question: Can I replace the builtin sum function globally for test purposes so that my large set of codes uses the replacement? The replacement would simply issue warnings.warn() if it detected an ndarray argument, then call the original sum I could then find the offending code and use the appropriate import to get numpy.sum -- http://mail.python.org/mailman/listinfo/python-list
Re: breaking out to the debugger (other than x=1/0 !)
That's perfect - and removing the "breakpoint" is not an issue for me as it is normally conditional on a debug level, which I can change from pydb if debuglvl>3: import pydb pydb.set_trace() 'in XXX: c to continue' The text line is a useful prompt (The example here is for pydb which works as well (and is more like gdb). On Oct 23, 12:07 pm, "Diez B. Roggisch" wrote: > bdb112 wrote: > > After a while programming in python, I still don't know how to break > > out to the debugger other than inserting an instruction to cause an > > exception. > > x=1/0 > > > In IDL one woudl write > > > stop,'reason for stopping...' > > at which point you can inspect locals (as in pdb) and continue (but > > you can't with pdb if python stopped because of an exception) > > > I am using ipython -pylab -pdb (python 2.5,2.6) > > Yes, I realise that I could start with the debugger, and set break > > points, but that can be slower and sometimes cause problems, and I > > like ipython's magic features. > > > Also, I don't know how to stop cleanly handing control back to ipython > > inside a program - e.g. after printing help text. > > I use > > import pdb; pdb.set_trace() > > Of course that can't be deleted as breakpoint - but it suits me well. > > Diez -- http://mail.python.org/mailman/listinfo/python-list
breaking out to the debugger (other than x=1/0 !)
After a while programming in python, I still don't know how to break out to the debugger other than inserting an instruction to cause an exception. x=1/0 In IDL one woudl write stop,'reason for stopping...' at which point you can inspect locals (as in pdb) and continue (but you can't with pdb if python stopped because of an exception) I am using ipython -pylab -pdb (python 2.5,2.6) Yes, I realise that I could start with the debugger, and set break points, but that can be slower and sometimes cause problems, and I like ipython's magic features. Also, I don't know how to stop cleanly handing control back to ipython inside a program - e.g. after printing help text. -- http://mail.python.org/mailman/listinfo/python-list
breaking out to the debugger (other than x=1/0 !)
After a while programming in python, I still don't know how to break out to the debugger other than inserting an instruction to cause an exception. x=1/0 In IDL one woudl write stop,'reason for stopping...' at which point you can inspect locals (as in pdb) and continue (but you can't with pdb if python stopped because of an exception) I am using ipython -pylab -pdb (python 2.5,2.6) Yes, I realise that I could start with the debugger, and set break points, but that can be slower and sometimes cause problems, and I like ipython's magic features. Also, I don't know how to stop cleanly handing control back to ipython inside a program - e.g. after printing help text. -- http://mail.python.org/mailman/listinfo/python-list
pylab/matplotlib large plot memory management - bug? or tuning parameter needed?
Summary: It is not straightforward to avoid memory leaks/consumption in pylab. If we define x = arange(1e6) # adjust size to make the increment visible, yet fast enough to plot # then repetition of plot(x,hold=0) # consumes increasing memory according to ubuntu system monitor Details: #versions: ubuntu 9.04 standard (py 2.6.2, ipython 0.91, matplotlib 0.98.5.2) , (results here) #or win32 matplotlib '0.98.3', python 2.5 (similar, although mem usage limits and reuses after 3-4 plot()s First, turn off output caching in ipython because it consumes memory (as it should) ipython -pylab -cs 0 plot(x,hold=0) plot(x,hold=0) plot(x,hold=0) # closing the window doesn't help much, neither does close() or any of the below individually # antidote plot(hold=0); gcf().clf(); close() # first often doesn't help! plot(hold=0); gcf().clf(); close() # need all three! plot(hold=0); gcf().clf(); close() As stated above, the windows version apparently starts more aggressive garbage collection after 3-4 plots. The ubuntu version reaches my system memory limit (1GB) without reclaiming memory - i.e. memory usage just keeps growing until swap space is used when array is 2e6 elements, consuming 100mB per plot. For 1e6 elements, memory usage grows for about 10 50MB steps, and then some garbage collection seems to happen, alothough more can be freeed with the triple line above. 1/ I am running under VMWare so maybe VMWare isn;t reporting the correct physical memory size to ubuntu/python - how to check this? 2/ possible bug - why doesn't closing the plot window release all memory it uses? Especially when this approaches machine memory size. 3/ Are there python/matplotlib memory management tuning parameters I can tweak? -- http://mail.python.org/mailman/listinfo/python-list
Re: clean way prepend an element to a numpy array
On Jul 21, 2:13 pm, Ben Finney wrote: > bdb112 writes: > > If I want to add an element at the beginning of an array, it seems > > like I must make a list, insert in place, then make an array again. > > The NumPy ‘ndarray’ type (which is what you get by default from the > ‘array’ factory function) is a far more complex type than (and is not > derived from) the Python list. > > For manipulating them, you'll want to study the NumPy documentation > http://docs.scipy.org/doc/numpy/reference/arrays.ndarray.html>. > Yes, I had had a look through that - nothing there that allows writing a succint clear statement (except for my .resize example in the original post), although if efficiency was a concern, then maybe resize and shift right, but that would really obscure the code. -- http://mail.python.org/mailman/listinfo/python-list
clean way prepend an element to a numpy array
If I want to add an element at the beginning of an array, it seems like I must make a list, insert in place, then make an array again. Of course, lists can be efficient too, and the insert() funtion works nicely in that case, but sometimes arrays are the best choice e.g. x=array([1,2,3]) # to put the element 0 at the head of the array listx=list(x) listx.insert(0,0) x=array(listx) # this extra effort distracts from the clarity of the code, and must slow things a little. # While I appreciate that array operations that cause memory re- allocation and copying are to be # avoided if at all possible, sometimes this is the best compromise between clarity and efficiency # A nicer piece of code would be x.insert(0,0) #or x.prepend(0) # It is a little easier to grow an array at the end (if it is not referenced) x.resize(4) I saw this interest syntax on this site x[:0]=0 I guess that is cute, but could be confusing(and doesn't work) -- http://mail.python.org/mailman/listinfo/python-list
Re: and [True,True] --> [True, True]?????
THanks Gabriel, Now I know about the zip function. Your explanation of Boolean ops on lists was clear. It leads to some intriguing results: bool([False]) --> True I wonder if python 3 changes any of this? > A and B means: check the boolean value of A; if it's false, return A. > Else, return B. > A non-empty list has a boolean value of true, so the second list is > returned. > > If you want an element-wise operation: > A = [False,True] > B = [True,True] > result = [a and b for a,b in zip(A,B)] > -- > Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
[False,True] and [True,True] --> [True, True]?????
Is there any obvious reason why [False,True] and [True,True] gives [True, True] Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] -- http://mail.python.org/mailman/listinfo/python-list
Re: print - bug or feature - concatenated format strings in a print statement
Thanks for all the replies:
I think I see now - % is a binary operator whose precedence rules are
shared with the modulo operator regardless of the nature of its
arguments, for language consistency.
I understand the arguments behind the format method, but hope that the
slightly idiosyncratic print( ..% ..) remains, as the vaguely
pictorial "printf" format string is clearer for a long line with
several arguments.
I will use the "implicit string concatenation" to solve my problem but
it is a little odd that an "invisible" operator is stronger than a
visible one. (+).
On Mar 16, 5:00 pm, bdb112 wrote:
> # is the difference between
> print(" %d, %d, buckle my shoe" % (1,2))
> # and
> print(" %d, " + " %d, buckle my shoe" % (1,2))
> # a bug or a feature?
>
> First output
> ... print(" %d " + " %d, buckle my shoe" % (1,2))
>
> Second output
> TypeError: not all arguments converted during string formatting
>
> Version Info:
> Python 2.5.1 (r251:54863, Oct 30 2007, 13:54:11)
> [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2
>
> also
>
> Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit
> (Intel)] on win32
--
http://mail.python.org/mailman/listinfo/python-list
Re: print - bug or feature - concatenated format strings in a print statement
#whoops, the first output is actually
1, 2, buckle my shoe
# in case it wasn't obvious
On Mar 16, 5:00 pm, bdb112 wrote:
> # is the difference between
> print(" %d, %d, buckle my shoe" % (1,2))
> # and
> print(" %d, " + " %d, buckle my shoe" % (1,2))
> # a bug or a feature?
>
> First output
> ... print(" %d " + " %d, buckle my shoe" % (1,2))
>
> Second output
> TypeError: not all arguments converted during string formatting
>
> Version Info:
> Python 2.5.1 (r251:54863, Oct 30 2007, 13:54:11)
> [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2
>
> also
>
> Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit
> (Intel)] on win32
--
http://mail.python.org/mailman/listinfo/python-list
print - bug or feature - concatenated format strings in a print statement
# is the difference between
print(" %d, %d, buckle my shoe" % (1,2))
# and
print(" %d, " + " %d, buckle my shoe" % (1,2))
# a bug or a feature?
First output
... print(" %d " + " %d, buckle my shoe" % (1,2))
Second output
TypeError: not all arguments converted during string formatting
Version Info:
Python 2.5.1 (r251:54863, Oct 30 2007, 13:54:11)
[GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2
also
Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit
(Intel)] on win32
--
http://mail.python.org/mailman/listinfo/python-list
Re: ipython / vs \ in readline on MS Windows (and ipython help grepper)
More info: import readline ? readline c:\python25\lib\site-packages\ipython\rlineimpl.py $Id: Magic.py 1096 2006-01-28 20:08:02Z vivainio $ sys.platform 'win32' sys.getfilesystemencoding() 'mbcs' sys.winver '2.5' $more /usr/local/bin/ipython #!/bin/bash C:/Python25/python.exe "C:\Python25\scripts\ipython" -pylab $* (Same problem if I use the windows Start menu Ipython IPython and pysh) On Mar 11, 7:34 am, bdb112 wrote: > Q1/ I run a standard python ditribution with ipython and readline > under cygwin. The tab filename completion works fine in the OS (bash > shell) as expected, and tab filename completion at the ipython command > line works, but with MS style path separators (backslash: run examples > \test.py) which the run command itself interprets unix style > ERROR: File `examplestest.py` not found. > > Also Q2/ can I "less" or "grep" the output from help(my_fun) > > Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit > (Intel)] > IPython 0.8.4 -- An enhanced Interactive Python. -- http://mail.python.org/mailman/listinfo/python-list
ipython / vs \ in readline on MS Windows (and ipython help grepper)
Q1/ I run a standard python ditribution with ipython and readline under cygwin. The tab filename completion works fine in the OS (bash shell) as expected, and tab filename completion at the ipython command line works, but with MS style path separators (backslash: run examples \test.py) which the run command itself interprets unix style ERROR: File `examplestest.py` not found. Also Q2/ can I "less" or "grep" the output from help(my_fun) Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] IPython 0.8.4 -- An enhanced Interactive Python. -- http://mail.python.org/mailman/listinfo/python-list
Re: docstring reference to another docstring
"""" Thanks for the quick reply - I expanded it to a working program,
it seems to do the job and works in my actual code (always good). As
you said, it assumes the called function's class is already defined.
Is there a way around this? (The module was originally ordered
"top-down").
""""
class Cluster():
"""Cluster docs...
"""
def plot():
"""Cluster.plot docs..
"""
class ClusterSet():
"""ClusterSet docs...
"""
__doc__ += "extra" + Cluster.__doc__
def plot():
"""ClusterSet.plot docs..
"""
__doc__ += "this is not what we want, it \
only happens at runtime, and refers to a local
variable"
print("ClusterSet.plot body")
plot.__doc__ += "extra" + Cluster.plot.__doc__
cs=ClusterSet()
print(cs.__doc__)
print(cs.plot.__doc__)
On Mar 10, 7:30 am, Chris Rebert wrote:
> On Mon, Mar 9, 2009 at 3:23 PM, bdb112 wrote:
> > A function of the class ClusterSet uses a similar function of the
> > class Cluster to do most of its work. Its docstring could have so
> > much in common with that in Cluster that it could be just a line or
> > two in addition to that of Cluster.
>
> > Is there a way for the ClusterSet docstring to tack the Cluster
> > docstring onto itself, rather than simply saying "See docstring for
> > Cluster"?
>
> #assume Cluster already defined by this point
> class ClusterSet(object):
> __doc__ = Cluster.__doc__
> #rest of class body
>
> Cheers,
> Chris
>
> --
> I have a blog:http://blog.rebertia.com
--
http://mail.python.org/mailman/listinfo/python-list
docstring reference to another docstring
A function of the class ClusterSet uses a similar function of the class Cluster to do most of its work. Its docstring could have so much in common with that in Cluster that it could be just a line or two in addition to that of Cluster. Is there a way for the ClusterSet docstring to tack the Cluster docstring onto itself, rather than simply saying "See docstring for Cluster"? -- http://mail.python.org/mailman/listinfo/python-list
