Re: Shared memory python between two separate shell-launched processes
On Feb 10, 6:22 pm, Jean-Paul Calderone calderone.jeanp...@gmail.com wrote: On Feb 10, 12:21 pm, Charles Fox (Sheffield) charles@gmail.com wrote: On Feb 10, 3:43 pm, Jean-Paul Calderone calderone.jeanp...@gmail.com wrote: On Feb 10, 9:30 am, Charles Fox (Sheffield) charles@gmail.com wrote: Hi guys, I'm working on debugging a large python simulation which begins by preloading a huge cache of data. I want to step through code on many runs to do the debugging. Problem is that it takes 20 seconds to load the cache at each launch. (Cache is a dict in a 200Mb cPickle binary file). So speed up the compile-test cycle I'm thinking about running a completely separate process (not a fork, but a processed launched form a different terminal) Why _not_ fork? Load up your data, then go into a loop forking and loading/ running the rest of your code in the child. This should be really easy to implement compared to doing something with shared memory, and solves the problem you're trying to solve of long startup time just as well. It also protects you from possible bugs where the data gets corrupted by the code that operates on it, since there's only one copy shared amongst all your tests. Is there some other benefit that the shared memory approach gives you? Of course, adding unit tests that exercise your code on a smaller data set might also be a way to speed up development. Jean-Paul Thanks Jean-Paul, I'll have a think about this. I'm not sure if it will get me exactly what I want though, as I would need to keep unloading my development module and reloading it, all within the forked process, and I don't see how my debugger (and emacs pdb tracking) will keep up with that to let me step though the code. Not really. Don't load your code at all in the parent. Then there's nothing to unload in each child process, just some code to load for the very first time ever (as far as that process is concerned). Jean-Paul Jean, sorry I'm still not sure what you mean, could you give a couple of lines of pseudocode to illustrate it? And explain how my emacs pdbtrack would still be able to pick it up? thanks, charles -- http://mail.python.org/mailman/listinfo/python-list
Re: PDB how to define a global inspection function?
On Feb 8, 11:37 am, Peter Otten __pete...@web.de wrote: CharlesFox(Sheffield) wrote: Hi guys, I'm new to this group and have a question about debugging. I'm stepping through my code (using emacs pdbtrack and python-mode.el) and would like to isnpect objects as I go. So I've defined a little object print function, def p(obj): print obj print obj.__class__ d=dir(obj) for a in d: print a, =, getattr(obj, a) however it only works if the function is imported by whatever module I am currently debugging. I'd like it to be available globally so I can stop and inspect anything as I step through various modules (including external libraries). Is there a way to put it in the global scope for pdb to use? Also is there a way to automatically import it whenever pdb starts up (like a matlab startup file)? (I'm not using ipython as it's not happy with pdbtrack in emacs, so am launching from emacs M- x pdb command). For debugging purposes it's OK to put your function into the __builtin__ namespace: def p(): print 42 ... import __builtin__ __builtin__.p = p Try it out: with open(tmp.py, w) as f: ... f.write(p()\n) ... import tmp 42 Thanks very much for your help, Peter, that's exactly what I was after :-) Charles -- http://mail.python.org/mailman/listinfo/python-list
Shared memory python between two separate shell-launched processes
Hi guys, I'm working on debugging a large python simulation which begins by preloading a huge cache of data. I want to step through code on many runs to do the debugging. Problem is that it takes 20 seconds to load the cache at each launch. (Cache is a dict in a 200Mb cPickle binary file). So speed up the compile-test cycle I'm thinking about running a completely separate process (not a fork, but a processed launched form a different terminal) that can load the cache once then dunk it in an area of shareed memory.Each time I debug the main program, it can start up quickly and read from the shared memory instead of loading the cache itself. But when I look at posix_ipc and POSH it looks like you have to fork the second process from the first one, rather than access the shared memory though a key ID as in standard C unix shared memory. Am I missing something? Are there any other ways to do this? thanks, Charles -- http://mail.python.org/mailman/listinfo/python-list
Re: Shared memory python between two separate shell-launched processes
On Feb 10, 3:43 pm, Jean-Paul Calderone calderone.jeanp...@gmail.com wrote: On Feb 10, 9:30 am, Charles Fox (Sheffield) charles@gmail.com wrote: Hi guys, I'm working on debugging a large python simulation which begins by preloading a huge cache of data. I want to step through code on many runs to do the debugging. Problem is that it takes 20 seconds to load the cache at each launch. (Cache is a dict in a 200Mb cPickle binary file). So speed up the compile-test cycle I'm thinking about running a completely separate process (not a fork, but a processed launched form a different terminal) Why _not_ fork? Load up your data, then go into a loop forking and loading/ running the rest of your code in the child. This should be really easy to implement compared to doing something with shared memory, and solves the problem you're trying to solve of long startup time just as well. It also protects you from possible bugs where the data gets corrupted by the code that operates on it, since there's only one copy shared amongst all your tests. Is there some other benefit that the shared memory approach gives you? Of course, adding unit tests that exercise your code on a smaller data set might also be a way to speed up development. Jean-Paul Thanks Jean-Paul, I'll have a think about this. I'm not sure if it will get me exactly what I want though, as I would need to keep unloading my development module and reloading it, all within the forked process, and I don't see how my debugger (and emacs pdb tracking) will keep up with that to let me step though the code. (this debugging is more about integration issues than single functions, I have a bunch of unit tests for the little bits but something is unhappy when I put them all together...) (I also had a reply by email, suggesting I use /dev/shm to store the data instead of the hard disc; this speeds things up a little but not much as the data still has to be transferred in bulk into my process. Unless I'm missing something and my process can just access the data in that shm without having to load its own copy?) -- http://mail.python.org/mailman/listinfo/python-list
PDB how to define a global inspection function?
Hi guys, I'm new to this group and have a question about debugging. I'm stepping through my code (using emacs pdbtrack and python-mode.el) and would like to isnpect objects as I go. So I've defined a little object print function, def p(obj): print obj print obj.__class__ d=dir(obj) for a in d: print a, =, getattr(obj, a) however it only works if the function is imported by whatever module I am currently debugging. I'd like it to be available globally so I can stop and inspect anything as I step through various modules (including external libraries). Is there a way to put it in the global scope for pdb to use? Also is there a way to automatically import it whenever pdb starts up (like a matlab startup file)? (I'm not using ipython as it's not happy with pdbtrack in emacs, so am launching from emacs M- x pdb command). thanks, Charles Fox -- http://mail.python.org/mailman/listinfo/python-list
array.shape() gives TypeError: 'tuple' object is not callable
Hi gys -- I am looking at Numpy but getting this error when I try to get array sizes. I'm using Ubuntu Edgy with standard repositories and scipy. Any ideas? Am I doing something wrong or is it my install of scipy? $ python Python 2.4.4c1 (#2, Oct 11 2006, 21:51:02) [GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)] on linux2 Type help, copyright, credits or license for more information. from numpy import * a=array([[1,2],[3,4]]) a array([[1, 2], [3, 4]]) a.shape() Traceback (most recent call last): File stdin, line 1, in ? TypeError: 'tuple' object is not callable thanks charles -- http://mail.python.org/mailman/listinfo/python-list
Hide comments in emacs python mode?
Hi guys, I'm playing with Python in emacs, with python mode. I'd like to be able to press a key to toggle the code comments on and off -- to switch between beautiful clean Python code, and the full text that tells me what's going in in English. Is this currently possible? I know there is a hide/show mode in emacs, it would need to be set to hide (1) whole lines that start with #, (2) parts of lines after the '#' for comments after code. Has anyone already written this? -- http://mail.python.org/mailman/listinfo/python-list
Re: newbie: self.member syntax seems /really/ annoying
hmm, I guess this is the difference between numerical programming and the rest -- sure, if I was writing a database server or something it would be great to have thisObject.veryLongName to know what everything is -- however when you are implementing a model from a published paper, the variables tend to be single greek or roman letter names, possibly with subscripts and superscripts, and it helps if the name you see on the screen is the same as the name on the paper, as is the case in matlab. You want the equation on screen to look as similar to the one on the paper as possible, especially if its going to be read by another programmer who is familiar with the paper. Maybe for now I will just fix up my emacs to display the world 'self' in 10% gray... :-) -- http://mail.python.org/mailman/listinfo/python-list
newbie: self.member syntax seems /really/ annoying
I've just started playing around with Python, as a possible replacement for a mix of C++, Matlab and Lisp. The language looks lovely and clean with one huge exception: I do a lot of numerical modeling, so I deal with objects (like neurons) described mathematically in papers, by equations like a_dot = -k(a-u) In other languages, this translates nicely into code, but as far as I can tell, Python needs the ugly: self.a_dot = -self.k(self.a-self.u) For large equations this is going to make my code seriously unreadable to the point of needing to switch back to Matlab -- and it seems to go against everything else about python's simplicity and elegance. Am I missing something? Is there something like a 'with' command that lets me set the scope, like with self: .a_dot = -.k(.a-.u) It's premature to make language suggestions as I am new to the language, but I would have though that making a 'with self' explicit in all methods would have been neat, so I could just write .a_dot = -.k(.a-.u) which would still avoid confusion with local function variables, since '.a' is different from 'a'. Please help if I am missing something -- this looks like a great language but I am going to mad trying to read numerical code full of 'self.'s breaking up the equations. -- http://mail.python.org/mailman/listinfo/python-list
Re: newbie: self.member syntax seems /really/ annoying
Thanks guys -- yeah these two stategies (short s.varname; and explicit rescoping, a=self.a etc) are more or less what I was using. That's still kind of annoying though. The s.varname approach still makes numerical code much harder to read. I had a nasty bug with the boilerplate approach when forgetting to reassign some of the variables back to members (self.a=a). And that's a lot of boilerplate you need -- I thought the python way was to minimize redundant code? (Ditching header files and curley brackets was a main reason for me coming here). I see the argument for making self explicit -- what would be wrong with just .a instead of self.a though? That's still explicit but much easier to read. (I think I've seen that somewhere else, is it C#?) -- http://mail.python.org/mailman/listinfo/python-list
