Re: script uses up all memory

2014-03-14 Thread Kent Engström
Larry Martell larry.mart...@gmail.com writes: I figured out what is causing this. Each pass through the loop it does: self.tools = Tool.objects.filter(ip__isnull=False) And that is what is causing the memory consumption. If I move that outside the loop and just do that once the memory issue

Re: script uses up all memory

2014-03-06 Thread Larry Martell
On Wed, Mar 5, 2014 at 5:27 PM, Larry Martell larry.mart...@gmail.com wrote: I have a script that forks off other processes and attempts to manage them. Here is a stripped down version of the script: self.sleepTime = 300 self.procs = {} self.startTimes = {}

Re: script uses up all memory

2014-03-06 Thread Larry Martell
On Thu, Mar 6, 2014 at 4:56 PM, Larry Martell larry.mart...@gmail.com wrote: On Wed, Mar 5, 2014 at 5:27 PM, Larry Martell larry.mart...@gmail.com wrote: I have a script that forks off other processes and attempts to manage them. Here is a stripped down version of the script:

Re: script uses up all memory

2014-03-06 Thread Chris Angelico
On Fri, Mar 7, 2014 at 8:56 AM, Larry Martell larry.mart...@gmail.com wrote: I figured out what is causing this. Each pass through the loop it does: self.tools = Tool.objects.filter(ip__isnull=False) And that is what is causing the memory consumption. If I move that outside the loop and just

Re: script uses up all memory

2014-03-06 Thread Chris Angelico
On Fri, Mar 7, 2014 at 9:07 AM, Larry Martell larry.mart...@gmail.com wrote: Apparently the object returned by that call is immutable as if I look at id(self.tools) each pass through the loop, it is different. Is there some way I can recover that memory? Not sure what mutability has to do with

Re: script uses up all memory

2014-03-06 Thread Larry Martell
On Thu, Mar 6, 2014 at 5:11 PM, Chris Angelico ros...@gmail.com wrote: On Fri, Mar 7, 2014 at 8:56 AM, Larry Martell larry.mart...@gmail.com wrote: I figured out what is causing this. Each pass through the loop it does: self.tools = Tool.objects.filter(ip__isnull=False) And that is what is

Re: script uses up all memory

2014-03-06 Thread Marko Rauhamaa
Chris Angelico ros...@gmail.com: Not all problems need to be solved perfectly :) But at very least, I would put a comment against your collect() call explaining what happens: that self.tools is involved in a refloop. Most Python code shouldn't have to call gc.collect(), so it's worth

Re: script uses up all memory

2014-03-06 Thread Chris Angelico
On Fri, Mar 7, 2014 at 9:21 AM, Larry Martell larry.mart...@gmail.com wrote: First I added del(self.tools) before the Django call. That did not stop the memory consumption. Then I added a call to gc.collect() after the del and that did solve it. gc.collect() returns 0 each time, so I'm going

Re: script uses up all memory

2014-03-06 Thread Chris Angelico
On Fri, Mar 7, 2014 at 9:34 AM, Marko Rauhamaa ma...@pacujo.net wrote: Chris Angelico ros...@gmail.com: Not all problems need to be solved perfectly :) But at very least, I would put a comment against your collect() call explaining what happens: that self.tools is involved in a refloop. Most

Re: script uses up all memory

2014-03-06 Thread Marko Rauhamaa
Chris Angelico ros...@gmail.com: I think this thread is proof that they are to be avoided. The GC wasn't doing its job unless explicitly called on. The true solution is to break the refloop; the quick fix is to call gc.collect(). I stand by the recommendation to put an explanatory comment

Re: script uses up all memory

2014-03-06 Thread Chris Angelico
On Fri, Mar 7, 2014 at 10:12 AM, Marko Rauhamaa ma...@pacujo.net wrote: Chris Angelico ros...@gmail.com: I think this thread is proof that they are to be avoided. The GC wasn't doing its job unless explicitly called on. The true solution is to break the refloop; the quick fix is to call

Re: script uses up all memory

2014-03-06 Thread Marko Rauhamaa
Chris Angelico ros...@gmail.com: Can you give a useful example of a closure that does create a refloop? Just the other day, I mentioned the state pattern: class MyStateMachine: def __init__(self): sm = self class IDLE: def ding(self):

Re: script uses up all memory

2014-03-06 Thread Chris Angelico
On Fri, Mar 7, 2014 at 10:53 AM, Marko Rauhamaa ma...@pacujo.net wrote: Chris Angelico ros...@gmail.com: Can you give a useful example of a closure that does create a refloop? Just the other day, I mentioned the state pattern: class MyStateMachine: def __init__(self):

Re: script uses up all memory

2014-03-06 Thread Marko Rauhamaa
Chris Angelico ros...@gmail.com: On Fri, Mar 7, 2014 at 10:53 AM, Marko Rauhamaa ma...@pacujo.net wrote: class MyStateMachine: def __init__(self): sm = self class IDLE: def ding(self): sm.open_door()

script uses up all memory

2014-03-05 Thread Larry Martell
I have a script that forks off other processes and attempts to manage them. Here is a stripped down version of the script: self.sleepTime = 300 self.procs = {} self.startTimes = {} self.cmd = ['python', '/usr/local/motor/motor/app/some_other_script.py']

Re: script uses up all memory

2014-03-05 Thread Chris Angelico
On Thu, Mar 6, 2014 at 9:27 AM, Larry Martell larry.mart...@gmail.com wrote: I have a script that forks off other processes and attempts to manage them. Here is a stripped down version of the script: self.sleepTime = 300 That's not a stand-alone script. What environment is it running

Re: script uses up all memory

2014-03-05 Thread Larry Martell
On Wed, Mar 5, 2014 at 5:39 PM, Chris Angelico ros...@gmail.com wrote: On Thu, Mar 6, 2014 at 9:27 AM, Larry Martell larry.mart...@gmail.com wrote: I have a script that forks off other processes and attempts to manage them. Here is a stripped down version of the script: self.sleepTime

Re: script uses up all memory

2014-03-05 Thread Chris Angelico
On Thu, Mar 6, 2014 at 11:20 AM, Larry Martell larry.mart...@gmail.com wrote: On Wed, Mar 5, 2014 at 5:39 PM, Chris Angelico ros...@gmail.com wrote: On Thu, Mar 6, 2014 at 9:27 AM, Larry Martell larry.mart...@gmail.com wrote: I have a script that forks off other processes and attempts to

Re: script uses up all memory

2014-03-05 Thread Larry Martell
On Wed, Mar 5, 2014 at 7:33 PM, Chris Angelico ros...@gmail.com wrote: On Thu, Mar 6, 2014 at 11:20 AM, Larry Martell larry.mart...@gmail.com wrote: On Wed, Mar 5, 2014 at 5:39 PM, Chris Angelico ros...@gmail.com wrote: On Thu, Mar 6, 2014 at 9:27 AM, Larry Martell larry.mart...@gmail.com