On Thu, Mar 6, 2014 at 4:56 PM, Larry Martell <larry.mart...@gmail.com> wrote: > On Wed, Mar 5, 2014 at 5:27 PM, Larry Martell <larry.mart...@gmail.com> wrote: >> I have a script that forks off other processes and attempts to manage >> them. Here is a stripped down version of the script: >> >> self.sleepTime = 300 >> self.procs = {} >> self.startTimes = {} >> self.cmd = ['python', >> '/usr/local/motor/motor/app/some_other_script.py'] >> >> while True: >> try: >> self.tools = Tool.objects.filter(ip__isnull=False) >> except Exception, e: >> print 'error from django call: ' + str(e) >> sys.exit(1) >> >> for tool in self.tools: >> name = tool.name >> if name in self.procs: >> if self.procs[name].poll() is None: >> if >> (datetime.datetime.now()-self.startTimes[name]) > >> datetime.timedelta(hours=12): >> # it's been running too long - kill it >> print 'killing script for ' + name + " >> it's been running too long" >> self.procs[name].kill() >> else: >> continue >> >> if self.procs[name].returncode: >> print 'scrikpt failed for ' + name + ', error >> = ' + str(self.procs[name].returncode) >> >> print 'starting script.py for ' + name + ' at ' + >> str(datetime.datetime.now()) >> try: >> self.procs[name] = subprocess.Popen(self.cmd) >> self.startTimes[name] = datetime.datetime.now() >> except Exception, e: >> print 'error from Popen: ' + str(e) >> sys.exit(1) >> else: >> print 'starting script.py for ' + name + ' at ' + >> str(datetime.datetime.now()) >> try: >> self.procs[name] = subprocess.Popen(self.cmd) >> self.startTimes[name] = datetime.datetime.now() >> except Exception, e: >> print 'error from Popen: ' + str(e) >> sys.exit(1) >> >> time.sleep(self.sleepTime) >> >> >> The script does what it's intended to do, however after about 2 hours >> it has used up all the memory available and the machine hangs. Can >> anyone see something that I am doing here that would be using memory >> like this? Perhaps some immutable object needs to be repeatedly >> recreated? > > I figured out what is causing this. Each pass through the loop it does: > > self.tools = Tool.objects.filter(ip__isnull=False) > > And that is what is causing the memory consumption. If I move that > outside the loop and just do that once the memory issue goes away. Now > I need to figure out why this is happening and how to prevent it as > they do want to query the db each pass through the loop in case it has > been updated.
Apparently the object returned by that call is immutable as if I look at id(self.tools) each pass through the loop, it is different. Is there some way I can recover that memory? -- https://mail.python.org/mailman/listinfo/python-list