Re: [web2py] Re: cron Too many open files regression?
Sounds like some kind of race condition between the cron scripts because it doesn't happen every time. Is there any chance the 3 cron scripts are dependent on each other in some way such as a file passed between or sharing a database. If there is any relationship between the cron scripts would it be possible to make a single script that just runs the 3 scripts sequentially and use that as your cron script. Do the scripts access the database and the database happens to be SQLite which locks for the duration of an access/ Without seeing some code and more details determining the cause is a guess. Ron
Re: [web2py] Re: cron Too many open files regression?
It was my understanding that Python's GC frees any resources not explicitly freed, and cron runs jobs in a try/except/finally block and rolls back any uncommitted transactions. I have reproduced the issue with the three running sequentially, and in every case, the logs indicate that my functions exit, meaning that the rollback (I'm committing) and GC actions should occur, or at the very least, the OS should close/release stuff when the crom process exits. Hmmm.. This is going to be interesting to track down. On Wed, Jul 6, 2011 at 5:56 AM, Massimo Di Pierro massimo.dipie...@gmail.com wrote: Yes that is the problem wit cron issue. If one of the processes locks a resource, cron does not know about it and keeps spawning processes as scheduled. The new processes find the resource locked and freeze of crash but use ram. This is not a bug because cron is not supposed to know what the tasks do. This is a general logic problem with cron. Using one single background process that loops and sleeps is much safer. Massimo On Jul 6, 1:34 am, ron_m ron.mco...@gmail.com wrote: Sounds like some kind of race condition between the cron scripts because it doesn't happen every time. Is there any chance the 3 cron scripts are dependent on each other in some way such as a file passed between or sharing a database. If there is any relationship between the cron scripts would it be possible to make a single script that just runs the 3 scripts sequentially and use that as your cron script. Do the scripts access the database and the database happens to be SQLite which locks for the duration of an access/ Without seeing some code and more details determining the cause is a guess. Ron -- John Duddy jdu...@gmail.com
Re: [web2py] Re: cron Too many open files regression?
Maybe one possibility is if your code has classes with a __del__ method in them, Massimo would have better perspective on if this is a possibility. If a class with this method is involved in a circular reference the garbage collector cannot clean it up because of uncertainty of execution of the __del__ method. This was discussed a lot on the group a few months back. I believe exec also has to be involved. Do a search for __del__ in the group to see what was said. There were also some tools mentioned in those threads that help with tracking down this sort of problem. I find in Python as soon as a file variable binding produced by open goes out of scope it is closed, you don't have to specifically call close on it. Normally objects are dropped as soon as the reference count reaches 0. The garbage collector was added to clean up the circular referenced objects that will never get a count down to 0 without some outside help. It searches the heap looking for objects that have no references except other objects also in the garbage and marks those as candidates to clean out. The fact that these file objects are piling up means something is holding the file reference which also cannot be cleaned up. Are you running on Linux? If so the /proc/pid_of_web2py/fd directory using ls -l will show you what files by name are open to the process which might provide some clues to which part of your code.
Re: [web2py] Re: cron Too many open files regression?
I am seeing lots of processes (20 or so) backing up, yet as far as I can tell, they are not doing anything. I'll add some tracing to see if they are actually processing, but slowly. On Tue, Jul 5, 2011 at 12:32 PM, Massimo Di Pierro massimo.dipie...@gmail.com wrote: Are you starting a lot of cron tasks that do not terminate before the next one starts? I suggest you use a background process and do not use cron. To my knowledge there is no bug but cron become unpredictable if jobs take too long. Massimo On Jul 5, 10:34 am, John Duddy jdu...@gmail.com wrote: I'm getting the following stack trace in my console after leaving my app running overnight. We use cron for several tasks. I saw the below thread, which indicated that the issue has been resolved: http://groups.google.com/group/web2py/browse_thread/thread/e788b0783e... Has there been a regression? Here's the stack trace. Thanks! Traceback (most recent call last): File /usr/local/lib/python2.7/threading.py, line 552, in __bootstrap_inner self.run() File /root/web2py/gluon/newcron.py, line 63, in run s.run() File /usr/local/lib/python2.7/sched.py, line 117, in run File /root/web2py/gluon/newcron.py, line 55, in launch crondance(self.path, 'hard', startup = False) File /root/web2py/gluon/newcron.py, line 234, in crondance cronmaster = token.acquire(startup=startup) File /root/web2py/gluon/newcron.py, line 101, in acquire self.master = open(self.path,'rb+') IOError: [Errno 24] Too many open files: '/root/web2py/applications/ admin/cron/cron.master' -- John Duddy jdu...@gmail.com
Re: [web2py] Re: cron Too many open files regression?
I have dug further, and it is definitely a cron issue. I have 3 cron tasks firing every minute, and they always completed in under 1 second. But every so often, the process lauched by web2py would seem to hang - it never executed my code (first line was logging, not executed for that process). These processes never exit. lsof shows the normal cacaphony of open files, lots of stuff under python2.7, a few shared libs pipes, cron.master, and a deleted file in / tmp. http://tmp.cd Any ideas on how to track this down? On Tue, Jul 5, 2011 at 12:47 PM, John Duddy jdu...@gmail.com wrote: I am seeing lots of processes (20 or so) backing up, yet as far as I can tell, they are not doing anything. I'll add some tracing to see if they are actually processing, but slowly. On Tue, Jul 5, 2011 at 12:32 PM, Massimo Di Pierro massimo.dipie...@gmail.com wrote: Are you starting a lot of cron tasks that do not terminate before the next one starts? I suggest you use a background process and do not use cron. To my knowledge there is no bug but cron become unpredictable if jobs take too long. Massimo On Jul 5, 10:34 am, John Duddy jdu...@gmail.com wrote: I'm getting the following stack trace in my console after leaving my app running overnight. We use cron for several tasks. I saw the below thread, which indicated that the issue has been resolved: http://groups.google.com/group/web2py/browse_thread/thread/e788b0783e. .. Has there been a regression? Here's the stack trace. Thanks! Traceback (most recent call last): File /usr/local/lib/python2.7/threading.py, line 552, in __bootstrap_inner self.run() File /root/web2py/gluon/newcron.py, line 63, in run s.run() File /usr/local/lib/python2.7/sched.py, line 117, in run File /root/web2py/gluon/newcron.py, line 55, in launch crondance(self.path, 'hard', startup = False) File /root/web2py/gluon/newcron.py, line 234, in crondance cronmaster = token.acquire(startup=startup) File /root/web2py/gluon/newcron.py, line 101, in acquire self.master = open(self.path,'rb+') IOError: [Errno 24] Too many open files: '/root/web2py/applications/ admin/cron/cron.master' -- John Duddy jdu...@gmail.com -- John Duddy jdu...@gmail.com
