The code running this is in gluon/newcron.py >From your stack trace you are using hard cron which is a class at line 44. It runs launch once a minute This runs crondance which puts a wrapper on the tasks that should be run adding web2py run line parameters if * or ** is at the beginning of the command. The crondance function then uses class cronlauncher which in turn uses subprocess.Popen to launch the task.
If you have web2py like processes building up do these processes have other processes with the web2py process pid listed as the parent process. You should be able to do a ps -ef pick the pid of one of the web2py processes and run another ps -ef with grep pid to filter. The process started with the Popen will have to close the pipe usually by exiting to release the thread from cronlauncher. Do these cron tasks run okay if you just launch them from the shell. That is a good test to determine that there are no problems outside the cron environment. Do you use a SQLite database? This database performs a global lock to protect from multiple access. Do the cron tasks take more than their scheduling interval to complete. The launcher will run another copy without regard for an existing copy still running. Do any of the tasks have locking code such as a file lock that other tasks depend on gaining access to. Two file locks shared by two processes with crossed access, one process locks a then b, other locks b then a will deadlock if the timing is right. Do any tasks read stdin, the subprocess.Popen call opens the stdin of the process to the launcher but will never write to it so a read on stdin would hang preventing exit. I hope this is some help, looks like a difficult one to find but likely something fairly simple to fix once you find it. Could you post your crontab file, it would be a start of something to look at.
