On Dec 28, 2010, at 1:18 PM, Lorin Rivers wrote:
> I am working on moving some of the computation-intensive processes to a
> cron-launched scheme. I have some that run every minute, every 2 minutes,
> every 5 minutes and every 15 minutes.
>
> In production, it's possible that the volume of incoming data would result in
> the task not finishing before its next cron slot fires again. In development
> , however, I can flood the system quite easily by importing data in larger
> chunks.
>
> At the moment, my laptop (fairly decent machine) is essentially unusable
> because there are so many python processes consuming RAM.
Do you see this log message?
logger.warning('WEB2PY CRON: Stale cron.master detected')
I don't think that there's a good global solution to this kind of problem; cron
is really oriented toward jobs that can finish in a short time period.
One approach would be to have a daemon process (or a pool of them, maybe one
per core) to do the work, and in your cron jobs simply queue work for them.
Alternatively (or perhaps in addition) you might want to introduce a little
back pressure to your clients, so as not to accept more work than your server
is capable of handling in a reasonable time.
With the latest version of web2py/Rocket, you can lower the limit on request
threads (I count 22 below), which would be a brute-force way of getting back
pressure.
>
> Here's a slightly cleaned-up sample of ps:
>
> 6716 ttys000 0:01.70 ../Python web2py.py -a 1234 -i 127.0.0.1 -p 8000
> 6722 ttys000 0:36.46 ../Python ../web2py.py -J -M -S Debug/DAQ/setup_DAQ
> -a <recycle>
> 6724 ttys000 0:00.00 (Python)
> 7239 ttys000 0:00.00 (Python)
> 7543 ttys000 0:00.00 (Python)
> 7641 ttys000 0:00.00 (Python)
> 7720 ttys000 0:00.00 (Python)
> 7819 ttys000 0:00.00 (Python)
> 8781 ttys000 0:00.00 (Python)
> 8783 ttys000 0:00.00 (Python)
> 8784 ttys000 0:00.00 (Python)
> 8787 ttys000 0:00.00 (Python)
> 8789 ttys000 0:00.00 (Python)
> 8790 ttys000 0:00.00 (Python)
> 8791 ttys000 0:00.00 (Python)
> 8792 ttys000 0:00.00 (Python)
> 8867 ttys000 0:00.00 (Python)
> 8868 ttys000 0:00.00 (Python)
> 8872 ttys000 0:00.00 (Python)
> 8946 ttys000 0:00.00 (Python)
> 8947 ttys000 0:00.00 (Python)
> 8949 ttys000 0:00.00 (Python)
> 8950 ttys000 0:00.00 (Python)
> 8951 ttys000 0:00.00 (Python)
> 8948 ttys000 0:00.46 ../Python ../web2py.py -J -M -S
> Debug/rounding/array_1minsRollup -a <recycle>
> 7320 ttys000 1:04.75 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 7424 ttys000 0:56.54 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 7544 ttys000 0:50.19 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 7643 ttys000 0:42.32 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 7722 ttys000 0:36.26 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 7820 ttys000 0:30.36 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 7899 ttys000 0:25.12 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 8013 ttys000 0:20.99 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 8090 ttys000 0:17.28 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 8189 ttys000 0:14.05 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 8271 ttys000 0:11.55 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 8364 ttys000 0:07.50 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 8455 ttys000 0:07.51 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 8557 ttys000 0:05.08 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 8635 ttys000 0:00.73 ../Python ../web2py.py -J -M -S
> Debug/rounding/data_tableRollup -a <recycle>
> 8458 ttys000 0:18.24 ../Python ../web2py.py -J -M -S
> Debug/rounding/panel_1minsRollup -a <recycle>
> 8638 ttys000 0:16.68 ../Python ../web2py.py -J -M -S
> Debug/rounding/panel_1minsRollup -a <recycle>
> 8786 ttys000 0:00.57 ../Python ../web2py.py -J -M -S
> Debug/rounding/panel_1minsRollup -a <recycle>
> 7061 ttys000 0:43.78 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 7140 ttys000 0:12.82 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 7240 ttys000 0:12.35 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 7319 ttys000 0:12.37 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 7423 ttys000 0:12.20 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 7541 ttys000 0:12.10 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 7640 ttys000 0:12.26 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 7721 ttys000 0:12.31 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 7818 ttys000 0:12.59 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 7898 ttys000 0:12.53 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 8012 ttys000 0:12.46 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 8092 ttys000 0:12.46 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 8188 ttys000 0:12.26 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 8270 ttys000 0:11.96 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 8363 ttys000 0:12.05 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 8454 ttys000 0:12.12 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 8556 ttys000 0:12.02 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 8632 ttys000 0:11.49 ../Python ../web2py.py -J -M -S
> Debug/rounding/stringdataRollup -a <recycle>
> 8785 ttys000 0:00.55 ../Python ../web2py.py -J -M -S
> Debug/rounding/string_1minsRollup -a <recycle>
> 8788 ttys000 0:00.57 ../Python ../web2py.py -J -M -S
> Debug/rounding/string_5minsRollup -a <recycle>
>
