What about trying to renice the task before the other routines get the chance to get a lot of CPU time?
It does sound strange that it would crash on a predictable basis. If Drupal is not being used, I would consider trying to remove it and see what happens. Dave On Mon, 2013-03-11 at 01:18 +0000, Toth, Csaba wrote: > Hey Linux Gurus, > > I wonder about your advices how I can solve this. > Our lab's web server at Vandy crashes every Sunday at 7:41am exactly. > I cannot track down the culprit. It's an Ubuntu Server Edition 12.04.1 > LTS Precise. > Apache has php5 and even Drupal. We don't use Drupal and very little > php. > > What I suspect that it's some scheduled task doing the harm. > The Anachron weekly starts at 7:30am. After that it start the > cron.weekly jobs with 10 minutes separation. Cron.weekly: > "0anacron > apt-xapian-index > debsums > man-db > sysklogd" > > 7:41 crash means that it can be the second thing it starts. I don't > know if 0anacron counts or not. Debsums: this is a package with which > you can identify damaged package. (The server had HDD hardware failure > during the summer. That was degrading for a while, and we had logical > file system issues too. I was down at the Medical Center DataCenter > and I had to issue more fscks to recover the filesystem fully. I'm > even happy that I could save the RAID volume, since it fell apart for > sometime too.) > I went ahead and started all scripts in cron.weekly manually from a > root command prompt, but nothing caused a crash. More info about the > crash: I cannot see anything in the Apache access logs for 7:41am, the > last log is for 7:37am. > > Error log: > > "[Sun Mar 10 01:29:09 2013] [error] [client 108.222.66.23] File does > not exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 01:50:02 2013] [error] [client 66.249.76.209] File does > not exist: /var/www/www.hiplab.org/people/malin/BMIF380 > [Sun Mar 10 03:36:04 2013] [error] [client 41.196.103.228] File does > not exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 03:36:08 2013] [error] [client 41.196.103.228] File does > not exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 03:36:40 2013] [error] [client 41.196.103.228] File does > not exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 03:52:46 2013] [error] [client 199.30.20.73] File does not > exist: /var/www/www.hiplab.org/people/weiyixia/icon.jpg > [Sun Mar 10 04:46:58 2013] [error] [client 78.162.114.93] File does > not exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 06:52:30 2013] [error] [client 69.163.180.32] File does > not exist: /var/www/www.hiplab.org/wp-admin > [Sun Mar 10 07:05:50 2013] [error] [client 69.163.187.202] File does > not exist: /var/www/www.hiplab.org/wordpress > [Sun Mar 10 07:26:48 2013] [error] [client 69.163.148.155] File does > not exist: /var/www/www.hiplab.org/wordpress > [Sun Mar 10 07:41:47 2013] [notice] Graceful restart requested, doing > restart > [Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: > FastCGI: read() from pipe failed (0) > [Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: > FastCGI: the PM is shutting down, Apache seems to have disappeared - > bye > *** glibc detected *** /usr/sbin/apache2: corrupted double-linked > list: 0x00007fda105bae40 *** > ======= Backtrace: ========= > /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fda0e8b1b96] > /lib/x86_64-linux-gnu/libc.so.6(+0x7eec8)[0x7fda0e8b1ec8] > /lib/x86_64-linux-gnu/libc.so.6(+0x80436)[0x7fda0e8b3436] > /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x75)[0x7fda0e8b5fb5] > /lib/x86_64-linux-gnu/libc.so.6(+0xbb50b)[0x7fda0e8ee50b] > /usr/lib/libapr-1.so.0(apr_dir_open+0x29)[0x7fda0ee24e76] > /usr/sbin/apache2(ap_process_resource_config+0xed)[0x7fda0f71030d] > /usr/sbin/apache2(+0x35aaa)[0x7fda0f702aaa] > /usr/sbin/apache2(+0x40741)[0x7fda0f70d741] > /usr/sbin/apache2(+0x42304)[0x7fda0f70f304] > /usr/sbin/apache2(ap_build_config+0x9c)[0x7fda0f70f8cc] > /usr/sbin/apache2(+0x42d01)[0x7fda0f70fd01] > /usr/sbin/apache2(ap_process_resource_config+0x209)[0x7fda0f710429] > /usr/sbin/apache2(ap_read_config+0x2df)[0x7fda0f71104f] > /usr/sbin/apache2(main+0xbb6)[0x7fda0f6f93d6] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main > +0xed)[0x7fda0e85476d] > /usr/sbin/apache2(+0x2c559)[0x7fda0f6f9559] > ======= Memory map: ========" > > In the beginning you can see nice tries from bots trying to grab > vulnerabilities: mofo bots tirelessly try to seek for admin consoles. > Then at07:41:47 restart requested, and apache segfaults. Crash dump is > generated also. > > One question: what else can start regularly weekly which is not > anacron? > Any other advice is welcome, I am clueless at this moment. > > Thanks, > Csaba > > > -- > -- > You received this message because you are subscribed to the Google > Groups "NLUG" group. > To post to this group, send email to nlug-talk@googlegroups.com > To unsubscribe from this group, send email to nlug-talk > +unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nlug-talk?hl=en > > --- > You received this message because you are subscribed to the Google > Groups "NLUG" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to nlug-talk+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > -- -- You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to nlug-talk@googlegroups.com To unsubscribe from this group, send email to nlug-talk+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en --- You received this message because you are subscribed to the Google Groups "NLUG" group. To unsubscribe from this group and stop receiving emails from it, send an email to nlug-talk+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
