Writing as the original reporter of that bug, here's a bit more info.. On 28/11/16 10:34, William Hay wrote: > What version of grid engine are you using?
The bug and the comments below refer to the Fedora packaged version, which will be quite a bit behind SoGE. Yes, we plan to try out SoGE on our next OS update. > Does setting SGE_ND not prevent forking for you? There is an issue that shadowd (at least in the Fedora packaged version) still daemonises even with SGE_ND set. In order to prevent that, one also has to set SGE_DEBUG_LEVEL="1 0 0 0 0 0 0 0" (minimal?), which results in a lot of spam if that node becomes the master. Even filtering this doesn't help a lot, as it burns a lot of CPU printing and dropping the lines. If that's still true in SoGE, making it respect SGE_ND would be a nice fix. > If setting SGE_ND does prevent forking and the problem is exessive log > messages then an example > of the messages in question would be helpful. Here's a small snippet from last night - it's just a few lines every few minutes. Mostly it's a regular scheduler report (does this go to stdout? maybe I should filter it!) and one legitimate error (definitely don't want to filter this). [root@gridmaster6 ~]# journalctl --since yesterday | grep sge_qmaster =========================== bit spammy: Nov 27 02:44:36 gridmaster6.npm.ac.uk sge_qmaster[48965]: --------------STOP-SCHEDULER-RUN------------- Nov 27 02:44:36 gridmaster6.npm.ac.uk sge_qmaster[48965]: Q:840, AQ:859 J:856(856), H:208(208), C:75, A:8, D:1, P:1, CKPT:0, US:18, PR:5, RQS:0, AR:0, S:nd:0/lf:0 Nov 27 02:44:36 gridmaster6.npm.ac.uk sge_qmaster[48965]: --------------STOP-SCHEDULER-RUN------------- Nov 27 02:44:36 gridmaster6.npm.ac.uk sge_qmaster[48965]: Q:840, AQ:859 J:855(855), H:208(208), C:75, A:8, D:1, P:1, CKPT:0, US:18, PR:5, RQS:0, AR:0, S:nd:0/lf:0 Nov 27 02:44:36 gridmaster6.npm.ac.uk sge_qmaster[48965]: --------------STOP-SCHEDULER-RUN------------- Nov 27 02:44:36 gridmaster6.npm.ac.uk sge_qmaster[48965]: Q:840, AQ:859 J:854(854), H:208(208), C:75, A:8, D:1, P:1, CKPT:0, US:18, PR:5, RQS:0, AR:0, S:nd:0/lf:0 Nov 27 02:53:39 gridmaster6.npm.ac.uk sge_qmaster[48965]: legitimate error (interleaved into the messages, so maybe it is stdout+stderr mixing): --------------STOP-Sjob 7561450.1 failed on host gridnode46 assumedly after job because: job 7561450.1 died through signal KILL (9) spam continues: Nov 27 02:58:06 gridmaster6.npm.ac.uk sge_qmaster[48965]: CHEDULER-RUN------------- Nov 27 02:58:06 gridmaster6.npm.ac.uk sge_qmaster[48965]: Q:840, AQ:859 J:855(855), H:208(208), C:75, A:8, D:1, P:1, CKPT:0, US:18, PR:5, RQS:0, AR:0, S:nd:0/lf:0 Nov 27 02:58:06 gridmaster6.npm.ac.uk sge_qmaster[48965]: --------------STOP-SCHEDULER-RUN------------- Nov 27 02:58:06 gridmaster6.npm.ac.uk sge_qmaster[48965]: Q:840, AQ:859 J:854(854), H:208(208), C:75, A:8, D:1, P:1, CKPT:0, US:18, PR:5, RQS:0, AR:0, S:nd:0/lf:0 =========================== Cheers, Mike. P.S. Here's the relevant configs, just for info. This isn't quite what's done in the Fedora patch, afaik, but we haven't updated yet! [root@gridmaster6 ~]# cat /etc/systemd/system/sgemaster.service -------------------------------------- # mggr 20121212 add dependency on autofs, because the grid engine spools are on NFS # mggr 20130423 remove lock file blocking shadow master takeovers # mggr 20140328 change type from "forking" to "simple" and force service not to daemonise by adding "SGE_ND=true" to /etc/sysconfig/gridengine; forking mode causes segfaults, see RH bugzilla #1082129 [Unit] Description=Gridengine master daemon and scheduler After=network.target autofs.service [Service] Type=simple EnvironmentFile=-/etc/sysconfig/gridengine ExecStart=/usr/bin/sge_qmaster # Remove the lock file that prevents shadow masters taking over after a clean shutdown ExecStopPost=/bin/su - sgeadmin -c 'rm -f $SGE_ROOT/$SGE_CELL/qmaster/lock' [Install] WantedBy=multi-user.target -------------------------------------- [root@gridmaster6 ~]# cat /etc/sysconfig/gridengine -------------------------------------- SGE_ROOT=/usr/share/gridengine # uncomment one of these to determine which grid we run on SGE_CELL=rsg-cell #SGE_CELL=rsgdev-cell # prevent SGE from daemonising qmaster, etc # required for systemd to control this as a "Type=simple" service # see bugzilla #1082129 SGE_ND=true # this much debug spam is a minimum requirement, or shadowd still daemonises :( # don't set this on the machine that's expected to be the QMaster, as it eats CPU # do set it on machines that are shadowds #SGE_DEBUG_LEVEL="1 0 0 0 0 0 0 0" # uncomment for crazy debug output #SGE_DEBUG_LEVEL="3 0 0 3 0 0 3 0" -------------------------------------- Please visit our new website at www.pml.ac.uk and follow us on Twitter @PlymouthMarine Winner of the Environment & Conservation category, the Charity Awards 2014. Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK. This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses. _______________________________________________ SGE-discuss mailing list [email protected] https://arc.liv.ac.uk/mailman/listinfo/sge-discuss
