Hello Reuti,
thanks for a clarification, I probably misunderstand man pages and used
both parameters: s_rt and -notify together before.
Our goal is to configure exactly working backfilling. We use for s_rt a
mean value of running time of finished jobs based on their category.
In a queue configuration is a big value for notify time.
When s_rt is expired, job is signaled but not killed thanks to notify time.
We receive an information in execd messages file:
"11/29/2011 11:11:28| main|service65|W|job 112756.1 exceeded soft
wallclock time - initiate soft notify method"
The job is correctly finished.
This is valid for the single core job or for the parallel job running only
inside 1 node.
I case of bigger parallel jobs they are killed directly after s_rt
expiration. It seems that GE considers s_rt as h_rt ?!
execd messages (master node):
"11/29/2011 14:06:33| main|n15|W|job 112781.1 exceeded hard wallclock time
- initiate terminate method"
qmaster messages
"11/29/2011 14:06:34|worker|sged3|E|master task of job 112781.1 failed -
killing job
11/29/2011 14:06:35|worker|ged3|W|job 112781.1 failed on host n15 assumedly
after job because: job 112781.1 died through signal KILL (9)"
execd strace:
"17649 open("/<path>/sge_spool//n15/messages", O_WRONLY|O_CREAT|O_APPEND,
0666) = 5
17649 write(5, "11/29/2011 14:06:33| main|r8i1n"..., 107) = 107
17649 close(5) = 0
17649 kill(2152, SIGTSTP) = 0
17649 futex(0x7f705cd06e04, 0x189 /* FUTEX_??? */, 4655823, {1322571994,
846698000}, ffffffff <unfinished ...>
17650 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
out)
17650 futex(0x7f705cd068f0, FUTEX_WAKE_PRIVATE, 1) = 0
17650 futex(0x7f705cd06954, 0x189 /* FUTEX_??? */, 4332213, {1322571994,
891528000}, ffffffff <unfinished ...>
17653 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
out)
17653 futex(0x7f705cd07100, FUTEX_WAKE_PRIVATE, 1) = 0
17653 futex(0x7f705cd07164, 0x189 /* FUTEX_??? */, 4440135, {1322571995,
703424000}, ffffffff <unfinished ...>
17649 <... futex resumed> ) = ? ERESTART_RESTARTBLOCK (To be
restarted)
17649 --- SIGCHLD (Child exited) @ 0 (0) ---
17649 rt_sigreturn(0x11) = -1 EINTR (Interrupted system
call)"
Do you have an idea what could be wrong?
Thanks
Best regards
baf035
Dne 22. listopadu 2011 16:16 Reuti <[email protected]> napsal(a):
> Hi,
>
> Am 22.11.2011 um 12:58 schrieb baf035:
>
> >
> > Hello all.
> >
> > 1) When s_rt time is reached a job is signaled by default by SIGUSR1.
> One of our applications is dying by the signal.
> > I changed in a global configuration in execd_params a parameter
> NOTIFY_KILL=none or NOTIFY_KILL=SIGCONT
> > bur without effect. Applications are still signaled by SIGUSR1. How
> could I change the behavior?
>
> these are two different things.
>
> a) when you defined s_rt, it will send SIGUSR1 at the specified time -
> there is no way to avoid this. If you don't want it, don't define s_rt.
>
> b) the notification by -notify will be send the specified time before the
> final kill/suspend. These notifications can be redefined by the commands
> you used. If you don't want it, don't use -notify at submission time.
>
> NB: The signal will be send to the complete processgroup, i.e. the job
> script and the called binary. If you want to have an action happening on
> SIGUSR1, you need a proper set "trap" for the shell and/or a proper signal
> handling in the binary. The default action for SIGUSR1 is to terminate:
> http://kernel.org/doc/man-pages/online/pages/man7/signal.7.html So you
> just face the default behavior.
>
>
> > 2) Is the way how to reduce warning messages "job <JID>.1should have
> finished since <num> s" in a qmaster messages file
> > when s_rt time is exceeded and a notify time is going on.
>
> Not that I'm aware of. But using s_rt and -notify at the same time doesn't
> work in a nice way together is my experience. I suggest to use only one of
> them at a time. Do you have a 2 stage warning in mind, which you want to
> implement? Using both the signal chain would be: 1) s_rt (sigusr1) warning,
> 2) -notify defined warning before kill, 3) kill
>
> -- Reuti
>
>
> > Our productive GE is SoGE 8.0.0a.
> >
> > Thanks for help. Best regards
> >
> > baf035
> >
> > _______________________________________________
> > users mailing list
> > [email protected]
> > https://gridengine.org/mailman/listinfo/users
>
>
_______________________________________________
users mailing list
[email protected]
https://gridengine.org/mailman/listinfo/users
