Hi Lars,
I attach strace file when a problem reappeared at the end of last year.
I used glue which applied your patch for confirmation.
It is the file which I picked with attrd by strace -p command right before I
stop Heartbeat.
Finally SIGTERM caught it, but attrd did not stop.
The attrd stopped afterwards when I sent SIGKILL.
* I acquire the information such as ltrace from now on.
Best Regards,
Hideo Yamauchi.
--- On Thu, 2012/1/5, renayama19661...@ybb.ne.jp <renayama19661...@ybb.ne.jp>
wrote:
> Hi Lars,
>
> > If you are able to reproduce,
> > you could try to find out what exactly attrd is doing.
> >
> > various ways to try to do that:
> > cat /proc/<pid-of-attrd>/stack # if your platform supports that
> > strace it,
> > ltrace it,
> > attach with gdb and provide a stack trace, or even start to single step it,
> > cause attrd to core dump, and analyse the core.
>
> All right.
> I investigate the cause a little more.
>
> Give me the time for investigation a little more.
>
> Best Regards,
> Hideo Yamauchi.
>
> --- On Fri, 2011/12/30, Lars Ellenberg <lars.ellenb...@linbit.com> wrote:
>
> > On Thu, Dec 22, 2011 at 09:54:47AM +0900, renayama19661...@ybb.ne.jp wrote:
> > > Hi Dejan,
> > > Hi Lars,
> > >
> > > In our environment, the problem recurred with the patch of Mr. Lars.
> > > After a problem occurred, I sent TERM signal, but attrd does not seem to
> > > receive TERM at all.
> >
> > If you are able to reproduce,
> > you could try to find out what exactly attrd is doing.
> >
> > various ways to try to do that:
> > cat /proc/<pid-of-attrd>/stack # if your platform supports that
> > strace it,
> > ltrace it,
> > attach with gdb and provide a stack trace, or even start to single step it,
> > cause attrd to core dump, and analyse the core.
> >
> > > The reconsideration of the patch is necessary for the solution to problem.
> > >
> > >
> > > Best Regards,
> > > Hideo Yamauchi.
> > >
> > >
> > > --- On Tue, 2011/11/15, renayama19661...@ybb.ne.jp
> > > <renayama19661...@ybb.ne.jp> wrote:
> > >
> > > > Hi Dejan,
> > > > Hi Lars,
> > > >
> > > > I understood it.
> > > > I try the operation of the patch in our environment.
> > > >
> > > > To Alan: Will you try a patch?
> > > >
> > > > Best Regards,
> > > > Hideo Yamauchi.
> > > >
> > > > --- On Tue, 2011/11/15, Dejan Muhamedagic <deja...@fastmail.fm> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > On Mon, Nov 14, 2011 at 01:17:37PM +0100, Lars Ellenberg wrote:
> > > > > > On Mon, Nov 14, 2011 at 11:58:09AM +1100, Andrew Beekhof wrote:
> > > > > > > On Mon, Nov 7, 2011 at 8:39 AM, Lars Ellenberg
> > > > > > > <lars.ellenb...@linbit.com> wrote:
> > > > > > > > On Thu, Nov 03, 2011 at 01:49:46AM +1100, Andrew Beekhof wrote:
> > > > > > > >> On Tue, Oct 18, 2011 at 12:19 PM,
> > > > > > > >> <renayama19661...@ybb.ne.jp> wrote:
> > > > > > > >> > Hi,
> > > > > > > >> >
> > > > > > > >> > We sometimes fail in a stop of attrd.
> > > > > > > >> >
> > > > > > > >> > Step1. start a cluster in 2 nodes
> > > > > > > >> > Step2. stop the first node.(/etc/init.d/heartbeat stop.)
> > > > > > > >> > Step3. stop the second node after time passed a
> > > > > > > >> > little.(/etc/init.d/heartbeat
> > > > > > > >> > stop.)
> > > > > > > >> >
> > > > > > > >> > The attrd catches the TERM signal, but does not stop.
> > > > > > > >>
> > > > > > > >> There's no evidence that it actually catches it, only that it
> > > > > > > >> is sent.
> > > > > > > >> I've seen it before but never figured out why it occurs.
> > > > > > > >
> > > > > > > > I had it once tracked down almost to where it occurs, but then
> > > > > > > > got distracted.
> > > > > > > > Yes the signal was delivered.
> > > > > > > >
> > > > > > > > I *think* it had to do with attrd doing a blocking read,
> > > > > > > > or looping in some internal message delivery function too often.
> > > > > > > >
> > > > > > > > I had a quick look at the code again now, to try and remember,
> > > > > > > > but I'm not sure.
> > > > > > > >
> > > > > > > > I *may* be that, because
> > > > > > > > xmlfromIPC(IPC_Channel * ch, int timeout) calls
> > > > > > > > msg = msgfromIPC_timeout(ch, MSG_ALLOWINTR, timeout,
> > > > > > > > &ipc_rc);
> > > > > > > >
> > > > > > > > And MSG_ALLOWINTR will cause msgfromIPC_ll() to
> > > > > > > > IPC_INTR:
> > > > > > > > if ( allow_intr){
> > > > > > > > goto startwait;
> > > > > > > >
> > > > > > > > Depending on the frequency of deliverd signals, it may cause
> > > > > > > > this goto
> > > > > > > > startwait loop to never exit, because the timeout always starts
> > > > > > > > again
> > > > > > > > from the full passed in timeout.
> > > > > > > >
> > > > > > > > If only one signal is deliverd, it may still take 120 seconds
> > > > > > > > (MAX_IPC_DELAY from crm.h) to be actually processed, as the
> > > > > > > > signal
> > > > > > > > handler only raises a flag for the next mainloop iteration.
> > > > > > > >
> > > > > > > > If a (non-fatal) signal is delivered every few seconds,
> > > > > > > > then the goto loop will never timeout.
> > > > > > > >
> > > > > > > > Please someone check this for plausibility ;-)
> > > > > > >
> > > > > > > Most plausible explanation I've heard so far... still odd that
> > > > > > > only
> > > > > > > attrd is affected.
> > > > > > > So what do we do about it?
> > > > > >
> > > > > > Reproduce, and confirm that this is what people are seeing.
> > > > > >
> > > > > > Make attrd non-blocking?
> > > > > >
> > > > > > Fix the ipc layer to not restart the full timeout,
> > > > > > but only the remaining partial time?
> > > > >
> > > > > Lars and I made a quick patch for cluster-glue (attached).
> > > > > Hideo-san, is there a way for you to verify if it helps? The
> > > > > patch is not perfect and under unfavourable circumstances it may
> > > > > still take a long time for the caller to exit, but it'd be good
> > > > > to know if this is the right spot.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Dejan
> > > > >
> > > > > > --
> > > > > > : Lars Ellenberg
> > > > > > : LINBIT | Your Way to High Availability
> > > > > > : DRBD/HA support and consulting http://www.linbit.com
> > > > > >
> > > > > > _______________________________________________
> > > > > > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> > > > > > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > > > > >
> > > > > > Project Home: http://www.clusterlabs.org
> > > > > > Getting started:
> > > > > > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > > > > Bugs:
> > > > > > http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
> > > > >
> > > >
> > >
> > > _______________________________________________
> > > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> > > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > >
> > > Project Home: http://www.clusterlabs.org
> > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> >
> > --
> > : Lars Ellenberg
> > : LINBIT | Your Way to High Availability
> > : DRBD/HA support and consulting http://www.linbit.com
> >
> > DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> >
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>restart_syscall(<... resuming interrupted call ...>) = 1
times({tms_utime=2, tms_stime=1, tms_cutime=0, tms_cstime=0}) = 433738625
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=1, tms_cutime=0, tms_cstime=0}) = 433738625
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=1, tms_cutime=0, tms_cstime=0}) = 433738625
recv(7, 0x95b15be, 3666, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b15be, 3666, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=1, tms_cutime=0, tms_cstime=0}) = 433738625
times({tms_utime=2, tms_stime=1, tms_cutime=0, tms_cstime=0}) = 433738625
times({tms_utime=2, tms_stime=1, tms_cutime=0, tms_cstime=0}) = 433738625
times({tms_utime=2, tms_stime=1, tms_cutime=0, tms_cstime=0}) = 433738625
times({tms_utime=2, tms_stime=1, tms_cutime=0, tms_cstime=0}) = 433738625
recv(8, "\207\0\0\0\315\253\0\0>>>\n__name__=attrd_updat"..., 3468,
MSG_DONTWAIT) = 143
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
rt_sigaction(SIGALRM, {0x6131bf, [], 0}, {SIG_IGN, [], 0}, 8) = 0
alarm(120) = 0
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
alarm(0) = 120
rt_sigaction(SIGALRM, {SIG_IGN, [], 0}, {0x6131bf, [], 0}, 8) = 0
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
time(NULL) = 1324470610
send(3, "<142>Dec 21 21:30:10 attrd: [220"..., 101, MSG_NOSIGNAL) = 101
time(NULL) = 1324470610
send(3, "<143>Dec 21 21:30:10 attrd: [220"..., 112, MSG_NOSIGNAL) = 112
time(NULL) = 1324470610
send(3, "<143>Dec 21 21:30:10 attrd: [220"..., 120, MSG_NOSIGNAL) = 120
time(NULL) = 1324470610
send(3, "<143>Dec 21 21:30:10 attrd: [220"..., 102, MSG_NOSIGNAL) = 102
time(NULL) = 1324470610
send(3, "<142>Dec 21 21:30:10 attrd: [220"..., 122, MSG_NOSIGNAL) = 122
recv(6, 0x95b090f, 2113, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b090f, 2113, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b090f, 2113, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
send(6, "\6\1\0\0\315\253\0\0>>>\n__name__=cib_command"..., 270,
MSG_DONTWAIT|MSG_NOSIGNAL) = 270
rt_sigaction(SIGALRM, {0x6131bf, [], 0}, {SIG_IGN, [], 0}, 8) = 0
alarm(120) = 0
recv(6, "\217\0\0\0\315\253\0\0>>>\n__name__=cib-reply\nt"..., 2113,
MSG_DONTWAIT) = 151
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b09a6, 1962, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
alarm(0) = 120
rt_sigaction(SIGALRM, {SIG_IGN, [], 0}, {0x6131bf, [], 0}, 8) = 0
recv(6, 0x95b09a6, 1962, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b09a6, 1962, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b09a6, 1962, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b09a6, 1962, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b09a6, 1962, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
send(6, "\254\0\0\0\315\253\0\0>>>\n__name__=cib_command"..., 180,
MSG_DONTWAIT|MSG_NOSIGNAL) = 180
rt_sigaction(SIGALRM, {0x6131bf, [], 0}, {SIG_IGN, [], 0}, 8) = 0
alarm(120) = 0
recv(6, "]\1\0\0\315\253\0\0>>>\n__name__=cib-reply\nt"..., 1962, MSG_DONTWAIT)
= 357
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
alarm(0) = 120
rt_sigaction(SIGALRM, {SIG_IGN, [], 0}, {0x6131bf, [], 0}, 8) = 0
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
send(6, "9\2\0\0\315\253\0\0>>>\n__name__=cib_command"..., 577,
MSG_DONTWAIT|MSG_NOSIGNAL) = 577
time(NULL) = 1324470610
send(3, "<142>Dec 21 21:30:10 attrd: [220"..., 101, MSG_NOSIGNAL) = 101
gettimeofday({1324470610, 723450}, NULL) = 0
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
send(4, "\236\0\0\0\315\253\0\0>>>\n__name__=attrd_trigg"..., 166,
MSG_DONTWAIT|MSG_NOSIGNAL) = 166
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738628
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738628
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738628
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738628
recv(7, "\221\0\0\0\315\253\0\0>>>\n__name__=cib-reply\nt"..., 3666,
MSG_DONTWAIT) = 153
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738628
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738628
poll([{fd=7, events=POLLIN|POLLPRI}, {fd=4, events=POLLIN|POLLPRI}], 2, 0) = 0
(Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738628
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738628
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738628
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
rt_sigaction(SIGALRM, {0x6131bf, [], 0}, {SIG_IGN, [], 0}, 8) = 0
alarm(120) = 0
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
alarm(0) = 120
rt_sigaction(SIGALRM, {SIG_IGN, [], 0}, {0x6131bf, [], 0}, 8) = 0
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
time(NULL) = 1324470610
send(3, "<143>Dec 21 21:30:10 attrd: [220"..., 105, MSG_NOSIGNAL) = 105
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(6, 0x95b0b0b, 1605, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=6, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738630
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738630
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738630
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af1a4, 932, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738630
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738630
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738630
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
recv(8, 0x95b29ab, 3325, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=8, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738630
poll([{fd=4, events=POLLIN|POLLPRI}, {fd=7, events=POLLIN|POLLPRI}, {fd=8,
events=POLLIN|POLLPRI}, {fd=5, events=POLLIN|POLLPRI}], 4, -1) = 1 ([{fd=4,
revents=POLLIN}])
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738630
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738631
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738631
recv(4, "\\\1\0\0\315\253\0\0>>>\n__name__=attrd_trigg"..., 932, MSG_DONTWAIT)
= 356
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738631
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738631
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738631
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 433738631
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
time(NULL) = 1324470610
send(3, "<142>Dec 21 21:30:10 attrd: [220"..., 92, MSG_NOSIGNAL) = 92
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738632
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738632
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738632
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738632
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738632
recv(8, "", 3325, MSG_DONTWAIT) = 0
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738632
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738632
poll([{fd=7, events=POLLIN|POLLPRI}, {fd=4, events=POLLIN|POLLPRI}, {fd=8,
events=POLLIN|POLLPRI}], 3, 0) = 1 ([{fd=8, revents=POLLIN|POLLHUP}])
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738632
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738632
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738633
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738633
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738633
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738633
close(8) = 0
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738633
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738633
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
recv(4, 0x95af308, 576, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=4, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738633
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = ? ERESTART_RESTARTBLOCK (To be
restarted)
--- SIGTERM (Terminated) @ 0 (0) ---
sigreturn() = ? (mask now [])
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
recv(7, 0x95b1657, 3513, MSG_DONTWAIT) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=0}], 1, 0) = 0 (Timeout)
times({tms_utime=2, tms_stime=3, tms_cutime=0, tms_cstime=0}) = 433738634
poll([{fd=7, events=POLLIN|POLLPRI}, {fd=4, events=POLLIN|POLLPRI}, {fd=5,
events=POLLIN|POLLPRI}], 3, -1_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
