Hi Matthew,
I am not quite sure about the POLLRDHUP.
On the server side (ceph-mon), tcp_read_wait does see the
POLLHUP - which should be the indicator that the
the other side is shutting down.
I have also taken a brief look at the client side (ceph mon stat).
It initiates a shutdown - but never finishes. See attached log file
from "ceph --log-file ceph-mon-stat.rsockets --debug-ms 30 mon stat".
I have also attached the corresponding log file for regualr TCP/IP
sockets.
It looks to me that in the rsockets case, the reaper is able to cleanup
even though there is still sth. left to do - and hence the shutdown
never completes.
Best Regards
Andreas Bluemle
On Mon, 12 Aug 2013 15:11:47 +0800
Matthew Anderson wrote:
> Hi Andreas,
>
> I think we're both working on the same thing, I've just changed the
> function calls over to rsockets in the source instead of using the
> pre-load library. It explains why we're having the exact same problem!
>
> From what I've been able to tell the entire problem revolves around
> rsockets not supporting POLLRDHUP. As far as I can tell the pipe will
> only be removed when tcp_read_wait returns -1. With rsockets it never
> receives the POLLRDHUP event after shutdown_socket() is called so the
> rpoll call blocks until timeout (900 seconds) and the pipe stays
> active.
>
> The question then would be how can we destroy a pipe without relying
> on POLLRDHUP? shutdown_socket() always gets called when the socket
> should be closed so could there might be a way to trick
> tcp_read_wait() into returning -1 by doing somethere in
> shutdown_socket() but I'm not sure how to go about it.
>
> Any ideas?
>
>
>
> On Mon, Aug 12, 2013 at 1:55 PM, Andreas Bluemle <
> andreas.blue...@itxperts.de> wrote:
>
> > Hi Matthew,
> >
> >
> > On Fri, 9 Aug 2013 09:11:07 +0200
> > Matthew Anderson wrote:
> >
> > > So I've had a chance to re-visit this since Bécholey Alexandre was
> > > kind enough to let me know how to compile Ceph with the RDMACM
> > > library (thankyou again!).
> > >
> > > At this stage it compiles and runs but there appears to be a
> > > problem with calling rshutdown in Pipe as it seems to just wait
> > > forever for the pipe to close which causes commands like 'ceph
> > > osd tree' to hang indefinitely after they work successfully.
> > > Debug MS is here - http://pastebin.com/WzMJNKZY
> > >
> >
> > I am currently looking at a very similar problem.
> > My test setup is to start ceph-mon monitors and check their state
> > using "ceph mon stat".
> >
> > The monitors (3 instances) and the "ceph mon stat" command are
> > started with LD_PRELOAD=.
> >
> > The behaviour is that the "ceph mon stat" command connects, sends
> > the request and receives the answer, which shows a healthy state
> > for the monitors. But the "ceph mon stat" does not terminate.
> >
> > On the monitor end I encounter an EOPNOTSUPP being set at the time
> > the connection shall terminate. This is detected in the
> > Pipe::tcp_read_wait() where the socket is poll'ed for IN and HUP
> > events.
> >
> > What I have found out already is that it is not the poll() / rpoll()
> > which set the error: they do return a HUP event and are happy.
> > As far as I can tell, the fact of the EOPNOTSUPP being set is
> > historical at that point, i.e. it must have been set at some
> > earlier stage.
> >
> > I am using ceph v0.61.7.
> >
> >
> > Best Regards
> >
> > Andreas
> >
> >
> > > I also tried RADOS bench but it appears to be doing something
> > > similar. Debug MS is here - http://pastebin.com/3aXbjzqS
> > >
> > > It seems like it's very close to working... I must be missing
> > > something small that's causing some grief. You can see the OSD
> > > coming up in the ceph monitor and the PG's all become
> > > active+clean. When shutting down the monitor I get the below
> > > which show's it waiting for the pipes to close -
> > >
> > > 2013-08-09 15:08:31.339394 7f4643cfd700 20 accepter.accepter
> > > closing 2013-08-09 15:08:31.382075 7f4643cfd700 10
> > > accepter.accepter stopping 2013-08-09 15:08:31.382115
> > > 7f464bd397c0 20 -- 172.16.0.1:6789/0 wait: stopped accepter
> > > thread 2013-08-09 15:08:31.382127 7f464bd397c0 20 --
> > > 172.16.0.1:6789/0 wait: stopping reaper thread 2013-08-09
> > > 15:08:31.382146 7f4645500700 10 -- 172.16.0.1:6789/0 reaper_entry
> >
