Re: [ceph-users] CephFS fuse client users stuck

2017-04-06 Thread Andras Pataki
Hi John, Have you managed to reproduce the test case on your side? Any hints on how to proceed, or if anything I could help with? I've been trying to understand the protocol between the MDS and the fuse client, but if you can point me to any docs on the rationale of what the implementation i

Re: [ceph-users] CephFS fuse client users stuck

2017-03-31 Thread Andras Pataki
Several clients on one node also works well for me (I guess the fuse client arbitrates then and the MDS perhaps doesn't need to do so much). So the clients need to be on different nodes for this test to fail. Andras On 03/31/2017 01:25 PM, John Spray wrote: On Fri, Mar 31, 2017 at 1:27 PM,

Re: [ceph-users] CephFS fuse client users stuck

2017-03-31 Thread John Spray
On Fri, Mar 31, 2017 at 1:27 PM, Andras Pataki wrote: > Hi John, > > It took a while but I believe now I have a reproducible test case for the > capabilities being lost issue in CephFS I wrote about a couple of weeks ago. > The quick summary of problem is that often processes hang using CephFS > e

Re: [ceph-users] CephFS fuse client users stuck

2017-03-31 Thread Andras Pataki
Hi John, It took a while but I believe now I have a reproducible test case for the capabilities being lost issue in CephFS I wrote about a couple of weeks ago. The quick summary of problem is that often processes hang using CephFS either for a while or sometimes indefinitely. The fuse clien

Re: [ceph-users] CephFS fuse client users stuck

2017-03-16 Thread Dan van der Ster
On Tue, Mar 14, 2017 at 5:55 PM, John Spray wrote: > On Tue, Mar 14, 2017 at 2:10 PM, Andras Pataki > wrote: >> Hi John, >> >> I've checked the MDS session list, and the fuse client does appear on that >> with 'state' as 'open'. So both the fuse client and the MDS agree on an >> open connection.

Re: [ceph-users] CephFS fuse client users stuck

2017-03-14 Thread Andras Pataki
Thanks for the decoding of the logs, now I see what to look for. Can you point me to any documentation that explains a bit more on the logic (about capabilities, Fb/Fw, how the communication between the client and the MDS works, etc.)? I've tried running the client and the MDS at log level 20,

Re: [ceph-users] CephFS fuse client users stuck

2017-03-14 Thread John Spray
On Tue, Mar 14, 2017 at 2:10 PM, Andras Pataki wrote: > Hi John, > > I've checked the MDS session list, and the fuse client does appear on that > with 'state' as 'open'. So both the fuse client and the MDS agree on an > open connection. > > Attached is the log of the ceph fuse client at debug lev

Re: [ceph-users] CephFS fuse client users stuck

2017-03-14 Thread Andras Pataki
Hi John, I've checked the MDS session list, and the fuse client does appear on that with 'state' as 'open'. So both the fuse client and the MDS agree on an open connection. Attached is the log of the ceph fuse client at debug level 20. The MDS got restarted at 9:44:20, and it went through

Re: [ceph-users] CephFS fuse client users stuck

2017-03-14 Thread Henrik Korkuc
On 17-03-14 00:08, John Spray wrote: On Mon, Mar 13, 2017 at 8:15 PM, Andras Pataki wrote: Dear Cephers, We're using the ceph file system with the fuse client, and lately some of our processes are getting stuck seemingly waiting for fuse operations. At the same time, the cluster is healthy, n

Re: [ceph-users] CephFS fuse client users stuck

2017-03-13 Thread John Spray
On Mon, Mar 13, 2017 at 8:15 PM, Andras Pataki wrote: > Dear Cephers, > > We're using the ceph file system with the fuse client, and lately some of > our processes are getting stuck seemingly waiting for fuse operations. At > the same time, the cluster is healthy, no slow requests, all OSDs up an