Hi, can you try to get a stack trace from ganesha (with gdb or from procfs) when it's stuck? Also, try to upgrade to ganesha 2.6. I'm running a bigger deployment with ~30 ganesha 2.6 gateways that are quite stable so far.
Paul 2018-04-16 17:30 GMT+02:00 Oliver Freyermuth <freyerm...@physik.uni-bonn.de> : > Am 16.04.2018 um 08:58 schrieb Oliver Freyermuth: > > Am 16.04.2018 um 02:43 schrieb Oliver Freyermuth: > >> Am 15.04.2018 um 23:04 schrieb John Spray: > >>> On Fri, Apr 13, 2018 at 5:16 PM, Oliver Freyermuth > >>> <freyerm...@physik.uni-bonn.de> wrote: > >>>> Dear Cephalopodians, > >>>> > >>>> in our cluster (CentOS 7.4, EC Pool, Snappy compression, Luminous > 12.2.4), > >>>> we often have all (~40) clients accessing one file in readonly mode, > even with multiple processes per client doing that. > >>>> > >>>> Sometimes (I do not yet know when, nor why!) the MDS ends up in a > situation like: > >>>> ----------------------------------------------------------- > >>>> 2018-04-13 18:08:34.378888 7f1ce4472700 0 log_channel(cluster) log > [WRN] : 292 slow requests, 5 included below; oldest blocked for > > 1745.864417 secs > >>>> 2018-04-13 18:08:34.378900 7f1ce4472700 0 log_channel(cluster) log > [WRN] : slow request 960.563534 seconds old, received at 2018-04-13 > 17:52:33.815273: client_request(client.34720:16487379 getattr pAsLsXsFs > #0x1000009ff6d 2018-04-13 17:52:33.814904 caller_uid=94894, > caller_gid=513{513,}) currently failed to rdlock, waiting > >>>> 2018-04-13 18:08:34.378904 7f1ce4472700 0 log_channel(cluster) log > [WRN] : slow request 30.636678 seconds old, received at 2018-04-13 > 18:08:03.742128: client_request(client.34302:16453640 getattr pAsLsXsFs > #0x1000009ff6d 2018-04-13 18:08:03.741630 caller_uid=94894, > caller_gid=513{513,}) currently failed to rdlock, waiting > >>>> 2018-04-13 18:08:34.378908 7f1ce4472700 0 log_channel(cluster) log > [WRN] : slow request 972.648926 seconds old, received at 2018-04-13 > 17:52:21.729881: client_request(client.34720:16487334 lookup > #0x1000001fcab/sometarball.tar.gz 2018-04-13 17:52:21.729450 > caller_uid=94894, caller_gid=513{513,}) currently failed to rdlock, waiting > >>>> 2018-04-13 18:08:34.378913 7f1ce4472700 0 log_channel(cluster) log > [WRN] : slow request 1685.953657 seconds old, received at 2018-04-13 > 17:40:28.425149: client_request(client.34810:16564864 lookup > #0x1000001fcab/sometarball.tar.gz 2018-04-13 17:40:28.424961 > caller_uid=94894, caller_gid=513{513,}) currently failed to rdlock, waiting > >>>> 2018-04-13 18:08:34.378918 7f1ce4472700 0 log_channel(cluster) log > [WRN] : slow request 1552.743795 seconds old, received at 2018-04-13 > 17:42:41.635012: client_request(client.34302:16453566 getattr pAsLsXsFs > #0x1000009ff6d 2018-04-13 17:42:41.634726 caller_uid=94894, > caller_gid=513{513,}) currently failed to rdlock, waiting > >>>> ----------------------------------------------------------- > >>>> As you can see (oldest blocked for > 1745.864417 secs) it stays in > that situation for quite a while. > >>>> The number of blocked requests is also not decreasing, but instead > slowly increasing whenever a new request is added to the queue. > >>>> > >>>> We have a setup of one active MDS, a standby-replay, and a standby. > >>>> Triggering a failover does not help, it only resets the "oldest > blocked" time. > >>> > >>> Sounds like a client issue (a client is holding a lock on a file but > >>> failing to relinquish it for another client's request to be > >>> processed). > >>> > >>> Are these kernel (and what version?) or fuse clients? > >> > >> The full cluster is running with Fuse clients, all on 12.2.4. > >> Additionally, there are 6 NFS Ganesha servers using FSAL_CEPH, i.e. > libcephfs. > >> > >> Cheers, > >> Oliver > > > > Related to that: In case it happens again, which it surely will sooner > or later, how can I diagnose > > which client is holding a lock and not relinquishing it? > > Is there a way to dump all held locks, ideally with the time period how > long they were held? > > > > ceph daemon mds.XXX help > > does not yield anything obvious. > > > > Cheers and thanks, > > Oliver > > It happened again - and I managed to track down the client causing it: > Dumping all operations in progress on the MDS, and looking for machines > without operations which were waiting for an rdlock, > one NFS Ganesha server was not affected. > Unmounting the NFS share on the clients did not have any effect - things > stayed in "stuck" state. > > I then stopped the NFS Ganesha server. This took quite a while, until > systemd noticed it's taking too long, and force-killed the server. > Shortly after, even before the NFS Ganesha libchepfs client was evicted, > the MDS got unstuck, processed all requests within about 2 seconds, and > everything was fine again. > > After that, the dead NFS Ganesha client was evicted, and I just restarted > the NFS Ganesha server. > > Since this appears to be reproducible (even though we still do not know > how exactly), this is a rather ugly issue. It seems NFS Ganesha with > libcephfs is holding locks > and not returning them. We are using the packages from: > https://eu.ceph.com/nfs-ganesha/ > Newer versions are available upstream, but I don't know if they fix this > critial issue. > > Is there somebody else with experience on that? > > As far as I can reproduce, the I/O pattern is: > 1. Machine with NFS-mount checks datafolder/sometarball.tar.gz exists. > 2. Machines in the cluster (with ceph-fuse) all access > datafolder/sometarball.tar.gz (and extract it to a local filesystem) > > (1) may happen several times while (2) is going on, and it seems this > somehow causes the NFS-server to (sometimes?) keep a persistent lock which > is never relinquished. > > Any help appreciated! > Oliver > > > > > >> > >>> > >>> John > >>> > >>>> > >>>> I checked the following things on the active MDS: > >>>> ----------------------------------------------------------- > >>>> # ceph daemon mds.mon001 objecter_requests > >>>> { > >>>> "ops": [], > >>>> "linger_ops": [], > >>>> "pool_ops": [], > >>>> "pool_stat_ops": [], > >>>> "statfs_ops": [], > >>>> "command_ops": [] > >>>> } > >>>> # ceph daemon mds.mon001 ops | grep event | grep -v "initiated" | > grep -v "failed to rdlock" | grep -v events > >>>> => no output, only "initiated" and "rdlock" are in the queue. > >>>> ----------------------------------------------------------- > >>>> > >>>> There's also almost no CPU load, almost no other I/O, and ceph is > deep-scrubbing ~pg (this also finishes and the next pg is scrubbed fine), > >>>> and the scrubbing is not even happening in the metadata pool (easy to > see in the Luminous dashboard): > >>>> ----------------------------------------------------------- > >>>> # ceph -s > >>>> cluster: > >>>> id: some_funny_hash > >>>> health: HEALTH_WARN > >>>> 1 MDSs report slow requests > >>>> > >>>> services: > >>>> mon: 3 daemons, quorum mon003,mon001,mon002 > >>>> mgr: mon001(active), standbys: mon002, mon003 > >>>> mds: cephfs_baf-1/1/1 up {0=mon001=up:active}, 1 > up:standby-replay, 1 up:standby > >>>> osd: 196 osds: 196 up, 196 in > >>>> > >>>> data: > >>>> pools: 2 pools, 4224 pgs > >>>> objects: 15649k objects, 61761 GB > >>>> usage: 114 TB used, 586 TB / 700 TB avail > >>>> pgs: 4223 active+clean > >>>> 1 active+clean+scrubbing+deep > >>>> > >>>> io: > >>>> client: 175 kB/s rd, 3 op/s rd, 0 op/s wr > >>>> ----------------------------------------------------------- > >>>> > >>>> Does anybody have any idea what's going on here? > >>>> > >>>> Yesterday, this also happened, but resolved itself after about 1 hour. > >>>> Right now, it's going on for about half an hour... > >>>> > >>>> Cheers, > >>>> Oliver > >>>> > >>>> > >>>> _______________________________________________ > >>>> ceph-users mailing list > >>>> ceph-users@lists.ceph.com > >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>>> > >> > >> > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- -- Paul Emmerich croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com