Dear Yan and Stefan, thanks for the additional information, it should help reproducing the issue.
The pdsh command executes a bash script that echoes a few values to stdout. Access should be read-only, however, we still have the FS mounted with atime enabled, so there is probably meta data write and synchronisation per access. Files accessed are ssh auth-keys in .ssh and the shell script. The shell script was located in the home-dir of the user and, following your explanations, to reproduce the issue I will create a directory with many entries and execute a test with the many-clients single-file-read load on it. I hope it doesn't take too long. Thanks for your input! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Yan, Zheng <uker...@gmail.com> Sent: 16 May 2019 09:35 To: Frank Schilder Subject: Re: [ceph-users] mimic: MDS standby-replay causing blocked ops (MDS bug?) On Thu, May 16, 2019 at 2:52 PM Frank Schilder <fr...@dtu.dk> wrote: > > Dear Yan, > > OK, I will try to trigger the problem again and dump the information > requested. Since it is not easy to get into this situation and I usually need > to resolve it fast (its not a test system), is there anything else worth > capturing? > just ceph daemon mds.x dump_ops_in_flight ceph daemon mds.x dump cache /tmp/cachedump.x > I will get back as soon as it happened again. > > In the meantime, I would be grateful if you could shed some light on the > following questions: > > - Is there a way to cancel an individual operation in the queue? It is a bit > harsh to have to fail an MDS for that. no > - What is the fragmentdir operation doing in a single MDS setup? I thought > this was only relevant if multiple MDS daemons are active on a file system. > It splits large directory to smaller parts. > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Yan, Zheng <uker...@gmail.com> > Sent: 16 May 2019 05:50 > To: Frank Schilder > Cc: Stefan Kooman; ceph-users@lists.ceph.com > Subject: Re: [ceph-users] mimic: MDS standby-replay causing blocked ops (MDS > bug?) > > > [...] > > This time I captured the MDS ops list (log output does not really contain > > more info than this list). It contains 12 ops and I will include it here in > > full length (hope this is acceptable): > > > > Your issues were caused by stuck internal op fragmentdir. Can you > dump mds cache and send the output to us? _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
