> From: uker...@gmail.com
> Date: Tue, 5 Jul 2016 21:14:12 +0800
> To: kenneth.waege...@ugent.be
> CC: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] mds0: Behind on trimming (58621/30)
>
> On Tue, Jul 5, 2016 at 7:56 PM, Kenneth Waegeman
> <kenneth.waege...@ugent.be> wrote:
> >
> >
> > On 04/07/16 11:22, Kenneth Waegeman wrote:
> >>
> >>
> >>
> >> On 01/07/16 16:01, Yan, Zheng wrote:
> >>>
> >>> On Fri, Jul 1, 2016 at 6:59 PM, John Spray <jsp...@redhat.com> wrote:
> >>>>
> >>>> On Fri, Jul 1, 2016 at 11:35 AM, Kenneth Waegeman
> >>>> <kenneth.waege...@ugent.be> wrote:
> >>>>>
> >>>>> Hi all,
> >>>>>
> >>>>> While syncing a lot of files to cephfs, our mds cluster got haywire:
> >>>>> the
> >>>>> mdss have a lot of segments behind on trimming: (58621/30)
> >>>>> Because of this the mds cluster gets degraded. RAM usage is about 50GB.
> >>>>> The
> >>>>> mdses were respawning and replaying continiously, and I had to stop all
> >>>>> syncs , unmount all clients and increase the beacon_grace to keep the
> >>>>> cluster up .
> >>>>>
> >>>>> [root@mds03 ~]# ceph status
> >>>>> cluster 92bfcf0a-1d39-43b3-b60f-44f01b630e47
> >>>>> health HEALTH_WARN
> >>>>> mds0: Behind on trimming (58621/30)
> >>>>> monmap e1: 3 mons at
> >>>>>
> >>>>> {mds01=10.141.16.1:6789/0,mds02=10.141.16.2:6789/0,mds03=10.141.16.3:6789/0}
> >>>>> election epoch 170, quorum 0,1,2 mds01,mds02,mds03
> >>>>> fsmap e78658: 1/1/1 up {0=mds03=up:active}, 2 up:standby
> >>>>> osdmap e19966: 156 osds: 156 up, 156 in
> >>>>> flags sortbitwise
> >>>>> pgmap v10213164: 4160 pgs, 4 pools, 253 TB data, 203 Mobjects
> >>>>> 357 TB used, 516 TB / 874 TB avail
> >>>>> 4151 active+clean
> >>>>> 5 active+clean+scrubbing
> >>>>> 4 active+clean+scrubbing+deep
> >>>>> client io 0 B/s rd, 0 B/s wr, 63 op/s rd, 844 op/s wr
> >>>>> cache io 68 op/s promote
> >>>>>
> >>>>>
> >>>>> Now it finally is up again, it is trimming very slowly (+-120 segments
> >>>>> /
> >>>>> min)
> >>>>
> >>>> Hmm, so it sounds like something was wrong that got cleared by either
> >>>> the MDS restart or the client unmount, and now it's trimming at a
> >>>> healthier rate.
> >>>>
> >>>> What client (kernel or fuse, and version)?
> >>>>
> >>>> Can you confirm that the RADOS cluster itself was handling operations
> >>>> reasonably quickly? Is your metadata pool using the same drives as
> >>>> your data? Were the OSDs saturated with IO?
> >>>>
> >>>> While the cluster was accumulating untrimmed segments, did you also
> >>>> have a "client xyz failing to advanced oldest_tid" warning?
> >>>
> >>> This does not prevent MDS from trimming log segment.
> >>>
> >>>> It would be good to clarify whether the MDS was trimming slowly, or
> >>>> not at all. If you can reproduce this situation, get it to a "behind
> >>>> on trimming" state, and the stop the client IO (but leave it mounted).
> >>>> See if the (x/30) number stays the same. Then, does it start to
> >>>> decrease when you unmount the client? That would indicate a
> >>>> misbehaving client.
> >>>
> >>> Behind on trimming on single MDS cluster should be caused by either
> >>> slow rados operations or MDS trim too few log segments on each tick.
> >>>
> >>> Kenneth, could you try setting mds_log_max_expiring to a large value
> >>> (such as 200)
> >>
> >> I've set the mds_log_max_expiring to 200 right now. Should I see something
> >> instantly?
> >
> > The trimming finished rather quick, although I don't have any accurate time
> > measures. Cluster looks running fine right now, but running incremental
> > sync. We will try with same data again to see if it is ok now.
> > Is this mds_log_max_expiring option production ready ? (Don't seem to find
> > it in documentation)
>
> It should be safe. Setting mds_log_max_expiring to 200 does not change
> the code path
>
> Yan, Zheng
>
> >
Zheng, Bump up this conf from 20 -> 200 seems increase the load(concurrent)
of flushing? would you prefer make this default?
Xiaoxi
> > Thank you!!
> >
> > K
> >
> >>
> >> This weekend , the trimming did not contunue and something happened to the
> >> cluster:
> >>
> >> mds.0.cache.dir(1000da74e85) commit error -2 v 2466977
> >> log_channel(cluster) log [ERR] : failed to commit dir 1000da74e85 object,
> >> errno -2
> >> mds.0.78429 unhandled write error (2) No such file or directory, force
> >> readonly...
> >> mds.0.cache force file system read-only
> >> log_channel(cluster) log [WRN] : force file system read-only
> >>
> >> and ceph health reported:
> >> mds0: MDS in read-only mode
> >>
> >> I restarted it and it is trimming again.
> >>
> >>
> >> Thanks again!
> >> Kenneth
> >>>
> >>> Regards
> >>> Yan, Zheng
> >>>
> >>>> John
> >>>> _______________________________________________
> >>>> ceph-users mailing list
> >>>> ceph-users@lists.ceph.com
> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >>
> >> _______________________________________________
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
