Re: [ceph-users] Update to Mimic with prior Snapshots leads to MDS damaged metadata
Hi! > We are fixing the release note. https://github.com/ceph/ceph/pull/22445 Thank you! It will help others. Cheers, Tobias Florek signature.asc Description: signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Update to Mimic with prior Snapshots leads to MDS damaged metadata
On Thu, Jun 7, 2018 at 2:44 PM, Tobias Florek wrote: > Hi! > > Thank you for your help! The cluster is running healthily for a day now. > > Regarding the problem, I just checked in the release notes [1] and on > docs.ceph.com and did not find the right invocation after an upgrade. > Maybe that ought to be fixed. > We are fixing the release note. https://github.com/ceph/ceph/pull/22445 >>> [upgrade from luminous to mimic with prior cephfs snapshots] >> The correct commands should be: >> >> ceph daemon scrub_path / force recursive repair >> ceph daemon scrub_path '~mdsdir' force recursive > > [1] https://ceph.com/releases/v13-2-0-mimic-released/ > [2] > https://www.google.com/search?q=site%3Adocs.ceph.com+scrub_path+inurl%3Amimic=site%3Adocs.ceph.com+scrub_path+inurl%3Amimic > > Cheers, > Tobias Florek ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Update to Mimic with prior Snapshots leads to MDS damaged metadata
Hi! Thank you for your help! The cluster is running healthily for a day now. Regarding the problem, I just checked in the release notes [1] and on docs.ceph.com and did not find the right invocation after an upgrade. Maybe that ought to be fixed. >> [upgrade from luminous to mimic with prior cephfs snapshots] > The correct commands should be: > > ceph daemon scrub_path / force recursive repair > ceph daemon scrub_path '~mdsdir' force recursive [1] https://ceph.com/releases/v13-2-0-mimic-released/ [2] https://www.google.com/search?q=site%3Adocs.ceph.com+scrub_path+inurl%3Amimic=site%3Adocs.ceph.com+scrub_path+inurl%3Amimic Cheers, Tobias Florek ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Update to Mimic with prior Snapshots leads to MDS damaged metadata
On Wed, Jun 6, 2018 at 3:25 PM, Tobias Florek wrote:
> Hi,
>
> I upgraded a ceph cluster to mimic yesterday according to the release
> notes. Specifically I did stop all standby MDS and then restarted the
> only active MDS with the new version.
>
> The cluster was installed with luminous. Its cephfs volume had snapshots
> prior to the update, but only one active MDS.
>
> The post-installation steps failed though:
> ceph daemon mds. scrub_path /
> returned an error, which I corrected with
> ceph daemon mds. scrub_path / repair
>
> While
> ceph daemon mds. scrub_path '~mdsdir'
> did not show any error.
>
The correct commands should be:
ceph daemon scrub_path / force recursive repair
ceph daemon scrub_path '~mdsdir' force recursive repair
>
> After some time, ceph health reported MDS damaged metadata:
>> ceph tell mds. damage ls | jq '.[].damage_type' | sort | uniq -c
> 398 "backtrace"
> 718 "dentry"
>
> Examples of damage:
>
> {
> "damage_type": "dentry",
> "id": 118195760,
> "ino": 1099513350198,
> "frag": "000100*",
> "dname":
> "1524578400.M820820P705532.dovecot-15-hgjlx,S=425674,W=431250:2,RS",
> "snap_id": "head",
> "path":
> "/path/to/mails/user/Maildir/.Trash/cur/1524578400.M820820P705532.dovecot-15-hgjlx,S=425674,W=431250:2,RS"
> },
> {
> "damage_type": "backtrace",
> "id": 121083841,
> "ino": 1099515215027,
> "path":
> "/path/to/mails/other_user/Maildir/.Junk/cur/1528189963.M416032P698926.dovecot-15-xmpkh,S=4010,W=4100:2,Sab"
> },
>
'ceph daemon scrub_path / force recursive repair'
should also fix above errors.
Regards
Yan, Zheng
>
> Directories with damage can still be listed by the kernel cephfs mount
> (4.16.7), but not the fuse mount, which stalls.
>
>
> Can anyone help? That's unfortunately a production cluster.
>
> Regards,
> Tobias Florek
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Update to Mimic with prior Snapshots leads to MDS damaged metadata
Hi,
I upgraded a ceph cluster to mimic yesterday according to the release
notes. Specifically I did stop all standby MDS and then restarted the
only active MDS with the new version.
The cluster was installed with luminous. Its cephfs volume had snapshots
prior to the update, but only one active MDS.
The post-installation steps failed though:
ceph daemon mds. scrub_path /
returned an error, which I corrected with
ceph daemon mds. scrub_path / repair
While
ceph daemon mds. scrub_path '~mdsdir'
did not show any error.
After some time, ceph health reported MDS damaged metadata:
> ceph tell mds. damage ls | jq '.[].damage_type' | sort | uniq -c
398 "backtrace"
718 "dentry"
Examples of damage:
{
"damage_type": "dentry",
"id": 118195760,
"ino": 1099513350198,
"frag": "000100*",
"dname":
"1524578400.M820820P705532.dovecot-15-hgjlx,S=425674,W=431250:2,RS",
"snap_id": "head",
"path":
"/path/to/mails/user/Maildir/.Trash/cur/1524578400.M820820P705532.dovecot-15-hgjlx,S=425674,W=431250:2,RS"
},
{
"damage_type": "backtrace",
"id": 121083841,
"ino": 1099515215027,
"path":
"/path/to/mails/other_user/Maildir/.Junk/cur/1528189963.M416032P698926.dovecot-15-xmpkh,S=4010,W=4100:2,Sab"
},
Directories with damage can still be listed by the kernel cephfs mount
(4.16.7), but not the fuse mount, which stalls.
Can anyone help? That's unfortunately a production cluster.
Regards,
Tobias Florek
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
