[ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-07 Thread 赵贺东 
Hi All,

Every time after we activate osd, we got “Structure needs cleaning” in 
/var/lib/ceph/osd/ceph-xxx/current/meta.


/var/lib/ceph/osd/ceph-xxx/current/meta
# ls -l
ls: reading directory .: Structure needs cleaning
total 0

Could Anyone say something about this error?

Thank you!


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-08 Thread Brad Hubbard 
On Thu, Mar 8, 2018 at 5:01 PM, 赵贺东  wrote:
> Hi All,
>
> Every time after we activate osd, we got “Structure needs cleaning” in 
> /var/lib/ceph/osd/ceph-xxx/current/meta.
>
>
> /var/lib/ceph/osd/ceph-xxx/current/meta
> # ls -l
> ls: reading directory .: Structure needs cleaning
> total 0
>
> Could Anyone say something about this error?

It's an indication of possible corruption on the filesystem containing "meta".

Can you unmount it and run a filesystem check on it?

At the time the filesystem first detected the corruption it would have
logged it to dmesg and possibly syslog which may give you a clue. Did
you lose power or have a kernel panic or something?

>
> Thank you!
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-08 Thread Wido den Hollander 



On 03/08/2018 08:01 AM, 赵贺东 wrote:

Hi All,

Every time after we activate osd, we got “Structure needs cleaning” in 
/var/lib/ceph/osd/ceph-xxx/current/meta.


/var/lib/ceph/osd/ceph-xxx/current/meta
# ls -l
ls: reading directory .: Structure needs cleaning
total 0

Could Anyone say something about this error?



Seems like XFS is broken. I recommend that you wipe that OSD and 
reformat it with ceph-disk/ceph-volume.


Also check the SMART values and verify that the disk isn't broken.

Do not attempt a XFS repair or something.

Wido


Thank you!


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-08 Thread 赵赵贺东 
Hi Brad,

Thank you for your attention.

> 在 2018年3月8日，下午4:47，Brad Hubbard  写道：
> 
> On Thu, Mar 8, 2018 at 5:01 PM, 赵贺东  wrote:
>> Hi All,
>> 
>> Every time after we activate osd, we got “Structure needs cleaning” in 
>> /var/lib/ceph/osd/ceph-xxx/current/meta.
>> 
>> 
>> /var/lib/ceph/osd/ceph-xxx/current/meta
>> # ls -l
>> ls: reading directory .: Structure needs cleaning
>> total 0
>> 
>> Could Anyone say something about this error?
> 
> It's an indication of possible corruption on the filesystem containing "meta".
> 
> Can you unmount it and run a filesystem check on it?
I did some xfs_repair operation, but no effect.Structure needs cleaning” still 
exist.



> 
> At the time the filesystem first detected the corruption it would have
> logged it to dmesg and possibly syslog which may give you a clue. Did
> you lose power or have a kernel panic or something?
We did not lose power.
You are right, we get a metadata corruption in dmesg&syslog every time just 
following the osd activating operation.

[  399.513525] XFS (sda1): Metadata corruption detected at 
xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0x48b9ff80
[  399.524709] XFS (sda1): Unmount and run xfs_repair
[  399.529511] XFS (sda1): First 64 bytes of corrupted metadata buffer:
[  399.535917] dd8f2000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb  
XFSB.s..
[  399.543959] dd8f2010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  

[  399.551983] dd8f2020: e5 30 40 22 51 8f 4f 1c 80 73 56 9b 71 aa 92 24  
.0@"Q.O..sV.q..$
[  399.560037] dd8f2030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff  

[  399.568118] XFS (sda1): metadata I/O error: block 0x48b9ff80 
("xfs_trans_read_buf_map") error 117 numblks 8
[  399.583179] XFS (sda1): Metadata corruption detected at 
xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0x48b9ff80
[  399.594378] XFS (sda1): Unmount and run xfs_repair
[  399.599182] XFS (sda1): First 64 bytes of corrupted metadata buffer:
[  399.605575] e47db000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb  
XFSB.s..
[  399.613613] e47db010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  

[  399.621637] e47db020: e5 30 40 22 51 8f 4f 1c 80 73 56 9b 71 aa 92 24  
.0@"Q.O..sV.q..$
[  399.629679] e47db030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff  

[  399.637856] XFS (sda1): metadata I/O error: block 0x48b9ff80 
("xfs_trans_read_buf_map") error 117 numblks 8
[  399.648165] XFS (sda1): Metadata corruption detected at 
xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0x48b9ff80
[  399.659378] XFS (sda1): Unmount and run xfs_repair
[  399.664196] XFS (sda1): First 64 bytes of corrupted metadata buffer:
[  399.670570] e47db000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb  
XFSB.s..
[  399.678610] e47db010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  

[  399.686643] e47db020: e5 30 40 22 51 8f 4f 1c 80 73 56 9b 71 aa 92 24  
.0@"Q.O..sV.q..$
[  399.694681] e47db030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff  

[  399.702794] XFS (sda1): metadata I/O error: block 0x48b9ff80 
("xfs_trans_read_buf_map") error 117 numblks 8


Thank you !


> 
>> 
>> Thank you!
>> 
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> -- 
> Cheers,
> Brad

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-08 Thread 赵贺东 
Hi Wido,

Thank you for attention!
> 在 2018年3月8日，下午4:21，Wido den Hollander  写道：
> 
> 
> 
> On 03/08/2018 08:01 AM, 赵贺东 wrote:
>> Hi All,
>> Every time after we activate osd, we got “Structure needs cleaning” in 
>> /var/lib/ceph/osd/ceph-xxx/current/meta.
>> /var/lib/ceph/osd/ceph-xxx/current/meta
>> # ls -l
>> ls: reading directory .: Structure needs cleaning
>> total 0
>> Could Anyone say something about this error?
> 
> Seems like XFS is broken. I recommend that you wipe that OSD and reformat it 
> with ceph-disk/ceph-volume.
Because our ceph is run on ubuntu14.04, ceph-volume needs systemd(systemd only 
on ubuntu16.04) support.
It makes things more complicated 

 if we want to use ceph-volume.

> Also check the SMART values and verify that the disk isn't broken.
Because, every disk has the same problem.And only triggered by osd activating 
operation.
If I deploy osd manually , I can see the “Structure needs cleaning” will come 
out just after I try to start ceph-osd daemon;
It means, after I ran “start ceph-osd id=”, and then "ls -l  
/var/lib/ceph/osd/ceph-xxx/current/meta” , there will be “Structure needs 
cleaning"
 
> 
> Do not attempt a XFS repair or something.
Yes xfs_repair has no effect.

Thank you!
> 
> Wido
> 
>> Thank you!
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-08 Thread Brad Hubbard 
On Thu, Mar 8, 2018 at 7:33 PM, 赵赵贺东  wrote:
> Hi Brad,
>
> Thank you for your attention.
>
>> 在 2018年3月8日，下午4:47，Brad Hubbard  写道：
>>
>> On Thu, Mar 8, 2018 at 5:01 PM, 赵贺东  wrote:
>>> Hi All,
>>>
>>> Every time after we activate osd, we got “Structure needs cleaning” in 
>>> /var/lib/ceph/osd/ceph-xxx/current/meta.
>>>
>>>
>>> /var/lib/ceph/osd/ceph-xxx/current/meta
>>> # ls -l
>>> ls: reading directory .: Structure needs cleaning
>>> total 0
>>>
>>> Could Anyone say something about this error?
>>
>> It's an indication of possible corruption on the filesystem containing 
>> "meta".
>>
>> Can you unmount it and run a filesystem check on it?
> I did some xfs_repair operation, but no effect.Structure needs cleaning” 
> still exist.
>
>
>
>>
>> At the time the filesystem first detected the corruption it would have
>> logged it to dmesg and possibly syslog which may give you a clue. Did
>> you lose power or have a kernel panic or something?
> We did not lose power.
> You are right, we get a metadata corruption in dmesg&syslog every time just 
> following the osd activating operation.
>
> [  399.513525] XFS (sda1): Metadata corruption detected at 
> xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0x48b9ff80
> [  399.524709] XFS (sda1): Unmount and run xfs_repair
> [  399.529511] XFS (sda1): First 64 bytes of corrupted metadata buffer:
> [  399.535917] dd8f2000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb  
> XFSB.s..
> [  399.543959] dd8f2010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  
> 
> [  399.551983] dd8f2020: e5 30 40 22 51 8f 4f 1c 80 73 56 9b 71 aa 92 24  
> .0@"Q.O..sV.q..$
> [  399.560037] dd8f2030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff  
> 
> [  399.568118] XFS (sda1): metadata I/O error: block 0x48b9ff80 
> ("xfs_trans_read_buf_map") error 117 numblks 8
> [  399.583179] XFS (sda1): Metadata corruption detected at 
> xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0x48b9ff80
> [  399.594378] XFS (sda1): Unmount and run xfs_repair
> [  399.599182] XFS (sda1): First 64 bytes of corrupted metadata buffer:
> [  399.605575] e47db000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb  
> XFSB.s..
> [  399.613613] e47db010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  
> 
> [  399.621637] e47db020: e5 30 40 22 51 8f 4f 1c 80 73 56 9b 71 aa 92 24  
> .0@"Q.O..sV.q..$
> [  399.629679] e47db030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff  
> 
> [  399.637856] XFS (sda1): metadata I/O error: block 0x48b9ff80 
> ("xfs_trans_read_buf_map") error 117 numblks 8
> [  399.648165] XFS (sda1): Metadata corruption detected at 
> xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0x48b9ff80
> [  399.659378] XFS (sda1): Unmount and run xfs_repair
> [  399.664196] XFS (sda1): First 64 bytes of corrupted metadata buffer:
> [  399.670570] e47db000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb  
> XFSB.s..
> [  399.678610] e47db010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  
> 
> [  399.686643] e47db020: e5 30 40 22 51 8f 4f 1c 80 73 56 9b 71 aa 92 24  
> .0@"Q.O..sV.q..$
> [  399.694681] e47db030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff  
> 
> [  399.702794] XFS (sda1): metadata I/O error: block 0x48b9ff80 
> ("xfs_trans_read_buf_map") error 117 numblks 8

I'd suggest the next step is to look for a matching XFS bug in your
distro and, if possible, try a different distro and see if you get the
same result.

>
>
> Thank you !
>
>
>>
>>>
>>> Thank you!
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> --
>> Cheers,
>> Brad
>



-- 
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-08 Thread 赵贺东 
Thank you for your suggestions.
We will upgrade ubuntu distro and linux kernel to see if the problem still 
exists or not.

> 在 2018年3月8日，下午5:51，Brad Hubbard  写道：
> 
> On Thu, Mar 8, 2018 at 7:33 PM, 赵赵贺东  > wrote:
>> Hi Brad,
>> 
>> Thank you for your attention.
>> 
>>> 在 2018年3月8日，下午4:47，Brad Hubbard  写道：
>>> 
>>> On Thu, Mar 8, 2018 at 5:01 PM, 赵贺东  wrote:
 Hi All,
 
 Every time after we activate osd, we got “Structure needs cleaning” in 
 /var/lib/ceph/osd/ceph-xxx/current/meta.
 
 
 /var/lib/ceph/osd/ceph-xxx/current/meta
 # ls -l
 ls: reading directory .: Structure needs cleaning
 total 0
 
 Could Anyone say something about this error?
>>> 
>>> It's an indication of possible corruption on the filesystem containing 
>>> "meta".
>>> 
>>> Can you unmount it and run a filesystem check on it?
>> I did some xfs_repair operation, but no effect.Structure needs cleaning” 
>> still exist.
>> 
>> 
>> 
>>> 
>>> At the time the filesystem first detected the corruption it would have
>>> logged it to dmesg and possibly syslog which may give you a clue. Did
>>> you lose power or have a kernel panic or something?
>> We did not lose power.
>> You are right, we get a metadata corruption in dmesg&syslog every time just 
>> following the osd activating operation.
>> 
>> [  399.513525] XFS (sda1): Metadata corruption detected at 
>> xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0x48b9ff80
>> [  399.524709] XFS (sda1): Unmount and run xfs_repair
>> [  399.529511] XFS (sda1): First 64 bytes of corrupted metadata buffer:
>> [  399.535917] dd8f2000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb  
>> XFSB.s..
>> [  399.543959] dd8f2010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  
>> 
>> [  399.551983] dd8f2020: e5 30 40 22 51 8f 4f 1c 80 73 56 9b 71 aa 92 24  
>> .0@"Q.O..sV.q..$
>> [  399.560037] dd8f2030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff  
>> 
>> [  399.568118] XFS (sda1): metadata I/O error: block 0x48b9ff80 
>> ("xfs_trans_read_buf_map") error 117 numblks 8
>> [  399.583179] XFS (sda1): Metadata corruption detected at 
>> xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0x48b9ff80
>> [  399.594378] XFS (sda1): Unmount and run xfs_repair
>> [  399.599182] XFS (sda1): First 64 bytes of corrupted metadata buffer:
>> [  399.605575] e47db000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb  
>> XFSB.s..
>> [  399.613613] e47db010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  
>> 
>> [  399.621637] e47db020: e5 30 40 22 51 8f 4f 1c 80 73 56 9b 71 aa 92 24  
>> .0@"Q.O..sV.q..$
>> [  399.629679] e47db030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff  
>> 
>> [  399.637856] XFS (sda1): metadata I/O error: block 0x48b9ff80 
>> ("xfs_trans_read_buf_map") error 117 numblks 8
>> [  399.648165] XFS (sda1): Metadata corruption detected at 
>> xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0x48b9ff80
>> [  399.659378] XFS (sda1): Unmount and run xfs_repair
>> [  399.664196] XFS (sda1): First 64 bytes of corrupted metadata buffer:
>> [  399.670570] e47db000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb  
>> XFSB.s..
>> [  399.678610] e47db010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  
>> 
>> [  399.686643] e47db020: e5 30 40 22 51 8f 4f 1c 80 73 56 9b 71 aa 92 24  
>> .0@"Q.O..sV.q..$
>> [  399.694681] e47db030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff  
>> 
>> [  399.702794] XFS (sda1): metadata I/O error: block 0x48b9ff80 
>> ("xfs_trans_read_buf_map") error 117 numblks 8
> 
> I'd suggest the next step is to look for a matching XFS bug in your
> distro and, if possible, try a different distro and see if you get the
> same result.
> 
>> 
>> 
>> Thank you !
>> 
>> 
>>> 
 
 Thank you!
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> 
>>> 
>>> 
>>> --
>>> Cheers,
>>> Brad
>> 
> 
> 
> 
> -- 
> Cheers,
> Brad

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com