Re: [ceph-users] After 13.2.2 upgrade: bluefs mount failed to replay log: (5) Input/output error
Hi Paul! I installed ceph-debuginfo and set these: debug bluestore = 20/20 debug osd = 20/20 debug bluefs = 20/20 debug bdev = 20/20 V: ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable) *LOGS* *OSD 29:* 2018-10-08 10:29:06.001 7f810511a1c0 20 bluefs _read left 0x4d000 len 0x1000 2018-10-08 10:29:06.001 7f810511a1c0 20 bluefs _read got 4096 2018-10-08 10:29:06.001 7f810511a1c0 10 bluefs _replay 0x12b3000: stop: uuid e510614a-7ca4-eb59-0383-010189889f01 != super.uuid 4df25e30-4769-47b5-b569-01b3f83de70c 2018-10-08 10:29:06.001 7f810511a1c0 10 bluefs _replay log file size was 0x12b3000 2018-10-08 10:29:06.001 7f810511a1c0 -1 bluefs _replay file with link count 0: file(ino 519 size 0x31e2f42 mtime 2018-10-02 12:24:22.632397 bdev 1 allocated 320 extents [1:0x700820+10,1:0x700900+10,1:0x700910+10,1:0x700920+10,1:0x700930+10,1:0x700940+10,1:0x700950+10,1:0x700960+10,1:0x700970+10,1:0x700980+10,1:0x700990+10,1:0x7009a0+10,1:0x7009b0+10,1:0x7009c0+10,1:0x7009d0+10,1:0x7009e0+10,1:0x7009f0+10,1:0x700a00+10,1:0x700a10+10,1:0x700a20+10,1:0x700a30+10,1:0x700a40+10,1:0x700a50+10,1:0x700a60+10,1:0x700a70+10,1:0x700a80+10,1:0x700a90+10,1:0x700aa0+10,1:0x700ab0+10,1:0x700ac0+10,1:0x700ad0+10,1:0x700ae0+10,1:0x700af0+10,1:0x700b00+10,1:0x700b10+10,1:0x700b20+10,1:0x700b30+10,1:0x700b40+10,1:0x700b50+10,1:0x700b60+10,1:0x700b70+10,1:0x700b80+10,1:0x700b90+10,1:0x700ba0+10,1:0x700bb0+10,1:0x700bc0+10,1:0x700bd0+10,1:0x700be0+10,1:0x700bf0+10,1:0x700c00+10]) 2018-10-08 10:29:06.001 7f810511a1c0 -1 bluefs mount failed to replay log: (5) Input/output error 2018-10-08 10:29:06.001 7f810511a1c0 20 bluefs _stop_alloc 2018-10-08 10:29:06.001 7f810511a1c0 10 bdev(0x558b8f1dea80 /var/lib/ceph/osd/ceph-29/block) discard_drain 2018-10-08 10:29:06.001 7f810511a1c0 1 stupidalloc 0x0x558b8f34d0a0 shutdown 2018-10-08 10:29:06.001 7f810511a1c0 -1 bluestore(/var/lib/ceph/osd/ceph-29) _open_db failed bluefs mount: (5) Input/output error 2018-10-08 10:29:06.001 7f810511a1c0 20 bdev aio_wait 0x558b8f34f440 done 2018-10-08 10:29:06.001 7f810511a1c0 1 bdev(0x558b8f1dea80 /var/lib/ceph/osd/ceph-29/block) close 2018-10-08 10:29:06.001 7f810511a1c0 10 bdev(0x558b8f1dea80 /var/lib/ceph/osd/ceph-29/block) _aio_stop 2018-10-08 10:29:06.066 7f80ed75f700 10 bdev(0x558b8f1dea80 /var/lib/ceph/osd/ceph-29/block) _aio_thread end 2018-10-08 10:29:06.073 7f810511a1c0 10 bdev(0x558b8f1dea80 /var/lib/ceph/osd/ceph-29/block) _discard_stop 2018-10-08 10:29:06.073 7f80ecf5e700 20 bdev(0x558b8f1dea80 /var/lib/ceph/osd/ceph-29/block) _discard_thread wake 2018-10-08 10:29:06.073 7f80ecf5e700 10 bdev(0x558b8f1dea80 /var/lib/ceph/osd/ceph-29/block) _discard_thread finish 2018-10-08 10:29:06.073 7f810511a1c0 10 bdev(0x558b8f1dea80 /var/lib/ceph/osd/ceph-29/block) _discard_stop stopped 2018-10-08 10:29:06.073 7f810511a1c0 1 bdev(0x558b8f1de000 /var/lib/ceph/osd/ceph-29/block) close 2018-10-08 10:29:06.073 7f810511a1c0 10 bdev(0x558b8f1de000 /var/lib/ceph/osd/ceph-29/block) _aio_stop 2018-10-08 10:29:06.315 7f80ee761700 10 bdev(0x558b8f1de000 /var/lib/ceph/osd/ceph-29/block) _aio_thread end 2018-10-08 10:29:06.321 7f810511a1c0 10 bdev(0x558b8f1de000 /var/lib/ceph/osd/ceph-29/block) _discard_stop 2018-10-08 10:29:06.321 7f80edf60700 20 bdev(0x558b8f1de000 /var/lib/ceph/osd/ceph-29/block) _discard_thread wake 2018-10-08 10:29:06.321 7f80edf60700 10 bdev(0x558b8f1de000 /var/lib/ceph/osd/ceph-29/block) _discard_thread finish 2018-10-08 10:29:06.321 7f810511a1c0 10 bdev(0x558b8f1de000 /var/lib/ceph/osd/ceph-29/block) _discard_stop stopped 2018-10-08 10:29:06.322 7f810511a1c0 -1 osd.29 0 OSD:init: unable to mount object store 2018-10-08 10:29:06.322 7f810511a1c0 -1 ** ERROR: osd init failed: (5) Input/output error *OSD 40 (keeps getting restarted by systemd):* 2018-10-08 10:33:01.867 7fbdd21441c0 20 read_log_and_missing 4754'11872 (4754'11871) modify 5:fd843365:::1000229.29b3:head by client.1109026.0:2115960 2018-09-23 02:48:36.736842 0 2018-10-08 10:33:01.867 7fbdd21441c0 10 bluefs _read_random h 0x5566a75fb480 0x4a2a19~1036 from file(ino 539 size 0x3fa66ff mtime 2018-10-02 12:19:02.174614 bdev 1 allocated 400 extents [1:0x7004e0+400]) 2018-10-08 10:33:01.867 7fbdd21441c0 20 bluefs _read_random read buffered 0x4a2a19~1036 of 1:0x7004e0+400 2018-10-08 10:33:01.867 7fbdd21441c0 5 bdev(0x5566a70dea80 /var/lib/ceph/osd/ceph-40/block) read_random 0x70052a2a19~1036 2018-10-08 10:33:01.867 7fbdd21441c0 20 bluefs _read_random got 4150 2018-10-08 10:33:01.867 7fbdd21441c0 20 bluestore.OmapIteratorImpl(0x5566bddf5f80) valid i
Re: [ceph-users] After 13.2.2 upgrade: bluefs mount failed to replay log: (5) Input/output error
There's "ceph-bluestore-tool repair/fsck"
In your scenario, a few more log files would be interesting: try
setting debug bluefs to 20/20. And if that's not enough log try also
setting debug osd, debug bluestore, and debug bdev to 20/20.
Paul
Am Mi., 3. Okt. 2018 um 13:48 Uhr schrieb Kevin Olbrich :
>
> The disks were deployed with ceph-deploy / ceph-volume using the default
> style (lvm) and not simple-mode.
>
> The disks were provisioned as a whole, no resizing. I never touched the disks
> after deployment.
>
> It is very strange that this first happened after the update, never met such
> an error before.
>
> I found a BUG in the tracker, that also shows such an error with count 0.
> That was closed with „can’t reproduce“ (don’t have the link ready). For me
> this seems like the data itself is fine and I just hit a bad transaction in
> the replay (which maybe caused the crash in the first place).
>
> I need one of three disks back. Object corruption would not be a problem
> (regarding drop of a journal), as this cluster hosts backups which will fail
> validation and regenerate. Just marking the OSD lost does not seem to be an
> option.
>
> Is there some sort of fsck for BlueFS?
>
> Kevin
>
>
> Igor Fedotov schrieb am Mi. 3. Okt. 2018 um 13:01:
>>
>> I've seen somewhat similar behavior in a log from Sergey Malinin in another
>> thread ("mimic: 3/4 OSDs crashed...")
>>
>> He claimed it happened after LVM volume expansion. Isn't this the case for
>> you?
>>
>> Am I right that you use LVM volumes?
>>
>>
>> On 10/3/2018 11:22 AM, Kevin Olbrich wrote:
>>
>> Small addition: the failing disks are in the same host.
>> This is a two-host, failure-domain OSD cluster.
>>
>>
>> Am Mi., 3. Okt. 2018 um 10:13 Uhr schrieb Kevin Olbrich :
>>>
>>> Hi!
>>>
>>> Yesterday one of our (non-priority) clusters failed when 3 OSDs went down
>>> (EC 8+2) together.
>>> This is strange as we did an upgrade from 13.2.1 to 13.2.2 one or two hours
>>> before.
>>> They failed exactly at the same moment, rendering the cluster unusable
>>> (CephFS).
>>> We are using CentOS 7 with latest updates and ceph repo. No cache SSDs, no
>>> external journal / wal / db.
>>>
>>> OSD 29 (no disk failure in dmesg):
>>> 2018-10-03 09:47:15.074 7fb8835ce1c0 0 set uid:gid to 167:167 (ceph:ceph)
>>> 2018-10-03 09:47:15.074 7fb8835ce1c0 0 ceph version 13.2.2
>>> (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable), process
>>> ceph-osd, pid 20899
>>> 2018-10-03 09:47:15.074 7fb8835ce1c0 0 pidfile_write: ignore empty
>>> --pid-file
>>> 2018-10-03 09:47:15.100 7fb8835ce1c0 0 load: jerasure load: lrc load: isa
>>> 2018-10-03 09:47:15.100 7fb8835ce1c0 1 bdev create path
>>> /var/lib/ceph/osd/ceph-29/block type kernel
>>> 2018-10-03 09:47:15.100 7fb8835ce1c0 1 bdev(0x561250a2
>>> /var/lib/ceph/osd/ceph-29/block) open path /var/lib/ceph/osd/ceph-29/block
>>> 2018-10-03 09:47:15.100 7fb8835ce1c0 1 bdev(0x561250a2
>>> /var/lib/ceph/osd/ceph-29/block) open size 1000198897664 (0xe8e080, 932
>>> GiB) block_size 4096 (4 KiB) rotational
>>> 2018-10-03 09:47:15.101 7fb8835ce1c0 1
>>> bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes kv_min_ratio 1 >
>>> kv_ratio 0.5
>>> 2018-10-03 09:47:15.101 7fb8835ce1c0 1
>>> bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes cache_size 536870912
>>> meta 0 kv 1 data 0
>>> 2018-10-03 09:47:15.101 7fb8835ce1c0 1 bdev(0x561250a2
>>> /var/lib/ceph/osd/ceph-29/block) close
>>> 2018-10-03 09:47:15.358 7fb8835ce1c0 1
>>> bluestore(/var/lib/ceph/osd/ceph-29) _mount path /var/lib/ceph/osd/ceph-29
>>> 2018-10-03 09:47:15.358 7fb8835ce1c0 1 bdev create path
>>> /var/lib/ceph/osd/ceph-29/block type kernel
>>> 2018-10-03 09:47:15.358 7fb8835ce1c0 1 bdev(0x561250a2
>>> /var/lib/ceph/osd/ceph-29/block) open path /var/lib/ceph/osd/ceph-29/block
>>> 2018-10-03 09:47:15.359 7fb8835ce1c0 1 bdev(0x561250a2
>>> /var/lib/ceph/osd/ceph-29/block) open size 1000198897664 (0xe8e080, 932
>>> GiB) block_size 4096 (4 KiB) rotational
>>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1
>>> bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes kv_min_ratio 1 >
>>> kv_ratio 0.5
>>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1
>>> bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes cache_size 536870912
>>> meta 0 kv 1 data 0
>>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bdev create path
>>> /var/lib/ceph/osd/ceph-29/block type kernel
>>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bdev(0x561250a20a80
>>> /var/lib/ceph/osd/ceph-29/block) open path /var/lib/ceph/osd/ceph-29/block
>>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bdev(0x561250a20a80
>>> /var/lib/ceph/osd/ceph-29/block) open size 1000198897664 (0xe8e080, 932
>>> GiB) block_size 4096 (4 KiB) rotational
>>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bluefs add_block_device bdev 1 path
>>> /var/lib/ceph/osd/ceph-29/block size 932 GiB
>>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bluefs mount
>>> 2018-10-03 09:47:15.538 7fb8835ce1c0 -1
Re: [ceph-users] After 13.2.2 upgrade: bluefs mount failed to replay log: (5) Input/output error
The disks were deployed with ceph-deploy / ceph-volume using the default
style (lvm) and not simple-mode.
The disks were provisioned as a whole, no resizing. I never touched the
disks after deployment.
It is very strange that this first happened after the update, never met
such an error before.
I found a BUG in the tracker, that also shows such an error with count 0.
That was closed with „can’t reproduce“ (don’t have the link ready). For me
this seems like the data itself is fine and I just hit a bad transaction in
the replay (which maybe caused the crash in the first place).
I need one of three disks back. Object corruption would not be a problem
(regarding drop of a journal), as this cluster hosts backups which will
fail validation and regenerate. Just marking the OSD lost does not seem to
be an option.
Is there some sort of fsck for BlueFS?
Kevin
Igor Fedotov schrieb am Mi. 3. Okt. 2018 um 13:01:
> I've seen somewhat similar behavior in a log from Sergey Malinin in
> another thread ("mimic: 3/4 OSDs crashed...")
>
> He claimed it happened after LVM volume expansion. Isn't this the case for
> you?
>
> Am I right that you use LVM volumes?
>
> On 10/3/2018 11:22 AM, Kevin Olbrich wrote:
>
> Small addition: the failing disks are in the same host.
> This is a two-host, failure-domain OSD cluster.
>
>
> Am Mi., 3. Okt. 2018 um 10:13 Uhr schrieb Kevin Olbrich :
>
>> Hi!
>>
>> Yesterday one of our (non-priority) clusters failed when 3 OSDs went down
>> (EC 8+2) together.
>> *This is strange as we did an upgrade from 13.2.1 to 13.2.2 one or two
>> hours before.*
>> They failed exactly at the same moment, rendering the cluster unusable
>> (CephFS).
>> We are using CentOS 7 with latest updates and ceph repo. No cache SSDs,
>> no external journal / wal / db.
>>
>> *OSD 29 (no disk failure in dmesg):*
>> 2018-10-03 09:47:15.074 7fb8835ce1c0 0 set uid:gid to 167:167 (ceph:ceph)
>> 2018-10-03 09:47:15.074 7fb8835ce1c0 0 ceph version 13.2.2
>> (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable), process
>> ceph-osd, pid 20899
>> 2018-10-03 09:47:15.074 7fb8835ce1c0 0 pidfile_write: ignore empty
>> --pid-file
>> 2018-10-03 09:47:15.100 7fb8835ce1c0 0 load: jerasure load: lrc load:
>> isa
>> 2018-10-03 09:47:15.100 7fb8835ce1c0 1 bdev create path
>> /var/lib/ceph/osd/ceph-29/block type kernel
>> 2018-10-03 09:47:15.100 7fb8835ce1c0 1 bdev(0x561250a2
>> /var/lib/ceph/osd/ceph-29/block) open path /var/lib/ceph/osd/ceph-29/block
>> 2018-10-03 09:47:15.100 7fb8835ce1c0 1 bdev(0x561250a2
>> /var/lib/ceph/osd/ceph-29/block) open size 1000198897664 (0xe8e080, 932
>> GiB) block_size 4096 (4 KiB) rotational
>> 2018-10-03 09:47:15.101 7fb8835ce1c0 1
>> bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes kv_min_ratio 1 >
>> kv_ratio 0.5
>> 2018-10-03 09:47:15.101 7fb8835ce1c0 1
>> bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes cache_size 536870912
>> meta 0 kv 1 data 0
>> 2018-10-03 09:47:15.101 7fb8835ce1c0 1 bdev(0x561250a2
>> /var/lib/ceph/osd/ceph-29/block) close
>> 2018-10-03 09:47:15.358 7fb8835ce1c0 1
>> bluestore(/var/lib/ceph/osd/ceph-29) _mount path /var/lib/ceph/osd/ceph-29
>> 2018-10-03 09:47:15.358 7fb8835ce1c0 1 bdev create path
>> /var/lib/ceph/osd/ceph-29/block type kernel
>> 2018-10-03 09:47:15.358 7fb8835ce1c0 1 bdev(0x561250a2
>> /var/lib/ceph/osd/ceph-29/block) open path /var/lib/ceph/osd/ceph-29/block
>> 2018-10-03 09:47:15.359 7fb8835ce1c0 1 bdev(0x561250a2
>> /var/lib/ceph/osd/ceph-29/block) open size 1000198897664 (0xe8e080, 932
>> GiB) block_size 4096 (4 KiB) rotational
>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1
>> bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes kv_min_ratio 1 >
>> kv_ratio 0.5
>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1
>> bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes cache_size 536870912
>> meta 0 kv 1 data 0
>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bdev create path
>> /var/lib/ceph/osd/ceph-29/block type kernel
>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bdev(0x561250a20a80
>> /var/lib/ceph/osd/ceph-29/block) open path /var/lib/ceph/osd/ceph-29/block
>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bdev(0x561250a20a80
>> /var/lib/ceph/osd/ceph-29/block) open size 1000198897664 (0xe8e080, 932
>> GiB) block_size 4096 (4 KiB) rotational
>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bluefs add_block_device bdev 1
>> path /var/lib/ceph/osd/ceph-29/block size 932 GiB
>> 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bluefs mount
>> 2018-10-03 09:47:15.538 7fb8835ce1c0 -1 bluefs _replay file with link
>> count 0: file(ino 519 size 0x31e2f42 mtime 2018-10-02 12:24:22.632397 bdev
>> 1 allocated 320 extents
>> [1:0x700820+10,1:0x700900+10,1:0x700910+10,1:0x700920+10,1:0x700930+10,1:0x700940+10,1:0x700950+10,1:0x700960+10,1:0x700970+10,1:0x700980+10,1:0x700990+10,1:0x7009a0+10,1:0x7009b0+10,1:0x7009c0+10,1:0x7009d000
Re: [ceph-users] After 13.2.2 upgrade: bluefs mount failed to replay log: (5) Input/output error
I've seen somewhat similar behavior in a log from Sergey Malinin in
another thread ("mimic: 3/4 OSDs crashed...")
He claimed it happened after LVM volume expansion. Isn't this the case
for you?
Am I right that you use LVM volumes?
On 10/3/2018 11:22 AM, Kevin Olbrich wrote:
Small addition: the failing disks are in the same host.
This is a two-host, failure-domain OSD cluster.
Am Mi., 3. Okt. 2018 um 10:13 Uhr schrieb Kevin Olbrich >:
Hi!
Yesterday one of our (non-priority) clusters failed when 3 OSDs
went down (EC 8+2) together.
*This is strange as we did an upgrade from 13.2.1 to 13.2.2 one or
two hours before.*
They failed exactly at the same moment, rendering the cluster
unusable (CephFS).
We are using CentOS 7 with latest updates and ceph repo. No cache
SSDs, no external journal / wal / db.
*OSD 29 (no disk failure in dmesg):*
2018-10-03 09:47:15.074 7fb8835ce1c0 0 set uid:gid to 167:167
(ceph:ceph)
2018-10-03 09:47:15.074 7fb8835ce1c0 0 ceph version 13.2.2
(02899bfda814146b021136e9d8e80eba494e1126) mimic (stable), process
ceph-osd, pid 20899
2018-10-03 09:47:15.074 7fb8835ce1c0 0 pidfile_write: ignore
empty --pid-file
2018-10-03 09:47:15.100 7fb8835ce1c0 0 load: jerasure load: lrc
load: isa
2018-10-03 09:47:15.100 7fb8835ce1c0 1 bdev create path
/var/lib/ceph/osd/ceph-29/block type kernel
2018-10-03 09:47:15.100 7fb8835ce1c0 1 bdev(0x561250a2
/var/lib/ceph/osd/ceph-29/block) open path
/var/lib/ceph/osd/ceph-29/block
2018-10-03 09:47:15.100 7fb8835ce1c0 1 bdev(0x561250a2
/var/lib/ceph/osd/ceph-29/block) open size 1000198897664
(0xe8e080, 932 GiB) block_size 4096 (4 KiB) rotational
2018-10-03 09:47:15.101 7fb8835ce1c0 1
bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes kv_min_ratio
1 > kv_ratio 0.5
2018-10-03 09:47:15.101 7fb8835ce1c0 1
bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes cache_size
536870912 meta 0 kv 1 data 0
2018-10-03 09:47:15.101 7fb8835ce1c0 1 bdev(0x561250a2
/var/lib/ceph/osd/ceph-29/block) close
2018-10-03 09:47:15.358 7fb8835ce1c0 1
bluestore(/var/lib/ceph/osd/ceph-29) _mount path
/var/lib/ceph/osd/ceph-29
2018-10-03 09:47:15.358 7fb8835ce1c0 1 bdev create path
/var/lib/ceph/osd/ceph-29/block type kernel
2018-10-03 09:47:15.358 7fb8835ce1c0 1 bdev(0x561250a2
/var/lib/ceph/osd/ceph-29/block) open path
/var/lib/ceph/osd/ceph-29/block
2018-10-03 09:47:15.359 7fb8835ce1c0 1 bdev(0x561250a2
/var/lib/ceph/osd/ceph-29/block) open size 1000198897664
(0xe8e080, 932 GiB) block_size 4096 (4 KiB) rotational
2018-10-03 09:47:15.360 7fb8835ce1c0 1
bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes kv_min_ratio
1 > kv_ratio 0.5
2018-10-03 09:47:15.360 7fb8835ce1c0 1
bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes cache_size
536870912 meta 0 kv 1 data 0
2018-10-03 09:47:15.360 7fb8835ce1c0 1 bdev create path
/var/lib/ceph/osd/ceph-29/block type kernel
2018-10-03 09:47:15.360 7fb8835ce1c0 1 bdev(0x561250a20a80
/var/lib/ceph/osd/ceph-29/block) open path
/var/lib/ceph/osd/ceph-29/block
2018-10-03 09:47:15.360 7fb8835ce1c0 1 bdev(0x561250a20a80
/var/lib/ceph/osd/ceph-29/block) open size 1000198897664
(0xe8e080, 932 GiB) block_size 4096 (4 KiB) rotational
2018-10-03 09:47:15.360 7fb8835ce1c0 1 bluefs add_block_device
bdev 1 path /var/lib/ceph/osd/ceph-29/block size 932 GiB
2018-10-03 09:47:15.360 7fb8835ce1c0 1 bluefs mount
2018-10-03 09:47:15.538 7fb8835ce1c0 -1 bluefs _replay file with
link count 0: file(ino 519 size 0x31e2f42 mtime 2018-10-02
12:24:22.632397 bdev 1 allocated 320 extents
[1:0x700820+10,1:0x700900+10,1:0x700910+10,1:0x700920+10,1:0x700930+10,1:0x700940+10,1:0x700950+10,1:0x700960+10,1:0x700970+10,1:0x700980+10,1:0x700990+10,1:0x7009a0+10,1:0x7009b0+10,1:0x7009c0+10,1:0x7009d0+10,1:0x7009e0+10,1:0x7009f0+10,1:0x700a00+10,1:0x700a10+10,1:0x700a20+10,1:0x700a30+10,1:0x700a40+10,1:0x700a50+10,1:0x700a60+10,1:0x700a70+10,1:0x700a80+10,1:0x700a90+10,1:0x700aa0+10,1:0x700ab0+10,1:0x700ac0+10,1:0x700ad0+10,1:0x700ae0+10,1:0x700af0+10,1:0x700b00+10,1:0x700b10+10,1:0x700b20+10,1:0x700b30+10,1:0x700b40+10,1:0x700b50+10,1:0x700b60+10,1:0x700b70+10,1:0x700b80+10,1:0x700b90+10,1:0x700ba0+10,1:0x700bb0+10,1:0x700bc0+10,1:0x700bd0+10,1:0x700be0+10,1:0x700bf0+10,1:0x700c00+10])
2018-10-03 09:47:15.538 7fb8835ce1c0 -1 bluefs mount fa
Re: [ceph-users] After 13.2.2 upgrade: bluefs mount failed to replay log: (5) Input/output error
Small addition: the failing disks are in the same host. This is a two-host, failure-domain OSD cluster. Am Mi., 3. Okt. 2018 um 10:13 Uhr schrieb Kevin Olbrich : > Hi! > > Yesterday one of our (non-priority) clusters failed when 3 OSDs went down > (EC 8+2) together. > *This is strange as we did an upgrade from 13.2.1 to 13.2.2 one or two > hours before.* > They failed exactly at the same moment, rendering the cluster unusable > (CephFS). > We are using CentOS 7 with latest updates and ceph repo. No cache SSDs, no > external journal / wal / db. > > *OSD 29 (no disk failure in dmesg):* > 2018-10-03 09:47:15.074 7fb8835ce1c0 0 set uid:gid to 167:167 (ceph:ceph) > 2018-10-03 09:47:15.074 7fb8835ce1c0 0 ceph version 13.2.2 > (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable), process > ceph-osd, pid 20899 > 2018-10-03 09:47:15.074 7fb8835ce1c0 0 pidfile_write: ignore empty > --pid-file > 2018-10-03 09:47:15.100 7fb8835ce1c0 0 load: jerasure load: lrc load: isa > 2018-10-03 09:47:15.100 7fb8835ce1c0 1 bdev create path > /var/lib/ceph/osd/ceph-29/block type kernel > 2018-10-03 09:47:15.100 7fb8835ce1c0 1 bdev(0x561250a2 > /var/lib/ceph/osd/ceph-29/block) open path /var/lib/ceph/osd/ceph-29/block > 2018-10-03 09:47:15.100 7fb8835ce1c0 1 bdev(0x561250a2 > /var/lib/ceph/osd/ceph-29/block) open size 1000198897664 (0xe8e080, 932 > GiB) block_size 4096 (4 KiB) rotational > 2018-10-03 09:47:15.101 7fb8835ce1c0 1 > bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes kv_min_ratio 1 > > kv_ratio 0.5 > 2018-10-03 09:47:15.101 7fb8835ce1c0 1 > bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes cache_size 536870912 > meta 0 kv 1 data 0 > 2018-10-03 09:47:15.101 7fb8835ce1c0 1 bdev(0x561250a2 > /var/lib/ceph/osd/ceph-29/block) close > 2018-10-03 09:47:15.358 7fb8835ce1c0 1 > bluestore(/var/lib/ceph/osd/ceph-29) _mount path /var/lib/ceph/osd/ceph-29 > 2018-10-03 09:47:15.358 7fb8835ce1c0 1 bdev create path > /var/lib/ceph/osd/ceph-29/block type kernel > 2018-10-03 09:47:15.358 7fb8835ce1c0 1 bdev(0x561250a2 > /var/lib/ceph/osd/ceph-29/block) open path /var/lib/ceph/osd/ceph-29/block > 2018-10-03 09:47:15.359 7fb8835ce1c0 1 bdev(0x561250a2 > /var/lib/ceph/osd/ceph-29/block) open size 1000198897664 (0xe8e080, 932 > GiB) block_size 4096 (4 KiB) rotational > 2018-10-03 09:47:15.360 7fb8835ce1c0 1 > bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes kv_min_ratio 1 > > kv_ratio 0.5 > 2018-10-03 09:47:15.360 7fb8835ce1c0 1 > bluestore(/var/lib/ceph/osd/ceph-29) _set_cache_sizes cache_size 536870912 > meta 0 kv 1 data 0 > 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bdev create path > /var/lib/ceph/osd/ceph-29/block type kernel > 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bdev(0x561250a20a80 > /var/lib/ceph/osd/ceph-29/block) open path /var/lib/ceph/osd/ceph-29/block > 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bdev(0x561250a20a80 > /var/lib/ceph/osd/ceph-29/block) open size 1000198897664 (0xe8e080, 932 > GiB) block_size 4096 (4 KiB) rotational > 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bluefs add_block_device bdev 1 > path /var/lib/ceph/osd/ceph-29/block size 932 GiB > 2018-10-03 09:47:15.360 7fb8835ce1c0 1 bluefs mount > 2018-10-03 09:47:15.538 7fb8835ce1c0 -1 bluefs _replay file with link > count 0: file(ino 519 size 0x31e2f42 mtime 2018-10-02 12:24:22.632397 bdev > 1 allocated 320 extents > [1:0x700820+10,1:0x700900+10,1:0x700910+10,1:0x700920+10,1:0x700930+10,1:0x700940+10,1:0x700950+10,1:0x700960+10,1:0x700970+10,1:0x700980+10,1:0x700990+10,1:0x7009a0+10,1:0x7009b0+10,1:0x7009c0+10,1:0x7009d0+10,1:0x7009e0+10,1:0x7009f0+10,1:0x700a00+10,1:0x700a10+10,1:0x700a20+10,1:0x700a30+10,1:0x700a40+10,1:0x700a50+10,1:0x700a60+10,1:0x700a70+10,1:0x700a80+10,1:0x700a90+10,1:0x700aa0+10,1:0x700ab0+10,1:0x700ac0+10,1:0x700ad0+10,1:0x700ae0+10,1:0x700af0+10,1:0x700b00+10,1:0x700b10+10,1:0x700b20+10,1:0x700b30+10,1:0x700b40+10,1:0x700b50+10,1:0x700b60+10,1:0x700b70+10,1:0x700b80+10,1:0x700b90+10,1:0x700ba0+10,1:0x700bb0+10,1:0x700bc0+10,1:0x700bd0+10,1:0x700be0+10,1:0x700bf0+10,1:0x700c00+10]) > 2018-10-03 09:47:15.538 7fb8835ce1c0 -1 bluefs mount failed to replay log: > (5) Input/output error > 2018-10-03 09:47:15.538 7fb8835ce1c0 1 stupidalloc 0x0x561250b8d030 > shutdown > 2018-10-03 09:47:15.538 7fb8835ce1c0 -1 > bluestore(/var/lib/ceph/osd/ceph-29) _open_db failed bluefs mount: (5) > Input/output error > 2018-10-03 09:47:15.538 7fb8835ce1c0 1 bdev(0x561250a20a80 > /var/lib/ceph/osd/ceph-29/block) close > 2018-10-03 09:47:15.616 7fb8835ce1c0 1 bdev(0x561250a2 > /var/lib/ceph/osd/ceph-2
