I'll keep trying to repro and gather diags, but running in containers is making
it very hard to run debug commands while the ceph daemons are down. Is this a
known problem with a solution?
In the meantime, what's the impact of running with the Bitmap Allocator instead
of the Hybrid one? I'm nervous about switching from the default without
understanding what that means.
Dave
-----Original Message-----
From: Igor Fedotov <ifedo...@suse.de>
Sent: 23 August 2021 14:22
To: Dave Piper <david.pi...@microsoft.com>; ceph-users@ceph.io
Subject: Re: [EXTERNAL] Re: [ceph-users] OSDs flapping with "_open_alloc loaded
132 GiB in 2930776 extents available 113 GiB"
Hi Dave,
so may be another bug in Hybid Allocator...
Could you please dump free extents for your "broken" osd(s) by issuing
"ceph-bluestore-tool --path <path-to-osd> --command free-dump". OSD to be
offline.
Preferably to have these reports after you reproduce the issue with hybrid
allocator once again - hence you'll need to switch back and wait till the
repro. But if that's inappropriate it would be OK to have such a dump for the
current state - hopefully it will reveal something interesting as well.
Thanks in advance,
Igor
On 8/20/2021 1:50 PM, Dave Piper wrote:
> Igor,
>
> We've hit this again on ceph 15.2.13 using the default allocator. Once again,
> configuring the OSDs to use the bitmap allocator has fixed up the issue.
>
> I'm still trying to gather the full set of debug logs from the crash. I think
> again the fact I'm running in containers is the issue here; the container
> seems to be dying before we've had time to flush the log stream to file. I'll
> keep looking for a way around this.
>
> Dave
>
> -----Original Message-----
> From: Igor Fedotov <ifedo...@suse.de>
> Sent: 12 August 2021 13:36
> To: Dave Piper <david.pi...@microsoft.com>; ceph-users@ceph.io
> Subject: Re: [EXTERNAL] Re: [ceph-users] OSDs flapping with "_open_alloc
> loaded 132 GiB in 2930776 extents available 113 GiB"
>
> Hi Dave,
>
> thanks for the update.
>
> I'm curious whether reverting back to default allocator on the latest release
> would be OK as well. Please try if possible.
>
>
> Thanks,
>
> Igor
>
> On 8/12/2021 2:00 PM, Dave Piper wrote:
>> Hi Igor,
>>
>> Just to update you on our progress.
>>
>> - We've not had another repro of this since switching to bitmap allocator /
>> upgrading to the latest octopus release. I'll try to gather the full set of
>> diags if we do see this again.
>> - I think my issues with an empty /var/lib/ceph/osd/ceph-N/ folder are
>> because we're running ceph in container which is using a mounted filesystem.
>> As soon as I stop the OSD, the container terminated and the filesystem
>> disappears. There's probably a way to decouple the ceph process from the
>> lifetime of the container, but I've not figured it out yet.
>>
>> Cheers again for all your help,
>>
>> Dave
>>
>> -----Original Message-----
>> From: Igor Fedotov <ifedo...@suse.de>
>> Sent: 26 July 2021 13:30
>> To: Dave Piper <david.pi...@microsoft.com>; ceph-users@ceph.io
>> Subject: Re: [EXTERNAL] Re: [ceph-users] OSDs flapping with "_open_alloc
>> loaded 132 GiB in 2930776 extents available 113 GiB"
>>
>> Dave,
>>
>> please see inline
>>
>> On 7/26/2021 1:57 PM, Dave Piper wrote:
>>> Hi Igor,
>>>
>>>> So to get more verbose but less log one can set both debug-bluestore and
>>>> debug-bluefs to 1/20. ...
>>> More verbose logging attached. I've trimmed the file to a single restart
>>> attempt to keep the filesize down; let me know if there's not enough here.
>> Jul 26 10:25:07 condor_sc0 docker[19100]: -9628>
>> 2021-07-26T10:25:05.512+0000 7f9b3ed48f40 20 bluefs _read got 32768
>> Jul 26 10:25:07 condor_sc0 docker[19100]: -9627>
>> 2021-07-26T10:25:05.512+0000 7f9b3ed48f40 10 bluefs _read h
>> 0x563e8bd3ff80 0xb2d0000~8000 from file(ino 316842 size 0xe6a476e
>> mtime
>> 2021-07-14T15:54:21.751044+0000 allocated e6b0000 extents
>> [1:0x1128740000~10000,1:0x1128780000~10000,1:0x112ad60000~10000,1:0x1
>> 165fa0000~10000,1:0x11662a0000~10000,1:0x1167410000~10000,1:0x116be10
>> 000~10000,1:0x112a330000~20000,1:0x1170890000~10000,1:0x11751e0000~10
>> 000,1:0x11770c0000~10000,1:0x1177760000~10000,1:0x1178050000~10000,1:
>> 0x117b2d0000~10000,1:0x117b580000~10000,1:0x12a9770000~10000,1:0x12a9
>> 820000~10000,1:0x12a98d0000~10000,1:0x12a98f0000~10000,1:0x12a9930000
>> ~10000,1:0x12a9a00000~10000,1:0x12a9b30000~10000,1:0x12a9bd0000~10000
>> ,1:0x12a9c30000~10000,1:0x12a9c70000~10000,1:0x12a9ca0000~10000,1:0x1
>> 2a9cc0000~10000,1:0x12a9df0000~10000,1:0x12a9e90000~10000,1:0x12a9ec0
>> 000~10000,1:0x12a9f30000~10000,1:0x12a9f90000~10000,1:0x12a9ff0000~10
>> 000,1:0x12aa020000~10000,1:0x12aa050000~10000,1:0x12aa090000~10000,1:
>> 0x12aa0b0000~10000,1:0x12aa150000~10000,1:0x12aa220000~10000,1:0x12aa
>> 370000~10000,1:0x12aa410000~10000,1:0x12aa440000~10000,1:0x12aa490000
>> ~10000,1:0x12aa4c0000~10000,1:0x12aa510000~10000,1:0x12aa5a0000~10000
>> ,1:0x12aa7b0000~10000,1:0x12aa910000~10000,1:0x12aa950000~10000,1:0x1
>> 2aa970000~10000,1:0x12aa9a0000~10000,1:0x12aa9d0000~10000,1:0x12aab60
>> 000~10000,1:0x12aac10000~10000,1:0x12ad420000~10000,1:0x12ad4d0000~10
>> 000,1:0x12ad7a0000~10000,1:0x12ad810000~10000,1:0x12e4450000~10000,1:
>> 0x12e7fe0000~10000,1:0x12ecc40000~10000,1:0x12f2510000~10000,1:0x12f2
>> bd0000~10000,1:0x12f3430000~10000,1:0x12f5210000~10000,1:0x132a050000
>> ~10000,1:0x133f8e0000~10000,1:0x13433d0000~10000,1:0x1344ef0000~10000
>> ,1:0x13478d0000~10000,1:0x134b7f0000~10000,1:0x134d450000~10000,1:0x1
>> 34dc60000~10000,1:0x134dc80000~10000,1:0x13511c0000~10000,1:0x1351230
>> 000~10000,1:0x1352af0000~10000,1:0x1355cc0000~10000,1:0x135ec80000~10
>> 000,1:0x136cfb0000~10000,1:0x13737a0000~10000,1:0x1374750000~10000,1:
>> 0x137fe70000~10000,1:0x1392590000~10000,1:0x1393a20000~10000,1:0x1396
>> 670000~10000,1:0x139d560000~10000,1:0x13a17d0000~10000,1:0x13a20e0000
>> ~10000,1:0x13a2ea0000~10000,1:0x13a5420000~10000,1:0x13a6d70000~10000
>> ,1:0x13abef0000~10000,1:0x13bc840000~10000,1:0x13caa00000~10000,1:0x1
>> 3cedf0000~10000,1:0x2a00540000~10000,1:0x2a02630000~10000,1:0x2a04100
>> 000~10000,1:0x2a08550000~10000,1:0x2a091e0000~10000,1:0x2a09410000~10
>> 000,1:0x2a17c70000~10000,1:0x2a1a860000~10000,1:0x2a1a930000~10000,1:
>> 0x2a1fa00000~10000,1:0x2a21bc0000~10000,1:0x2a24d20000~10000,1:0x2a2b
>> 5a0000~10000,1:0x2a2c9b0000~10000,1:0x2a36c30000~10000,1:0x2a37c40000
>> ~10000,1:0x2a39c60000~10000,1:0x2a43480000~10000,1:0x2a43b80000~10000
>> ,1:0x2a46250000~10000,1:0x2a4ba60000~10000,1:0x2a526b0000~10000,1:0x2
>> a55b00000~10000,1:0x2a59920000~10000,1:0x2a5fee0000~10000,1:0x2a686c0
>> 000~10000,1:0x2a76280000~10000,1:0x2a8b7d0000~10000,1:0x2a91f00000~10
>> 000,1:0x2a98930000~10000,1:0x2a9b4f0000~10000,1:0x2a9e720000~10000,1:
>> 0x2aa5920000~10000,1:0x2aac110000~10000,1:0x2ab57f0000~10000,1:0x2ac2
>> 690000~10000,1:0x2ac57e0000~10000,1:0x2ad6710000~10000,1:0x2ad9040000
>> ~10000,1:0x2afa900000~10000,1:0x2afb110000~10000,1:0x2afd070000~10000
>> ,1:0x2b15220000~10000,1:0x2b189b0000~10000,1:0x2b330f0000~10000,1:0x2
>> b33810000~10000,1:0x2b365a0000~10000,1:0x2b36aa0000~10000,1:0x2b36d60
>> 000~10000,1:0x2b397b0000~10000,1:0x2b3f510000~10000,1:0x2b42960000~10
>> 000,1:0x2b43e60000~10000,1:0x2b5e900000~10000,1:0x2b62060000~10000,1:
>> 0x2b63060000~10000,1:0x2b652f0000~10000,1:0x2b7bf60000~10000,1:0x2b87
>> 6d0000~10000,1:0x2b8a3f0000~10000,1:0x2b94810000~10000,1:0x2b951e0000
>> ~10000,1:0x2b957e0000~10000,1:0x2b98a80000~10000,1:0x2b99480000~10000
>> ,1:0x2b9b880000~10000,1:0x2ba1030000~10000,1:0x2ba2170000~10000,1:0x2
>> bb6670000~10000,1:0x2bd2870000~10000,1:0x2bd2a80000~10000,1:0x2bed7a0
>> 000~10000,1:0x2bf3800000~10000,1:0x2bf44f0000~10000,1:0x2bf8d40000~10
>> 000,1:0x2c129b0000~10000,1:0x2c142a0000~10000,1:0x2c171b0000~10000,1:
>> 0x2c1abd0000~10000,1:0x2c1d810000~10000,1:0x2c23ea0000~10000,1:0x2c25
>> 9b0000~10000,1:0x2c25f20000~10000,1:0x2c28330000~10000,1:0x2c320f0000
>> ~10000,1:0x2c48430000~10000,1:0x2c56500000~10000,1:0x2c60fa0000~10000
>> ,1:0x2c65a40000~10000,1:0x2c68f40000~10000,1:0x2c9c130000~10000,1:0x2
>> ca1350000~10000,1:0x2ca5880000~10000,1:0x2cd4380000~10000,1:0x2ce2b40
>> 000~10000,1:0x2cf0e00000~10000,1:0x2d005a0000~10000,1:0x2d0f700000~10
>> 000,1:0x2d0fc00000~10000,1:0x2d19ed0000~10000,1:0x2d3ddc0000~10000,1:
>> 0x2d3fbc0000~10000,1:0x2d43040000~10000,1:0x2d6de50000~10000,1:0x13bc
>> d60000~20000,1:0x13bf180000~20000,1:0x13c3ec0000~20000,1:0x13c5a00000
>> ~20000,1:0x13c6070000~20000,1:0x13ce790000~20000,1:0x13cfc70000~20000
>> ,1:0x13d0970000~20000,1:0x2a08d60000~20000,1:0x2a09170000~20000,1:0x2
>> a09b30000~20000,1:0x2a11950000~20000,1:0x2a17890000~20000,1:0x2a28550
>> 000~20000,1:0x2a34d20000~20000,1:0x2a57410000~20000,1:0x2a58c00000~20
>> 000,1:0x2a5c7c0000~20000,1:0x2a5cf40000~20000,1:0x2a663b0000~20000,1:
>> 0x2a68660000~20000,1:0x2a687e0000~20000,1:0x2a73d50000~20000,1:0x2a9f
>> ca0000~20000,1:0x2aa5300000~20000,1:0x2aaaaf0000~20000,1:0x2ab7700000
>> ~20000,1:0x2ac1d70000~20000,1:0x2ac6f30000~20000,1:0x2ad93a0000~20000
>> ,1:0x2ad9ad0000~20000,1:0x2ada2f0000~20000,1:0x2ae5ca0000~20000,1:0x2
>> ae82c0000~20000,1:0x2aebe10000~20000,1:0x13bcd20000~20000,1:0x2b0ff10
>> 000~20000,1:0x2b11530000~20000,1:0x2b18790000~20000,1:0x2b35c80000~20
>> 000,1:0x2b4d290000~20000,1:0x2b54d70000~20000,1:0x2b5cf40000~20000,1:
>> 0x2b5d990000~20000,1:0x2b5e3e0000~20000,1:0x2b5ee10000~20000,1:0x2b5f
>> 860000~20000,1:0x2b5fd80000~20000,1:0x2b8fc60000~20000,1:0x2b91fd0000
>> ~20000,1:0x2b934f0000~20000,1:0x2ba72b0000~20000,1:0x2baaef0000~20000
>> ,1:0x2bb7b80000~20000,1:0x2bbe290000~20000,1:0x2bc7460000~20000,1:0x2
>> bf7200000~20000,1:0x2c09340000~20000,1:0x2c217e0000~20000,1:0x2c231e0
>> 000~20000,1:0x2c45f30000~20000,1:0x2c4c4f0000~20000,1:0x2c4c810000~20
>> 000,1:0x2c4efa0000~20000,1:0x2c95750000~20000,1:0x2c99b30000~20000,1:
>> 0x2c9b130000~20000,1:0x2ca5360000~20000,1:0x2cb9540000~20000,1:0x2cb9
>> c40000~20000,1:0x2cd8e90000~20000,1:0x2d399b0000~20000,1:0x2b05b20000
>> ~20000,1:0x2d3d140000~20000,1:0x2d3dca0000~20000,1:0x11754f0000~30000
>> ,1:0x12e67e0000~30000,1:0x1304830000~30000,1:0x139eba0000~30000,1:0x1
>> 3a32e0000~30000,1:0x13b1680000~30000,1:0x13b3c00000~30000,1:0x13b95a0
>> 000~30000,1:0x13bd6d0000~30000,1:0x13c3790000~30000,1:0x2a0bea0000~30
>> 000,1:0x2a193b0000~30000,1:0x2a19830000~30000,1:0x2a1ab80000~30000,1:
>> 0x2a1b570000~30000,1:0x2a29740000~30000,1:0x2a2a170000~30000,1:0x2a2b
>> c00000~30000,1:0x2a2ed80000~30000,1:0x2a36800000~30000,1:0x2a38750000
>> ~30000,1:0x2a3e360000~10000,1:0x2d3a330000~20000,1:0x2a43150000~30000
>> ,1:0x2a456f0000~30000,1:0x2a54c40000~30000,1:0x2a63f60000~30000,1:0x2
>> a665d0000~30000,1:0x2a6b350000~30000,1:0x2a791a0000~30000,1:0x2a7a5d0
>> 000~30000,1:0x2a7f2e0000~30000,1:0x2a846c0000~30000,1:0x2a84df0000~30
>> 000,1:0x2aa91f0000~30000,1:0x2aaf450000~30000,1:0x2ab3130000~30000,1:
>> 0x2ab4ac0000~30000,1:0x2aca290000~30000,1:0x2addc80000~30000,1:0x2ae9
>> de0000~30000,1:0x2af3770000~30000,1:0x2b04740000~30000,1:0x2b06190000
>> ~30000,1:0x2b07620000~30000,1:0x2b09880000~30000,1:0x2b294d0000~10000
>> ,1:0x2a3e340000~20000,1:0x2b38870000~30000,1:0x2b5d460000~30000,1:0x2
>> b5deb0000~30000,1:0x2b5f330000~30000,1:0x2b602a0000~30000,1:0x2b64580
>> 000~30000,1:0x2b81730000~30000,1:0x2b8b710000~30000,1:0x2b9b990000~30
>> 000,1:0x2b9ed00000~30000,1:0x2ba1540000~30000,1:0x2ba5080000~30000,1:
>> 0x2bafe50000~30000,1:0x2bd3ab0000~10000,1:0x2b294e0000~20000,1:0x2bd5
>> 890000~30000,1:0x2bdc5d0000~30000,1:0x2bdc900000~30000,1:0x2bf19b0000
>> ~30000,1:0x2c08e10000~30000,1:0x2c25bc0000~30000,1:0x2c33c00000~30000
>> ,1:0x2c509d0000~30000,1:0x2c67770000~30000,1:0x2c6f5a0000~30000,1:0x2
>> c784e0000~30000,1:0x2c8c030000~10000,1:0x2bd3a90000~20000,1:0x2c8f2e0
>> 000~30000,1:0x2c97b30000~30000,1:0x2cbb770000~30000,1:0x2cc0290000~30
>> 000,1:0x2cc7110000~30000,1:0x2cd3ef0000~30000,1:0x2cde190000~30000,1:
>> 0x2cf6f10000~30000,1:0x2d3ce10000~30000,1:0x2d63960000~30000,1:0x1125
>> ef0000~40000,1:0x112ac00000~40000,1:0x112ac80000~40000,1:0x1162280000
>> ~40000,1:0x11686e0000~20000,1:0x2c8c010000~20000,1:0x116adf0000~40000
>> ,1:0x116dd40000~40000,1:0x1170380000~40000,1:0x1175ac0000~40000,1:0x1
>> 176e80000~40000,1:0x11777c0000~40000,1:0x1177820000~40000,1:0x1178270
>> 000~40000,1:0x11787f0000~40000,1:0x11686c0000~20000,1:0x117fc30000~20
>> 000,1:0x11807e0000~40000,1:0x1183a90000~40000,1:0x11845d0000~40000,1:
>> 0x118aad0000~40000,1:0x118d230000~40000,1:0x118d4a0000~40000,1:0x1196
>> d80000~40000,1:0x12a84c0000~40000,1:0x12ae7c0000~20000,1:0x117fc10000
>> ~20000,1:0x12b01a0000~40000,1:0x12b2710000~40000,1:0x12c70b0000~40000
>> ,1:0x12cd280000~40000,1:0x12cee60000~40000,1:0x12d4690000~40000,1:0x1
>> 2d8a90000~40000,1:0x12dd5e0000~40000,1:0x12df360000~40000,1:0x12e5e70
>> 000~10000,1:0x12e5ea0000~10000,1:0x12ae7e0000~20000,1:0x12e6b10000~40
>> 000,1:0x12e9730000~40000,1:0x12e9870000~40000,1:0x12ea390000~40000,1:
>> 0x12eae80000~40000,1:0x12ec6b0000~40000,1:0x1312b90000~40000,1:0x1321
>> 7e0000~40000,1:0x1327060000~40000,1:0x12e5e80000~20000,1:0x1333780000
>> ~20000,1:0x1338b00000~40000,1:0x1338ef0000~40000,1:0x1339290000~40000
>> ,1:0x133e320000~40000,1:0x133ebe0000~40000,1:0x1341650000~40000,1:0x1
>> 3467e0000~40000,1:0x13469e0000~40000,1:0x1333760000~20000,1:0x1347f10
>> 000~20000,1:0x1348c70000~40000,1:0x134c250000~40000,1:0x13502b0000~40
>> 000,1:0x13685c0000~40000,1:0x136b9f0000~40000,1:0x136cb60000~40000,1:
>> 0x136ce00000~40000,1:0x13731d0000~40000,1:0x13796c0000~20000,1:0x1347
>> ef0000~20000,1:0x137b700000~40000,1:0x1384a40000~40000,1:0x138ee10000
>> ~40000,1:0x1399730000~40000,1:0x13b8130000~40000,1:0x13bb250000~40000
>> ,1:0x13c3ad0000~40000,1:0x13c5c20000~40000,1:0x13c7320000~10000,1:0x1
>> 3c7350000~10000,1:0x13796e0000~20000,1:0x13c74e0000~40000,1:0x13c8180
>> 000~40000,1:0x13cb700000~40000,1:0x13cbc40000~40000,1:0x2a06450000~40
>> 000,1:0x2a0d440000~40000,1:0x2a17450000~40000,1:0x2a1c4d0000~40000,1:
>> 0x2a300b0000~10000,1:0x2a300e0000~10000,1:0x13c7330000~20000,1:0x2a37
>> f50000~40000,1:0x2a39320000~40000,1:0x2a3e690000~40000,1:0x2a47d50000
>> ~40000,1:0x2a4d5b0000~40000,1:0x2a555d0000~40000,1:0x2a77a60000~40000
>> ,1:0x2a7de60000~40000,1:0x2a300c0000~20000,1:0x2a888a0000~20000,1:0x2
>> a8a670000~40000,1:0x2a97750000~40000,1:0x2a9cec0000~40000,1:0x2aa8970
>> 000~40000,1:0x2aa8bb0000~40000,1:0x2ac61e0000~40000,1:0x2ad5cc0000~40
>> 000,1:0x2adc150000~40000,1:0x2a88880000~20000,1:0x2afd290000~20000,1:
>> 0x2b06880000~40000,1:0x2b09db0000~40000,1:0x2b16790000~40000,1:0x2b26
>> a90000~40000,1:0x2b2bf90000~40000,1:0x2b2f9d0000~40000,1:0x2b48570000
>> ~40000,1:0x2b4f890000~40000,1:0x2b5ca00000~10000,1:0x2b5ca30000~10000
>> ,1:0x2afd270000~20000,1:0x2b604d0000~40000,1:0x2b61620000~40000,1:0x2
>> b69e50000~40000,1:0x2b6fb00000~40000,1:0x2b7a820000~40000,1:0x2b91080
>> 000~40000,1:0x2b964f0000~40000,1:0x2b97f70000~40000,1:0x2b98f80000~20
>> 000,1:0x2b5ca10000~20000,1:0x2bb3590000~40000,1:0x2bcdca0000~40000,1:
>> 0x2bda690000~40000,1:0x2be93c0000~40000,1:0x2bfd6b0000~40000,1:0x2c35
>> b40000~40000,1:0x2c49df0000~40000,1:0x2c5f560000~40000,1:0x2c9d240000
>> ~20000,1:0x2b98fa0000~20000,1:0x2cc4a50000~40000,1:0x2ccb1c0000~40000
>> ,1:0x2cda090000~40000,1:0x2ced0b0000~40000,1:0x2d3d260000~40000,1:0x2
>> d43a40000~40000,1:0x2d7f160000~40000,1:0x2d7fdb0000~40000,1:0x11259b0
>> 000~50000,1:0x112a9b0000~30000,1:0x2c9d260000~20000,1:0x112aab0000~50
>> 000,1:0x116a9a0000~50000,1:0x1174ed0000~50000,1:0x1178990000~50000,1:
>> 0x117f3a0000~50000,1:0x1183c60000~50000,1:0x118a020000~50000,1:0x118b
>> 320000~50000,1:0x112a9e0000~20000,1:0x118c9f0000~30000,1:0x12964f0000
>> ~50000,1:0x12acdd0000~50000,1:0x12b16b0000~50000,1:0x12d8cd0000~50000
>> ,1:0x12e78d0000~50000,1:0x12ee0c0000~50000,1:0x12ff530000~50000,1:0x1
>> 2ff780000~10000,1:0x118c9d0000~20000,1:0x12ff7b0000~20000,1:0x1306460
>> 000~50000,1:0x13073b0000~50000,1:0x130d8e0000~50000,1:0x1315b40000~50
>> 000,1:0x131da80000~50000,1:0x1324a10000~50000,1:0x1325360000~20000,1:
>> 0x13253a0000~10000,1:0x12ff790000~20000,1:0x1331830000~50000,1:0x133d
>> 7f0000~50000,1:0x1343ca0000~50000,1:0x1350a80000~50000,1:0x13570a0000
>> ~50000,1:0x1358080000~50000,1:0x135ae50000~50000,1:0x135bb30000~20000
>> ,1:0x135bb70000~10000,1:0x1325380000~20000,1:0x1360ca0000~50000,1:0x1
>> 368510000~50000,1:0x1368f00000~50000,1:0x136b5a0000~50000,1:0x1377320
>> 000~50000,1:0x1377470000~50000,1:0x1377640000~50000,1:0x135bb50000~20
>> 000,1:0x137b010000~30000,1:0x13896b0000~50000,1:0x13981a0000~50000,1:
>> 0x13a0980000~50000,1:0x13bc9d0000~50000,1:0x13be130000~50000,1:0x13c0
>> be0000~50000,1:0x137aff0000~20000,1:0x13c3160000~30000,1:0x13c3c10000
>> ~50000,1:0x13c49a0000~50000,1:0x13c6280000~50000,1:0x13c6f90000~50000
>> ,1:0x13cc5f0000~50000,1:0x13cf5e0000~50000,1:0x2a038b0000~50000,1:0x1
>> 3c3140000~20000,1:0x2a065c0000~30000,1:0x2a140b0000~50000,1:0x2a14e00
>> 000~50000,1:0x2a17d80000~50000,1:0x2a1ec10000~50000,1:0x2a318d0000~50
>> 000,1:0x2a38ae0000~50000,1:0x2a38d50000~30000,1:0x2a065a0000~20000,1:
>> 0x2a40ab0000~50000,1:0x2a43dd0000~50000,1:0x2a4c060000~50000,1:0x2a4e
>> e30000~50000,1:0x2a597d0000~50000,1:0x2a655f0000~50000,1:0x2a67700000
>> ~50000,1:0x2a38d80000~20000,1:0x2a7c9f0000~30000,1:0x2a84070000~50000
>> ,1:0x2ab4760000~50000,1:0x2abed50000~50000,1:0x2ac7d50000~50000,1:0x2
>> ac8560000~50000,1:0x2acaf20000~50000,1:0x2ad42f0000~20000,1:0x2ad4330
>> 000~10000,1:0x2a7c9d0000~20000,1:0x2af58b0000~50000,1:0x2afb420000~50
>> 000,1:0x2affa60000~50000,1:0x2b03920000~50000,1:0x2b05d40000~50000,1:
>> 0x2b06fc0000~50000,1:0x2b12ce0000~50000,1:0x2b17860000~20000,1:0x2b17
>> 8a0000~10000,1:0x2ad4310000~20000,1:0x2b4fd10000~50000,1:0x2b97520000
>> ~50000,1:0x2b9e3b0000~50000,1:0x2be5ec0000~50000,1:0x2c11280000~50000
>> ,1:0x2c29b60000~50000,1:0x2c2a7b0000~50000,1:0x2c2de50000~30000,1:0x2
>> b17880000~20000,1:0x2c6ba50000~50000,1:0x2c9d680000~50000,1:0x11622d0
>> 000~60000,1:0x1166c20000~60000,1:0x1171050000~60000,1:0x1172020000~60
>> 000,1:0x11771d0000~30000,1:0x1177220000~10000,1:0x2c2de80000~20000,1:
>> 0x1177870000~60000,1:0x117abd0000~60000,1:0x117b840000~60000,1:0x117f
>> f90000~60000,1:0x1183a20000~60000,1:0x118b110000~40000,1:0x1177200000
>> ~20000,1:0x118b180000~60000,1:0x118e1d0000~60000,1:0x128cdf0000~60000
>> ,1:0x128ce60000~60000,1:0x12a82a0000~10000,1:0x118b150000~20000,1:0x1
>> 2a82d0000~30000,1:0x12b39e0000~60000,1:0x12c03a0000~60000,1:0x12c9310
>> 000~60000,1:0x12cbc50000~60000,1:0x12a82b0000~20000,1:0x12da380000~40
>> 000,1:0x12e7490000~60000,1:0x12ec7c0000~60000,1:0x12ede10000~60000,1:
>> 0x12ee660000~40000,1:0x12da360000~20000,1:0x12ef4c0000~60000,1:0x12f1
>> 490000~60000,1:0x130e480000~60000,1:0x130f3a0000~60000,1:0x1311430000
>> ~10000,1:0x12ee6a0000~20000,1:0x1311460000~30000,1:0x1311a90000~60000
>> ,1:0x1311d00000~60000,1:0x1312260000~60000,1:0x131e4f0000~40000,1:0x1
>> 311440000~20000,1:0x131f360000~60000,1:0x1322ab0000~60000,1:0x1347d50
>> 000~60000,1:0x134ef60000~60000,1:0x135b0f0000~20000,1:0x131e530000~20
>> 000,1:0x135b130000~20000,1:0x1369950000~60000,1:0x136a480000~60000,1:
>> 0x137e590000~60000,1:0x137eb30000~40000,1:0x135b110000~20000,1:0x1382
>> d30000~60000,1:0x138e070000~60000,1:0x1395e40000~60000,1:0x1397330000
>> ~60000,1:0x13a0cd0000~10000,1:0x137eb70000~20000,1:0x13a0d00000~30000
>> ,1:0x13c9ff0000~60000,1:0x13cc9f0000~60000,1:0x2a08090000~60000,1:0x2
>> a252c0000~30000,1:0x2a25310000~10000,1:0x13a0ce0000~20000,1:0x2a33bc0
>> 000~60000,1:0x2a38250000~60000,1:0x2a41100000~60000,1:0x2a436e0000~60
>> 000,1:0x2a252f0000~20000,1:0x2a79bb0000~40000,1:0x2a80a70000~60000,1:
>> 0x2a8aac0000~60000,1:0x2a9d9c0000~60000,1:0x2ab09f0000~30000,1:0x2ab0
>> a40000~10000,1:0x2a79b90000~20000,1:0x2abd150000~60000,1:0x2ac0950000
>> ~60000,1:0x2ac10f0000~60000,1:0x2ada7e0000~60000,1:0x2ab0a20000~20000
>> ,1:0x2af8660000~40000,1:0x2b02800000~60000,1:0x2b02960000~60000,1:0x2
>> b031c0000~60000,1:0x2af8640000~20000,1:0x2b15350000~40000,1:0x2b344c0
>> 000~60000,1:0x2b35520000~60000,1:0x2b3a5d0000~60000,1:0x2b97710000~20
>> 000,1:0x2b15330000~20000,1:0x2b97750000~20000,1:0x2bc22f0000~60000,1:
>> 0x2bdf940000~60000,1:0x2bea590000~60000,1:0x2bf59f0000~40000,1:0x2b97
>> 730000~20000,1:0x2bf9450000~60000,1:0x2c00080000~60000,1:0x2c75870000
>> ~60000,1:0x2c93de0000~60000,1:0x2bf5a30000~20000,1:0x2c9dc00000~40000
>> ,1:0x2cc47f0000~60000,1:0x2cd4220000~60000,1:0x2cec450000~60000,1:0x2
>> d3a0d0000~20000,1:0x2d3a110000~20000,1:0x2c9dbe0000~20000,1:0x2d3c940
>> 000~60000,1:0x11663b0000~70000,1:0x1169f00000~70000,1:0x116a370000~30
>> 000,1:0x116a3c0000~20000,1:0x2d3a0f0000~20000,1:0x116d510000~70000,1:
>> 0x116f290000~70000,1:0x117f2e0000~70000,1:0x1182780000~20000,1:0x116a
>> 3a0000~20000,1:0x11827c0000~30000,1:0x1188270000~70000,1:0x1189c90000
>> ~70000,1:0x118ac20000~70000,1:0x118d0b0000~20000,1:0x11827a0000~20000
>> ,1:0x118d0f0000~30000,1:0x118d6c0000~70000,1:0x118e440000~70000,1:0x1
>> 28ccd0000~70000,1:0x128ced0000~20000,1:0x118d0d0000~20000,1:0x128cf10
>> 000~30000,1:0x128dbb0000~70000,1:0x1297320000~70000,1:0x12a5500000~70
>> 000,1:0x12b9b40000~20000,1:0x128cef0000~20000,1:0x12b9b80000~30000,1:
>> 0x12e03b0000~70000,1:0x12e5d00000~70000,1:0x12e6670000~70000,1:0x12f2
>> 9c0000~10000,1:0x12b9b60000~20000,1:0x12f29f0000~40000,1:0x1300220000
>> ~70000,1:0x1300480000~70000,1:0x131c0c0000~70000,1:0x131c230000~10000
>> ,1:0x12f29d0000~20000,1:0x131c260000~40000,1:0x1323350000~70000,1:0x1
>> 32afe0000~70000,1:0x132c340000~70000,1:0x131c240000~20000,1:0x1331ba0
>> 000~50000,1:0x1331df0000~70000,1:0x1335830000~70000,1:0x1336e50000~50
>> 000,1:0x1331b80000~20000,1:0x1340220000~70000,1:0x134da70000~70000,1:
>> 0x1350580000~70000,1:0x1353cb0000~40000,1:0x1353d10000~10000,1:0x1336
>> ea0000~20000,1:0x1360630000~70000,1:0x13648d0000~70000,1:0x1369190000
>> ~70000,1:0x1371770000~50000,1:0x1353cf0000~20000,1:0x1373c10000~70000
>> ,1:0x137fdf0000~70000,1:0x1385a80000~70000,1:0x138a460000~30000,1:0x1
>> 3717c0000~20000,1:0x138a4b0000~20000,1:0x138ad60000~70000,1:0x138ef70
>> 000~70000,1:0x13986f0000~70000,1:0x139e190000~70000,1:0x13afbc0000~40
>> 000,1:0x13afc20000~10000,1:0x138a490000~20000,1:0x13bc490000~70000,1:
>> 0x13bd080000~70000,1:0x13c3e10000~70000,1:0x13cb8d0000~70000,1:0x13cc
>> 340000~70000,1:0x13afc00000~20000,1:0x29ffe30000~50000,1:0x2a09670000
>> ~70000,1:0x2a10b00000~70000,1:0x2a20210000~70000,1:0x2a32650000~70000
>> ,1:0x2a3a230000~30000,1:0x29ffe10000~20000,1:0x2a3a280000~20000,1:0x2
>> a4bdf0000~70000,1:0x2a54e90000~70000,1:0x2a56aa0000~70000,1:0x2a79cf0
>> 000~70000,1:0x2a7ac00000~50000,1:0x2a3a260000~20000,1:0x2a7c060000~70
>> 000,1:0x2a85e10000~70000,1:0x2a93f70000~70000,1:0x2aa12c0000~70000,1:
>> 0x2ac3160000~70000,1:0x2a7ac50000~20000,1:0x2ad15d0000~50000,1:0x2b01
>> 8b0000~70000,1:0x2b0bf80000~70000,1:0x2b419f0000~70000,1:0x2b5c490000
>> ~70000,1:0x2c04d10000~40000,1:0x2c04d70000~10000,1:0x2ad15b0000~20000
>> ,1:0x2c0d910000~70000,1:0x2c0fa80000~70000,1:0x2c150c0000~70000,1:0x2
>> c192b0000~70000,1:0x2c383a0000~70000,1:0x2c4eb30000~10000,1:0x2c04d50
>> 000~20000,1:0x2c4eb60000~40000,1:0x2cc7340000~70000,1:0x2ce1dd0000~70
>> 000,1:0x2d00bb0000~70000,1:0x2d083c0000~70000,1:0x2d414d0000~50000,1:
>> 0x2c4eb40000~20000,1:0x1125570000~80000,1:0x1125de0000~80000,1:0x112a
>> cd0000~80000,1:0x116c310000~80000,1:0x11708b0000~30000,1:0x2d41520000
>> ~20000,1:0x1170900000~30000,1:0x1172cb0000~80000,1:0x1175720000~80000
>> ,1:0x11792f0000~80000,1:0x117baa0000~80000,1:0x117e810000~30000,1:0x1
>> 1708e0000~20000,1:0x117e860000~30000,1:0x117ef60000~80000,1:0x1183350
>> 000~80000,1:0x1185d90000~80000,1:0x1188c80000~80000,1:0x117e840000~20
>> 000,1:0x118d760000~60000,1:0x118d890000~80000,1:0x128d8b0000~80000,1:
>> 0x12981c0000~80000,1:0x12a7ed0000~60000,1:0x118d740000~20000,1:0x12ad
>> 1b0000~80000,1:0x12ad860000~80000,1:0x12cb1a0000~80000,1:0x12cece0000
>> ~80000,1:0x12da0e0000~30000,1:0x12a7f30000~20000,1:0x12da130000~30000
>> ,1:0x12e6c50000~80000,1:0x12eac00000~80000,1:0x12eb0c0000~80000,1:0x1
>> 2eb840000~80000,1:0x12f1c90000~20000,1:0x12da110000~20000,1:0x12f1cd0
>> 000~40000,1:0x12ffea0000~80000,1:0x130ce30000~80000,1:0x130e6a0000~80
>> 000,1:0x130fa60000~80000,1:0x12f1cb0000~20000,1:0x13237a0000~60000,1:
>> 0x132ac60000~80000,1:0x132b230000~80000,1:0x132c0c0000~80000,1:0x1338
>> 7e0000~80000,1:0x1323780000~20000,1:0x133c6a0000~60000,1:0x13414d0000
>> ~80000,1:0x13485b0000~80000,1:0x134eaa0000~80000,1:0x1353ab0000~60000
>> ,1:0x133c680000~20000,1:0x13560a0000~80000,1:0x1366b40000~80000,1:0x1
>> 367960000~80000,1:0x13683d0000~80000,1:0x1370260000~50000,1:0x13702d0
>> 000~10000,1:0x1353b10000~20000,1:0x13743d0000~80000,1:0x1383b20000~80
>> 000,1:0x138a9b0000~80000,1:0x138c6e0000~80000,1:0x1390ee0000~30000,1:
>> 0x13702b0000~20000,1:0x1390f30000~30000,1:0x13928b0000~80000,1:0x1398
>> d80000~80000,1:0x139f970000~80000,1:0x13b1ab0000~80000,1:0x1390f10000
>> ~20000,1:0x13ba5f0000~60000,1:0x13ca490000~80000,1:0x13d10f0000~80000
>> ,1:0x2a05d50000~80000,1:0x2a075d0000~60000,1:0x13ba5d0000~20000,1:0x2
>> a224d0000~80000,1:0x2a2e600000~80000,1:0x2a390a0000~80000,1:0x2a481f0
>> 000~80000,1:0x2a4bc70000~30000,1:0x2a07630000~20000,1:0x2a4bcc0000~30
>> 000,1:0x2a5a6c0000~80000,1:0x2a5d160000~80000,1:0x2a75700000~80000,1:
>> 0x2a7dce0000~80000,1:0x2a86e90000~20000,1:0x2a4bca0000~20000,1:0x2a86
>> ed0000~40000,1:0x2a8d920000~80000,1:0x2a98f40000~80000,1:0x2abae10000
>> ~60000,1:0x2a86eb0000~20000,1:0x2ad2320000~80000,1:0x2b00d30000~80000
>> ,1:0x2b07c10000~80000,1:0x2b66200000~40000,1:0x2abae70000~20000,1:0x2
>> b66260000~20000,1:0x2b6ba90000~80000,1:0x2b7d880000~80000,1:0x2be9700
>> 000~80000,1:0x2c15230000~10000,1:0x2b66240000~20000,1:0x2c15260000~50
>> 000,1:0x2c2f160000~80000,1:0x2cad5a0000~80000,1:0x2ce3050000~50000,1:
>> 0x2ce30c0000~10000,1:0x2c15240000~20000,1:0x2ce3cd0000~80000,1:0x2d0f
>> 980000~80000,1:0x2d59e90000~80000,1:0x2d5a210000~10000,1:0x2ce30a0000
>> ~20000,1:0x2d5a240000~50000,1:0x11279a0000~90000,1:0x1171430000~90000
>> ,1:0x1175f00000~30000,1:0x2d5a220000~20000,1:0x1175f50000~40000,1:0x1
>> 177b00000~90000,1:0x11781d0000~90000,1:0x11782f0000~70000,1:0x1175f30
>> 000~20000,1:0x117b3e0000~90000,1:0x11803d0000~90000,1:0x1180650000~90
>> 000,1:0x1178360000~20000,1:0x1187320000~70000,1:0x11884f0000~90000,1:
>> 0x118c930000~90000,1:0x118d610000~10000,1:0x1187300000~20000,1:0x118d
>> 640000~60000,1:0x118d7f0000~90000,1:0x1284d20000~90000,1:0x128cd50000
>> ~20000,1:0x118d620000~20000,1:0x128cd90000~50000,1:0x1297030000~90000
>> ,1:0x12aec10000~90000,1:0x12b2f50000~40000,1:0x128cd70000~20000,1:0x1
>> 2b2fb0000~30000,1:0x12b4530000~90000,1:0x12ba3b0000~90000,1:0x12d58e0
>> 000~60000,1:0x12d5960000~10000,1:0x12b2f90000~20000,1:0x12dd820000~80
>> 000]) Jul 26 10:25:07 condor_sc0 docker[19100]: -9626>
>> 2021-07-26T10:25:05.512+0000 7f9b3ed48f40 20 bluefs _read fetching
>> 0x0~80000 of 1:0x1185d90000~80000
>> Jul 26 10:25:07 condor_sc0 docker[19100]: -9625>
>> 2021-07-26T10:25:05.513+0000 7f9b3ed48f40 20 bluefs _read left
>> 0x80000 len 0x8000
>>
>> ^^^ I expected this stuff to count up to -1 where the most interesting stuff
>> should be...
>>
>>
>> Jul 26 10:25:09 condor_sc0 docker[19550]: Error response from daemon:
>> No such container: ceph-osd-1
>>
>> Jul 26 10:25:19 condor_sc0 systemd[1]: ceph-osd@1.service holdoff
>> time over, scheduling restart.
>> Jul 26 10:25:19 condor_sc0 systemd[1]: Stopped Ceph OSD.
>> Jul 26 10:25:19 condor_sc0 systemd[1]: Starting Ceph OSD...
>>
>>>> It would be also great to collect the output for the following commands:
>>>> ...
>>> I've tried running ceph-bluestore-tool previously on this system but both
>>> commands fails with the following error:
>>>
>>> [qs-admin@condor_sc0 ~]$ sudo docker exec 419d997e5a05
>>> ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-1 --command
>>> bluefs-stats error from cold_open: (11) Resource temporarily
>>> unavailable
>>> 2021-07-26T10:50:15.032+0000 7f9a9bf68240 -1
>>> bluestore(/var/lib/ceph/osd/ceph-1) _lock_fsid failed to lock
>>> /var/lib/ceph/osd/ceph-1/fsid (is another ceph-osd still
>>> running?)(11) Resource temporarily unavailable
>>> [qs-admin@condor_sc0 ~]$
>>>
>>> There's only one OSD running on this server; should I be stopping it / the
>>> other OSDs in the cluster before running the `ceph-bluestore-tool` command?
>>> Previously when the OSDs were failing to start, /var/lib/ceph/osd/ceph-1/
>>> was empty but it now contains the following:
>> Each folder under /var/lib/ceph/osd/ceph-N/ should be used
>> exclusively by a single OSD. ceph-bluestore-tool to be run when the
>> respective OSD is shutdown, other OSDs referring to different folders
>> shouldn't matter.
>>
>> Is the above valid for your case?
>>
>>> [qs-admin@condor_sc0 ~]$ sudo docker exec 419d997e5a05 ls
>>> /var/lib/ceph/osd/ceph-1 block ceph_fsid fsid keyring ready
>>> require_osd_release type whoami
>>> [qs-admin@condor_sc0 ~]$
>>>
>>>
>>>> And finally you can try to switch to bitmap allocator as a workaround ...
>>> Switching to the bitmap allocator as you suggested has led to both failing
>>> OSDs starting up successfully. I've now got 3/3 OSDs up and in! The
>>> cluster still has MDS issues that were blocked behind getting the OSDs
>>> running as mentioned in my original post, but I think these are unrelated
>>> to the OSD problem as it's an issue we've seen in isolation elsewhere.
>>>
>>> So - that's a big step forward! Should I retry with my original config on
>>> the latest octopus release and see if this is now fixed?
>> yeah, looks like this is a hybrid allocator bug - hence you can
>> upgrade and check if this is fixed.
>>
>>
>>> Cheers again,
>>>
>>> Dave
>>>
>>>
>>> -----Original Message-----
>>> From: Igor Fedotov <ifedo...@suse.de>
>>> Sent: 26 July 2021 11:14
>>> To: Dave Piper <david.pi...@microsoft.com>; ceph-users@ceph.io
>>> Subject: Re: [EXTERNAL] Re: [ceph-users] OSDs flapping with "_open_alloc
>>> loaded 132 GiB in 2930776 extents available 113 GiB"
>>>
>>> Hi Dave,
>>>
>>> Some notes first:
>>>
>>> 1) The following behavior is fine, BlueStore mounts in two stages - the
>>> first one is read-only and among other things it loads allocation map from
>>> DB. And that's exactly the case here.
>>>
>>> Jul 26 08:55:31 condor_sc0 docker[15282]:
>>> 2021-07-26T08:55:31.703+0000
>>> 7f0e15b3df40 1 bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc
>>> loaded
>>> 132 GiB in 2930776 extents available 113 GiB Jul 26 08:55:31
>>> condor_sc0 docker[15282]: 2021-07-26T08:55:31.703+0000
>>> 7f0e15b3df40 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all
>>> background work Jul 26 08:55:31 condor_sc0 docker[15282]:
>>> 2021-07-26T08:55:31.704+0000
>>> 7f0e15b3df40 4 rocksdb: [db/db_impl.cc:563] Shutdown complete
>>>
>>> 2) What's really broken is the following allocation attempt:
>>>
>>> Jul 26 08:55:34 condor_sc0 docker[15282]:
>>> 2021-07-26T08:55:34.767+0000
>>> 7f0e15b3df40 1 bluefs _allocate failed to allocate 0x100716 on bdev
>>> 1, free 0xd0000; fallback to bdev 2 Jul 26 08:55:34 condor_sc0
>>> docker[15282]: 2021-07-26T08:55:34.767+0000
>>> 7f0e15b3df40 1 bluefs _allocate unable to allocate 0x100716 on bdev
>>> 2, free 0xffffffffffffffff; fallback to slow device expander Jul 26
>>> 08:55:35 condor_sc0 docker[15282]: 2021-07-26T08:55:35.042+0000
>>> 7f0e15b3df40 -1 bluestore(/var/lib/ceph/osd/ceph-1)
>>> allocate_bluefs_freespace failed to allocate on 0x40000000 min_size
>>> 0x110000 > allocated total 0x0 bluefs_shared_alloc_size 0x10000
>>> allocated 0x0 available 0x 1c09738000 Jul 26 08:55:35 condor_sc0
>>> docker[15282]: 2021-07-26T08:55:35.044+0000
>>> 7f0e15b3df40 -1 bluefs _allocate failed to expand slow device to fit
>>> +0x100716
>>> Jul 26 08:55:35 condor_sc0 docker[15282]:
>>> 2021-07-26T08:55:35.044+0000
>>> 7f0e15b3df40 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
>>> 0x100716
>>>
>>> This occurs during BlueFS recovery and that's an attempt to get more space
>>> to write out the bluefs log. This shouldn't fail given the plenty of free
>>> space:
>>>
>>> ... available 0x 1c09738000 ...
>>>
>>>
>>> So to get more verbose but less log one can set both debug-bluestore and
>>> debug-bluefs to 1/20. This way just last 10000 lines of the log preceeding
>>> the crash would be at level 20. Which seems sufficient for the
>>> troubleshooting.
>>>
>>> It would be also great to collect the output for the following commands:
>>>
>>> ceph-bluestore-tool --path <osd-dir> --command bluefs-bdev-sizes
>>>
>>> ceph-bluestore-tool --path <osd-dir> --command bluefs-stats
>>>
>>>
>>> And finally you can try to switch to bitmap allocator as a workaround -
>>> we've fixed a couple of issues in Hybrid one which prevented from proper
>>> allcoations under some circumstances. The fixes were made after v15.2.11
>>> release hence this might be the case. So please try setting:
>>>
>>> bluestore_allocator = bitmap
>>>
>>> bluefs_allocator = bitmap
>>>
>>>
>>> Thanks,
>>>
>>> Igor
>>>
>>>
>>> On 7/26/2021 12:14 PM, Dave Piper wrote:
>>>> Hi Igor,
>>>>
>>>> Thanks for your time looking into this.
>>>>
>>>> I've attached a 5 minute window of OSD logs, which includes several
>>>> restart attempt (each one takes ~25 seconds).
>>>>
>>>> When I said it looked like we were starting up in a different state, I'm
>>>> referring to how "Recovered from manifest file" log appears twice, with
>>>> different logs afterwards. This behaviour seems to repeat reliably on each
>>>> restart of the OSD. My interpretation of this was that when the initial
>>>> recovery attempt leads to the rocksdb shutdown, ceph is automatically
>>>> trying to start the OSD in some alternative state but that this is also
>>>> failing (with the bdev errors I copied in). Possibly I'm inferring too
>>>> much.
>>>>
>>>> I tried turning up the logging levels for rocksdb and bluestore but
>>>> they're both very spammy so I've not included this in the attached logs.
>>>> Let me know if you think that would be helpful.
>>>>
>>>> My ceph version is 15.2.11. We're running a containerized deployment using
>>>> docker image ceph-daemon:v5.0.10-stable-5.0-octopus-centos-8 .
>>>>
>>>> [qs-admin@condor_sc0 metaswitch]$ sudo docker exec b732f9135b42
>>>> ceph version ceph version 15.2.11
>>>> (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)
>>>>
>>>> Cheers,
>>>>
>>>> Dave
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Igor Fedotov <ifedo...@suse.de>
>>>> Sent: 23 July 2021 20:45
>>>> To: Dave Piper <david.pi...@microsoft.com>; ceph-users@ceph.io
>>>> Subject: [EXTERNAL] Re: [ceph-users] OSDs flapping with "_open_alloc
>>>> loaded 132 GiB in 2930776 extents available 113 GiB"
>>>>
>>>> Hi Dave,
>>>>
>>>> The follow log line indicates that allocator has just completed loading
>>>> information about free disk blocks into memory. And it looks perfectly
>>>> fine.
>>>>
>>>>> _open_alloc loaded 132 GiB in 2930776 extents available 113 GiB
>>>>
>>>>
>>>> Subsequent rocksdb shutdown looks weird without any other log output
>>>> indicating the issue.
>>>> Curious what do you mean under "
>>>>
>>>> After that we seem to try starting up in a slightly different state and
>>>> get a different set of errors:
>>>>
>>>> "
>>>> The resulted errors show lack of disk space at some point but I'd
>>>> definitely like to get the full startup log.
>>>>
>>>> Please also specify which Octopus version do you have?
>>>>
>>>> Thanks,
>>>> Igor
>>>>
>>>> On 7/23/2021 6:48 PM, Dave Piper wrote:
>>>>> Hi all,
>>>>>
>>>>> We've got a containerized test cluster with 3 OSDs and ~ 220GiB of data.
>>>>> Shortly after upgrading from nautilus -> octopus, 2 of the 3 OSDs have
>>>>> started flapping. I've also got alarms about the MDS being damaged, which
>>>>> we've seen elsewhere and have a recovery process for, but I'm unable to
>>>>> run this (I suspect because I've only got 1 functioning OSD). My RGWs are
>>>>> also failing to start, again I suspect because of the bad state of OSDs.
>>>>> I've tried restarting all OSDs, rebooting all servers, checked auth (all
>>>>> looks fine) - but I'm still in the same state.
>>>>>
>>>>> My OSDs seem to be failing at the "_open_alloc opening allocation
>>>>> metadata" step; looking at logs for each OSD restart, the OSD writes this
>>>>> log, then no logs for a few minutes and then logs:
>>>>>
>>>>> bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc loaded 132 GiB
>>>>> in 2930776 extents available 113 GiB
>>>>> rocksdb: [db/db_impl.cc:390] Shutdown: canceling all
>>>>> background work
>>>>>
>>>>> After that we seem to try starting up in a slightly different state and
>>>>> get a different set of errors:
>>>>>
>>>>> bluefs _allocate failed to allocate 0x100716 on bdev 1, free
>>>>> 0xd0000; fallback to bdev 2
>>>>> bluefs _allocate unable to allocate 0x100716 on bdev 2,
>>>>> free 0xffffffffffffffff; fallback to slow device expander
>>>>>
>>>>> and eventually crash and log a heap of stack dumps.
>>>>>
>>>>> I don't know what extents are but I seem to have a lot of them, and more
>>>>> than I've got capacity for? Maybe I'm running out of RAM or disk space
>>>>> somewhere, but I've got 21GB of free RAM on the server, and each OSD has
>>>>> a 350GiB device attached to it.
>>>>>
>>>>>
>>>>>
>>>>> I'm wondering if anyone has seen anything like this before or can suggest
>>>>> next debug steps to take?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Dave
>>>>>
>>>>>
>>>>>
>>>>> Full OSD logs surrounding the "_open_alloc opening allocation metadata"
>>>>> step:
>>>>>
>>>>>
>>>>> Jul 23 00:07:13 condor_sc0 container_name/ceph-osd-1[1709]:
>>>>> 2021-07-23T00:07:13.818+0000 7f3de111bf40 4 rocksdb: EVENT_LOG_v1
>>>>> {"time_micros": 1626998833819439, "job": 1, "event":
>>>>> "recovery_started", "log_files": [392088, 392132]}
>>>>>
>>>>> Jul 23 00:07:13 condor_sc0 container_name/ceph-osd-1[1709]:
>>>>> 2021-07-23T00:07:13.818+0000 7f3de111bf40 4 rocksdb:
>>>>> [db/db_impl_open.cc:583] Recovering log #392088 mode 0
>>>>>
>>>>> Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
>>>>> 2021-07-23T00:07:17.240+0000 7f3de111bf40 4 rocksdb:
>>>>> [db/db_impl_open.cc:583] Recovering log #392132 mode 0
>>>>>
>>>>> Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
>>>>> 2021-07-23T00:07:17.486+0000 7f3de111bf40 4 rocksdb: EVENT_LOG_v1
>>>>> {"time_micros": 1626998837486404, "job": 1, "event":
>>>>> "recovery_finished"}
>>>>>
>>>>> Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
>>>>> 2021-07-23T00:07:17.486+0000 7f3de111bf40 1
>>>>> bluestore(/var/lib/ceph/osd/ceph-1) _open_db opened rocksdb path
>>>>> db options
>>>>> compression=kNoCompression,max_write_buffer_number=4,min_write_buf
>>>>> fer
>>>>> _
>>>>> number_to_merge=1,recycle_log_file_num=4,write_buffer_size=2684354
>>>>> 56,
>>>>> w
>>>>> ritable_file_max_buffer_size=0,compaction_readahead_size=2097152,m
>>>>> ax_
>>>>> b
>>>>> ackground_compactions=2
>>>>>
>>>>> Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
>>>>> 2021-07-23T00:07:17.524+0000 7f3de111bf40 1 freelist init
>>>>>
>>>>> Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
>>>>> 2021-07-23T00:07:17.524+0000 7f3de111bf40 1 freelist
>>>>> _init_from_label
>>>>>
>>>>> Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
>>>>> 2021-07-23T00:07:17.529+0000 7f3de111bf40 1
>>>>> bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc opening allocation
>>>>> metadata
>>>>>
>>>>> Jul 23 00:07:18 condor_sc0 container_name/ceph-osd-1[1709]:
>>>>> 2021-07-23T00:07:18.238+0000 7f3de111bf40 1 HybridAllocator
>>>>> _spillover_range constructing fallback allocator
>>>>>
>>>>> Jul 23 00:07:20 condor_sc0 container_name/ceph-osd-1[1709]:
>>>>> 2021-07-23T00:07:20.563+0000 7f3de111bf40 1
>>>>> bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc loaded 132 GiB in
>>>>> 2930776 extents available 113 GiB
>>>>>
>>>>> Jul 23 00:07:20 condor_sc0 container_name/ceph-osd-1[1709]:
>>>>> 2021-07-23T00:07:20.563+0000 7f3de111bf40 4 rocksdb:
>>>>> [db/db_impl.cc:390] Shutdown: canceling all background work
>>>>>
>>>>> Jul 23 00:07:20 condor_sc0 container_name/ceph-osd-1[1709]:
>>>>> 2021-07-23T00:07:20.565+0000 7f3de111bf40 4 rocksdb:
>>>>> [db/db_impl.cc:563] Shutdown complete
>>>>>
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send
>>>>> an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
