[ceph-users] Re: OSDs unable to mount BlueFS after reboot

2021-09-15 Thread Davíð Steinn Geirsson
Just realised the debug paste I sent was for OSD 5 but the other info is for OSD 0. They are both having the same issue, but for completeness sake here is the debug output from OSD 0: http://paste.debian.net/1211873/ All daemons in the cluster are running ceph pacific 16.2.5. Regards, Davíð On W

[ceph-users] Re: OSDs unable to mount BlueFS after reboot

2021-09-15 Thread Stefan Kooman
On 9/15/21 18:06, Davíð Steinn Geirsson wrote: Just realised the debug paste I sent was for OSD 5 but the other info is for OSD 0. They are both having the same issue, but for completeness sake here is the debug output from OSD 0: http://paste.debian.net/1211873/ All daemons in the cluster are r

[ceph-users] Re: OSDs unable to mount BlueFS after reboot

2021-09-15 Thread Davíð Steinn Geirsson
Hi, On Wed, Sep 15, 2021 at 08:39:11PM +0200, Stefan Kooman wrote: > On 9/15/21 18:06, Davíð Steinn Geirsson wrote: > > Just realised the debug paste I sent was for OSD 5 but the other info is for > > OSD 0. They are both having the same issue, but for completeness sake here > > is the debug outpu

[ceph-users] Re: OSDs unable to mount BlueFS after reboot

2021-09-15 Thread Stefan Kooman
On 9/15/21 21:02, Davíð Steinn Geirsson wrote: Hi, On Wed, Sep 15, 2021 at 08:39:11PM +0200, Stefan Kooman wrote: On 9/15/21 18:06, Davíð Steinn Geirsson wrote: Just realised the debug paste I sent was for OSD 5 but the other info is for OSD 0. They are both having the same issue, but for comp

[ceph-users] Re: OSDs unable to mount BlueFS after reboot

2021-09-15 Thread Davíð Steinn Geirsson
On Wed, Sep 15, 2021 at 09:16:17PM +0200, Stefan Kooman wrote: > On 9/15/21 21:02, Davíð Steinn Geirsson wrote: > > Hi, > > > > On Wed, Sep 15, 2021 at 08:39:11PM +0200, Stefan Kooman wrote: > > > On 9/15/21 18:06, Davíð Steinn Geirsson wrote: > > > > Just realised the debug paste I sent was for O

[ceph-users] Re: OSDs unable to mount BlueFS after reboot

2021-09-16 Thread Stefan Kooman
On 9/16/21 00:09, Davíð Steinn Geirsson wrote: You might get more information with increasing debug for rocksdb / bluefs / bluestore ceph config set osd.0 debug_rocksdb = 20/20 ceph config set osd.0 debug_bluefs = 20/20 ceph config set osd.0 debug_bluestore = 20/20 These debug tuneables give

[ceph-users] Re: OSDs unable to mount BlueFS after reboot

2021-09-16 Thread Davíð Steinn Geirsson
On Thu, Sep 16, 2021 at 08:17:48AM +0200, Stefan Kooman wrote: > On 9/16/21 00:09, Davíð Steinn Geirsson wrote: > > > > You might get more information with increasing debug for rocksdb / bluefs > > > / > > > bluestore > > > > > > ceph config set osd.0 debug_rocksdb = 20/20 > > > ceph config set

[ceph-users] Re: OSDs unable to mount BlueFS after reboot

2021-09-20 Thread Stefan Kooman
On 9/16/21 13:42, Davíð Steinn Geirsson wrote: The 4 affected drives are of 3 different types from 2 different vendors: ST16000NM001G-2KK103 ST12000VN0007-2GS116 WD60EFRX-68MYMN1 They are all connected through an LSI2308 SAS controller in IT mode. Other drives that did not fail are also connec

[ceph-users] Re: OSDs unable to mount BlueFS after reboot

2021-09-20 Thread Davíð Steinn Geirsson
On Mon, Sep 20, 2021 at 10:38:37AM +0200, Stefan Kooman wrote: > On 9/16/21 13:42, Davíð Steinn Geirsson wrote: > > > > > The 4 affected drives are of 3 different types from 2 different vendors: > > ST16000NM001G-2KK103 > > ST12000VN0007-2GS116 > > WD60EFRX-68MYMN1 > > > > They are all connecte

[ceph-users] Re: OSDs unable to mount BlueFS after reboot

2021-09-20 Thread Stefan Kooman
On 9/20/21 12:00, Davíð Steinn Geirsson wrote: Does the SAS controller run the latest firmware? As far as I can tell yes. Avago's website does not seem to list these anymore, but they are running firmware version 20 which is the latest I can find references to in a web search. This machine h