[ceph-users] Re: OSDs cannot join cluster anymore
Reminds me of https://tracker.ceph.com/issues/57007 which wasn't fixed in pacific until 16.2.11, so this is probably just the result of a cephadm bug unfortunately. On Fri, Jun 23, 2023 at 5:16 PM Malte Stroem wrote: > Hello Eugen, > > thanks. > > We found the cause. > > Somehow all > > /var/lib/ceph/fsid/osd.XX/config > > files on every host were still filled with expired information about the > mons. > > So refreshing the files helped to bring the osds up again. Damn. > > All other configs for the mons, mds', rgws and so on were up to date. > > I do not know why the osd config files did not get refreshed however I > guess something went wrong draining the nodes we removed from the cluster. > > Best regards, > Malte > > Am 21.06.23 um 22:11 schrieb Eugen Block: > > I still can’t really grasp what might have happened here. But could you > > please clarify which of the down OSDs (or Hosts) are supposed to be down > > and which you’re trying to bring back online? Obviously osd.40 is one of > > your attempts. But what about the hosts cephx01 and cephx08? Are those > > the ones refusing to start their OSDs? And the remaining up OSDs you > > haven’t touched yet, correct? > > And regarding debug logs, you should set it with ceph config set because > > the local ceph.conf won’t have an effect. It could help to have the > > startup debug logs from one of the OSDs. > > > > Zitat von Malte Stroem : > > > >> Hello Eugen, > >> > >> recovery and rebalancing was finished however now all PGs show missing > >> OSDs. > >> > >> Everything looks like the PGs are missing OSDs although it finished > >> correctly. > >> > >> As if we shut down the servers immediately. > >> > >> But we removed the nodes the way it is described in the documentation. > >> > >> We just added new disks and they join the cluster immediately. > >> > >> So the old OSDs removed from the cluster are available, I restored > >> OSD.40 but it does not want to join the cluster. > >> > >> Following are the outputs of the mentioned commands: > >> > >> ceph -s > >> > >> cluster: > >> id: X > >> health: HEALTH_WARN > >> 1 failed cephadm daemon(s) > >> 1 filesystem is degraded > >> 1 MDSs report slow metadata IOs > >> 19 osds down > >> 4 hosts (50 osds) down > >> Reduced data availability: 1220 pgs inactive > >> Degraded data redundancy: 132 pgs undersized > >> > >> services: > >> mon: 3 daemons, quorum cephx02,cephx04,cephx06 (age 4m) > >> mgr: cephx02.xx(active, since 92s), standbys: cephx04.yy, > >> cephx06.zz mds: 2/2 daemons up, 2 standby > >> osd: 130 osds: 78 up (since 13m), 97 in (since 35m); 171 remapped > pgs > >> rgw: 1 daemon active (1 hosts, 1 zones) > >> > >> data: > >> volumes: 1/2 healthy, 1 recovering > >> pools: 12 pools, 1345 pgs > >> objects: 11.02k objects, 1.9 GiB > >> usage: 145 TiB used, 669 TiB / 814 TiB avail > >> pgs: 86.617% pgs unknown > >> 4.089% pgs not active > >> 39053/33069 objects misplaced (118.095%) > >> 1165 unknown > >> 77 active+undersized+remapped > >> 55 undersized+remapped+peered > >> 38 active+clean+remapped > >> 10 active+clean > >> > >> ceph osd tree > >> > >> ID CLASS WEIGHT TYPE NAMESTATUS REWEIGHT > >> PRI-AFF > >> -214.36646 root ssds > >> -610.87329 host cephx01-ssd > >> 186ssd 0.87329 osd.186down 1.0 > >> 1.0 > >> -760.87329 host cephx02-ssd > >> 263ssd 0.87329 osd.263 up 1.0 > >> 1.0 > >> -850.87329 host cephx04-ssd > >> 237ssd 0.87329 osd.237 up 1.0 > >> 1.0 > >> -880.87329 host cephx06-ssd > >> 236ssd 0.87329 osd.236 up 1.0 > >> 1.0 > >> -940.87329 host cephx08-ssd > >> 262ssd 0.87329 osd.262down 1.0 > >> 1.0 > >> -1 1347.07397 root default > >> -62 261.93823 host cephx01 > >> 139hdd10.91409 osd.139down 0 > >> 1.0 > >> 140hdd10.91409 osd.140down 0 > >> 1.0 > >> 142hdd10.91409 osd.142down 0 > >> 1.0 > >> 144hdd10.91409 osd.144down 0 > >> 1.0 > >> 146hdd10.91409 osd.146down 0 > >> 1.0 > >> 148hdd10.91409 osd.148down 0 > >> 1.0 > >> 150hdd10.91409 osd.150down 0 > >> 1.0 > >> 152hdd10.91409 osd.152down 0 > >> 1.0 > >> 154hdd10.91409 osd.154down 1.0 > >>
[ceph-users] Re: OSDs cannot join cluster anymore
Oh, okay. I believe there was a thread reporting something very similar as well some time ago. I don’t remember the details but having outdated information on the OSDs was part of it. Were the nodes you removed also MON nodes? But it’s great that you found the root cause. Zitat von Malte Stroem : Hello Eugen, thanks. We found the cause. Somehow all /var/lib/ceph/fsid/osd.XX/config files on every host were still filled with expired information about the mons. So refreshing the files helped to bring the osds up again. Damn. All other configs for the mons, mds', rgws and so on were up to date. I do not know why the osd config files did not get refreshed however I guess something went wrong draining the nodes we removed from the cluster. Best regards, Malte Am 21.06.23 um 22:11 schrieb Eugen Block: I still can’t really grasp what might have happened here. But could you please clarify which of the down OSDs (or Hosts) are supposed to be down and which you’re trying to bring back online? Obviously osd.40 is one of your attempts. But what about the hosts cephx01 and cephx08? Are those the ones refusing to start their OSDs? And the remaining up OSDs you haven’t touched yet, correct? And regarding debug logs, you should set it with ceph config set because the local ceph.conf won’t have an effect. It could help to have the startup debug logs from one of the OSDs. Zitat von Malte Stroem : Hello Eugen, recovery and rebalancing was finished however now all PGs show missing OSDs. Everything looks like the PGs are missing OSDs although it finished correctly. As if we shut down the servers immediately. But we removed the nodes the way it is described in the documentation. We just added new disks and they join the cluster immediately. So the old OSDs removed from the cluster are available, I restored OSD.40 but it does not want to join the cluster. Following are the outputs of the mentioned commands: ceph -s cluster: id: X health: HEALTH_WARN 1 failed cephadm daemon(s) 1 filesystem is degraded 1 MDSs report slow metadata IOs 19 osds down 4 hosts (50 osds) down Reduced data availability: 1220 pgs inactive Degraded data redundancy: 132 pgs undersized services: mon: 3 daemons, quorum cephx02,cephx04,cephx06 (age 4m) mgr: cephx02.xx(active, since 92s), standbys: cephx04.yy, cephx06.zz mds: 2/2 daemons up, 2 standby osd: 130 osds: 78 up (since 13m), 97 in (since 35m); 171 remapped pgs rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/2 healthy, 1 recovering pools: 12 pools, 1345 pgs objects: 11.02k objects, 1.9 GiB usage: 145 TiB used, 669 TiB / 814 TiB avail pgs: 86.617% pgs unknown 4.089% pgs not active 39053/33069 objects misplaced (118.095%) 1165 unknown 77 active+undersized+remapped 55 undersized+remapped+peered 38 active+clean+remapped 10 active+clean ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -21 4.36646 root ssds -61 0.87329 host cephx01-ssd 186 ssd 0.87329 osd.186 down 1.0 1.0 -76 0.87329 host cephx02-ssd 263 ssd 0.87329 osd.263 up 1.0 1.0 -85 0.87329 host cephx04-ssd 237 ssd 0.87329 osd.237 up 1.0 1.0 -88 0.87329 host cephx06-ssd 236 ssd 0.87329 osd.236 up 1.0 1.0 -94 0.87329 host cephx08-ssd 262 ssd 0.87329 osd.262 down 1.0 1.0 -1 1347.07397 root default -62 261.93823 host cephx01 139 hdd 10.91409 osd.139 down 0 1.0 140 hdd 10.91409 osd.140 down 0 1.0 142 hdd 10.91409 osd.142 down 0 1.0 144 hdd 10.91409 osd.144 down 0 1.0 146 hdd 10.91409 osd.146 down 0 1.0 148 hdd 10.91409 osd.148 down 0 1.0 150 hdd 10.91409 osd.150 down 0 1.0 152 hdd 10.91409 osd.152 down 0 1.0 154 hdd 10.91409 osd.154 down 1.0 1.0 156 hdd 10.91409 osd.156 down 1.0 1.0 158 hdd 10.91409 osd.158 down 1.0 1.0 160 hdd 10.91409 osd.160 down 1.0 1.0 162 hdd 10.91409 osd.162 down 1.0 1.0 164 hdd 10.91409 osd.164 down 1.0 1.0 166 hdd 10.91409
[ceph-users] Re: OSDs cannot join cluster anymore
Hello Eugen, thanks. We found the cause. Somehow all /var/lib/ceph/fsid/osd.XX/config files on every host were still filled with expired information about the mons. So refreshing the files helped to bring the osds up again. Damn. All other configs for the mons, mds', rgws and so on were up to date. I do not know why the osd config files did not get refreshed however I guess something went wrong draining the nodes we removed from the cluster. Best regards, Malte Am 21.06.23 um 22:11 schrieb Eugen Block: I still can’t really grasp what might have happened here. But could you please clarify which of the down OSDs (or Hosts) are supposed to be down and which you’re trying to bring back online? Obviously osd.40 is one of your attempts. But what about the hosts cephx01 and cephx08? Are those the ones refusing to start their OSDs? And the remaining up OSDs you haven’t touched yet, correct? And regarding debug logs, you should set it with ceph config set because the local ceph.conf won’t have an effect. It could help to have the startup debug logs from one of the OSDs. Zitat von Malte Stroem : Hello Eugen, recovery and rebalancing was finished however now all PGs show missing OSDs. Everything looks like the PGs are missing OSDs although it finished correctly. As if we shut down the servers immediately. But we removed the nodes the way it is described in the documentation. We just added new disks and they join the cluster immediately. So the old OSDs removed from the cluster are available, I restored OSD.40 but it does not want to join the cluster. Following are the outputs of the mentioned commands: ceph -s cluster: id: X health: HEALTH_WARN 1 failed cephadm daemon(s) 1 filesystem is degraded 1 MDSs report slow metadata IOs 19 osds down 4 hosts (50 osds) down Reduced data availability: 1220 pgs inactive Degraded data redundancy: 132 pgs undersized services: mon: 3 daemons, quorum cephx02,cephx04,cephx06 (age 4m) mgr: cephx02.xx(active, since 92s), standbys: cephx04.yy, cephx06.zz mds: 2/2 daemons up, 2 standby osd: 130 osds: 78 up (since 13m), 97 in (since 35m); 171 remapped pgs rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/2 healthy, 1 recovering pools: 12 pools, 1345 pgs objects: 11.02k objects, 1.9 GiB usage: 145 TiB used, 669 TiB / 814 TiB avail pgs: 86.617% pgs unknown 4.089% pgs not active 39053/33069 objects misplaced (118.095%) 1165 unknown 77 active+undersized+remapped 55 undersized+remapped+peered 38 active+clean+remapped 10 active+clean ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -21 4.36646 root ssds -61 0.87329 host cephx01-ssd 186 ssd 0.87329 osd.186 down 1.0 1.0 -76 0.87329 host cephx02-ssd 263 ssd 0.87329 osd.263 up 1.0 1.0 -85 0.87329 host cephx04-ssd 237 ssd 0.87329 osd.237 up 1.0 1.0 -88 0.87329 host cephx06-ssd 236 ssd 0.87329 osd.236 up 1.0 1.0 -94 0.87329 host cephx08-ssd 262 ssd 0.87329 osd.262 down 1.0 1.0 -1 1347.07397 root default -62 261.93823 host cephx01 139 hdd 10.91409 osd.139 down 0 1.0 140 hdd 10.91409 osd.140 down 0 1.0 142 hdd 10.91409 osd.142 down 0 1.0 144 hdd 10.91409 osd.144 down 0 1.0 146 hdd 10.91409 osd.146 down 0 1.0 148 hdd 10.91409 osd.148 down 0 1.0 150 hdd 10.91409 osd.150 down 0 1.0 152 hdd 10.91409 osd.152 down 0 1.0 154 hdd 10.91409 osd.154 down 1.0 1.0 156 hdd 10.91409 osd.156 down 1.0 1.0 158 hdd 10.91409 osd.158 down 1.0 1.0 160 hdd 10.91409 osd.160 down 1.0 1.0 162 hdd 10.91409 osd.162 down 1.0 1.0 164 hdd 10.91409 osd.164 down 1.0 1.0 166 hdd 10.91409 osd.166 down 1.0 1.0 168 hdd 10.91409 osd.168 down 1.0 1.0 170 hdd 10.91409 osd.170 down 1.0 1.0 172 hdd 10.91409 osd.172 down 1.0 1.0 174 hdd
[ceph-users] Re: OSDs cannot join cluster anymore
On 6/21/23 11:20, Malte Stroem wrote: Hello Eugen, recovery and rebalancing was finished however now all PGs show missing OSDs. Everything looks like the PGs are missing OSDs although it finished correctly. As if we shut down the servers immediately. But we removed the nodes the way it is described in the documentation. We just added new disks and they join the cluster immediately. So the old OSDs removed from the cluster are available, I restored OSD.40 but it does not want to join the cluster. Are the osd.$id keys still there of the removed OSDs (check with ceph auth list)? Otherwise you might need to import the keyring into the cluster (/var/lib/ceph/osd/ceph-$id/keyring) and provide it proper CAPS. Gr. Stefan ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: OSDs cannot join cluster anymore
I still can’t really grasp what might have happened here. But could you please clarify which of the down OSDs (or Hosts) are supposed to be down and which you’re trying to bring back online? Obviously osd.40 is one of your attempts. But what about the hosts cephx01 and cephx08? Are those the ones refusing to start their OSDs? And the remaining up OSDs you haven’t touched yet, correct? And regarding debug logs, you should set it with ceph config set because the local ceph.conf won’t have an effect. It could help to have the startup debug logs from one of the OSDs. Zitat von Malte Stroem : Hello Eugen, recovery and rebalancing was finished however now all PGs show missing OSDs. Everything looks like the PGs are missing OSDs although it finished correctly. As if we shut down the servers immediately. But we removed the nodes the way it is described in the documentation. We just added new disks and they join the cluster immediately. So the old OSDs removed from the cluster are available, I restored OSD.40 but it does not want to join the cluster. Following are the outputs of the mentioned commands: ceph -s cluster: id: X health: HEALTH_WARN 1 failed cephadm daemon(s) 1 filesystem is degraded 1 MDSs report slow metadata IOs 19 osds down 4 hosts (50 osds) down Reduced data availability: 1220 pgs inactive Degraded data redundancy: 132 pgs undersized services: mon: 3 daemons, quorum cephx02,cephx04,cephx06 (age 4m) mgr: cephx02.xx(active, since 92s), standbys: cephx04.yy, cephx06.zz mds: 2/2 daemons up, 2 standby osd: 130 osds: 78 up (since 13m), 97 in (since 35m); 171 remapped pgs rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/2 healthy, 1 recovering pools: 12 pools, 1345 pgs objects: 11.02k objects, 1.9 GiB usage: 145 TiB used, 669 TiB / 814 TiB avail pgs: 86.617% pgs unknown 4.089% pgs not active 39053/33069 objects misplaced (118.095%) 1165 unknown 77 active+undersized+remapped 55 undersized+remapped+peered 38 active+clean+remapped 10 active+clean ceph osd tree ID CLASS WEIGHT TYPE NAMESTATUS REWEIGHT PRI-AFF -214.36646 root ssds -610.87329 host cephx01-ssd 186ssd 0.87329 osd.186down 1.0 1.0 -760.87329 host cephx02-ssd 263ssd 0.87329 osd.263 up 1.0 1.0 -850.87329 host cephx04-ssd 237ssd 0.87329 osd.237 up 1.0 1.0 -880.87329 host cephx06-ssd 236ssd 0.87329 osd.236 up 1.0 1.0 -940.87329 host cephx08-ssd 262ssd 0.87329 osd.262down 1.0 1.0 -1 1347.07397 root default -62 261.93823 host cephx01 139hdd10.91409 osd.139down 0 1.0 140hdd10.91409 osd.140down 0 1.0 142hdd10.91409 osd.142down 0 1.0 144hdd10.91409 osd.144down 0 1.0 146hdd10.91409 osd.146down 0 1.0 148hdd10.91409 osd.148down 0 1.0 150hdd10.91409 osd.150down 0 1.0 152hdd10.91409 osd.152down 0 1.0 154hdd10.91409 osd.154down 1.0 1.0 156hdd10.91409 osd.156down 1.0 1.0 158hdd10.91409 osd.158down 1.0 1.0 160hdd10.91409 osd.160down 1.0 1.0 162hdd10.91409 osd.162down 1.0 1.0 164hdd10.91409 osd.164down 1.0 1.0 166hdd10.91409 osd.166down 1.0 1.0 168hdd10.91409 osd.168down 1.0 1.0 170hdd10.91409 osd.170down 1.0 1.0 172hdd10.91409 osd.172down 1.0 1.0 174hdd10.91409 osd.174down 1.0 1.0 176hdd10.91409 osd.176down 1.0 1.0 178hdd10.91409 osd.178down 1.0 1.0 180hdd10.91409 osd.180down 1.0 1.0 182hdd10.91409 osd.182down 1.0 1.0 184hdd10.91409 osd.184down 1.0 1.0 -67 261.93823 host cephx02 138hdd10.91409 osd.138 up 1.0
[ceph-users] Re: OSDs cannot join cluster anymore
Hello Eugen, recovery and rebalancing was finished however now all PGs show missing OSDs. Everything looks like the PGs are missing OSDs although it finished correctly. As if we shut down the servers immediately. But we removed the nodes the way it is described in the documentation. We just added new disks and they join the cluster immediately. So the old OSDs removed from the cluster are available, I restored OSD.40 but it does not want to join the cluster. Following are the outputs of the mentioned commands: ceph -s cluster: id: X health: HEALTH_WARN 1 failed cephadm daemon(s) 1 filesystem is degraded 1 MDSs report slow metadata IOs 19 osds down 4 hosts (50 osds) down Reduced data availability: 1220 pgs inactive Degraded data redundancy: 132 pgs undersized services: mon: 3 daemons, quorum cephx02,cephx04,cephx06 (age 4m) mgr: cephx02.xx(active, since 92s), standbys: cephx04.yy, cephx06.zz mds: 2/2 daemons up, 2 standby osd: 130 osds: 78 up (since 13m), 97 in (since 35m); 171 remapped pgs rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/2 healthy, 1 recovering pools: 12 pools, 1345 pgs objects: 11.02k objects, 1.9 GiB usage: 145 TiB used, 669 TiB / 814 TiB avail pgs: 86.617% pgs unknown 4.089% pgs not active 39053/33069 objects misplaced (118.095%) 1165 unknown 77 active+undersized+remapped 55 undersized+remapped+peered 38 active+clean+remapped 10 active+clean ceph osd tree ID CLASS WEIGHT TYPE NAMESTATUS REWEIGHT PRI-AFF -214.36646 root ssds -610.87329 host cephx01-ssd 186ssd 0.87329 osd.186down 1.0 1.0 -760.87329 host cephx02-ssd 263ssd 0.87329 osd.263 up 1.0 1.0 -850.87329 host cephx04-ssd 237ssd 0.87329 osd.237 up 1.0 1.0 -880.87329 host cephx06-ssd 236ssd 0.87329 osd.236 up 1.0 1.0 -940.87329 host cephx08-ssd 262ssd 0.87329 osd.262down 1.0 1.0 -1 1347.07397 root default -62 261.93823 host cephx01 139hdd10.91409 osd.139down 0 1.0 140hdd10.91409 osd.140down 0 1.0 142hdd10.91409 osd.142down 0 1.0 144hdd10.91409 osd.144down 0 1.0 146hdd10.91409 osd.146down 0 1.0 148hdd10.91409 osd.148down 0 1.0 150hdd10.91409 osd.150down 0 1.0 152hdd10.91409 osd.152down 0 1.0 154hdd10.91409 osd.154down 1.0 1.0 156hdd10.91409 osd.156down 1.0 1.0 158hdd10.91409 osd.158down 1.0 1.0 160hdd10.91409 osd.160down 1.0 1.0 162hdd10.91409 osd.162down 1.0 1.0 164hdd10.91409 osd.164down 1.0 1.0 166hdd10.91409 osd.166down 1.0 1.0 168hdd10.91409 osd.168down 1.0 1.0 170hdd10.91409 osd.170down 1.0 1.0 172hdd10.91409 osd.172down 1.0 1.0 174hdd10.91409 osd.174down 1.0 1.0 176hdd10.91409 osd.176down 1.0 1.0 178hdd10.91409 osd.178down 1.0 1.0 180hdd10.91409 osd.180down 1.0 1.0 182hdd10.91409 osd.182down 1.0 1.0 184hdd10.91409 osd.184down 1.0 1.0 -67 261.93823 host cephx02 138hdd10.91409 osd.138 up 1.0 1.0 141hdd10.91409 osd.141 up 1.0 1.0 143hdd10.91409 osd.143 up 1.0 1.0 145hdd10.91409 osd.145 up 1.0 1.0 147hdd10.91409 osd.147 up 1.0 1.0 149hdd10.91409 osd.149 up 1.0 1.0 151hdd10.91409 osd.151 up 1.0 1.0 153hdd10.91409 osd.153 up 1.0 1.0 155hdd10.91409 osd.155 up 1.0 1.0 157
[ceph-users] Re: OSDs cannot join cluster anymore
Hi, Yes, we drained the nodes. It needed two weeks to finish the process, and yes, I think this is the root cause. So we still have the nodes but when I try to restart one of those OSDs it still cannot join: if the nodes were drained successfully (can you confirm that all PGs were active+clean after draining before you removed the nodes?) then the disks on the removed nodes wouldn't have any data to bring back. The question would be, why do the remaining OSDs still reference removed OSDs. Or am I misunderstanding something? I think it would help to know the whole story, can you provide more details? Also some more general cluster info would be helpful: $ ceph -s $ ceph osd tree $ ceph health detail Zitat von Malte Stroem : Hello Eugen, thank you. Yesterday I thought: Well, Eugen can help! Yes, we drained the nodes. It needed two weeks to finish the process, and yes, I think this is the root cause. So we still have the nodes but when I try to restart one of those OSDs it still cannot join: Jun 21 09:46:03 ceph-node bash[2323668]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-66/block Jun 21 09:46:03 ceph-node bash[2323668]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-19 Jun 21 09:46:03 ceph-node bash[2323668]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-66 Jun 21 09:46:03 ceph-node bash[2323668]: --> ceph-volume lvm activate successful for osd ID: 66 Jun 21 09:51:04 ceph-node bash[2323668]: debug 2023-06-21T07:51:04.176+ 7fabef5a1200 0 monclient(hunting): authenticate timed out after 300 Jun 21 09:56:04 ceph-node bash[2323668]: debug 2023-06-21T07:56:04.179+ 7fabef5a1200 0 monclient(hunting): authenticate timed out after 300 Jun 21 10:01:04 ceph-node bash[2323668]: debug 2023-06-21T08:01:04.177+ 7fabef5a1200 0 monclient(hunting): authenticate timed out after 300 Jun 21 10:06:04 ceph-node bash[2323668]: debug 2023-06-21T08:06:04.179+ 7fabef5a1200 0 monclient(hunting): authenticate timed out after 300 Jun 21 10:11:04 ceph-node bash[2323668]: debug 2023-06-21T08:11:04.174+ 7fabef5a1200 0 monclient(hunting): authenticate timed out after 300 Same messages on all OSDs. We still have some nodes running and did not restart those OSDs. Best, Malte Am 21.06.23 um 09:50 schrieb Eugen Block: Hi, can you share more details what exactly you did? How did you remove the nodes? Hopefully, you waited for the draining to finish? But if the remaining OSDs wait for removed OSDs it sounds like the draining was not finished. Zitat von Malte Stroem : Hello, we removed some nodes from our cluster. This worked without problems. Now, lots of OSDs do not want to join the cluster anymore if we reboot one of the still available nodes. It always runs into timeouts: --> ceph-volume lvm activate successful for osd ID: XX monclient(hunting): authenticate timed out after 300 MONs and MGRs are running fine. Network is working, netcat to the MONs' ports are open. Setting a higher debug level has no effect even if we add it to the ceph.conf file. The PGs are pretty unhappy, e. g.: 7.143 87771 0 0 0 0 314744902235 0 0 10081 10081 down 2023-06-20T09:16:03.546158+ 961275'1395646 961300:9605547 [209,NONE,NONE] 209 [209,NONE,NONE] 209 961231'1395512 2023-06-19T23:46:40.101791+ 961231'1395512 2023-06-19T23:46:40.101791+ PG query wants us to set an OSD lost however I do not want to do this. OSDs are blocked by OSDs from the removed nodes: ceph osd blocked-by osd num_blocked 152 38 244 41 144 54 ... We added the removed hosts again and tried to start the OSDs on this node and they also failed into the timeout mentioned above. This is a containerized cluster running version 16.2.10. Replication is 3, some pools use an erasure coded profile. Best regards, Malte ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: OSDs cannot join cluster anymore
Hello Eugen, thank you. Yesterday I thought: Well, Eugen can help! Yes, we drained the nodes. It needed two weeks to finish the process, and yes, I think this is the root cause. So we still have the nodes but when I try to restart one of those OSDs it still cannot join: Jun 21 09:46:03 ceph-node bash[2323668]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-66/block Jun 21 09:46:03 ceph-node bash[2323668]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-19 Jun 21 09:46:03 ceph-node bash[2323668]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-66 Jun 21 09:46:03 ceph-node bash[2323668]: --> ceph-volume lvm activate successful for osd ID: 66 Jun 21 09:51:04 ceph-node bash[2323668]: debug 2023-06-21T07:51:04.176+ 7fabef5a1200 0 monclient(hunting): authenticate timed out after 300 Jun 21 09:56:04 ceph-node bash[2323668]: debug 2023-06-21T07:56:04.179+ 7fabef5a1200 0 monclient(hunting): authenticate timed out after 300 Jun 21 10:01:04 ceph-node bash[2323668]: debug 2023-06-21T08:01:04.177+ 7fabef5a1200 0 monclient(hunting): authenticate timed out after 300 Jun 21 10:06:04 ceph-node bash[2323668]: debug 2023-06-21T08:06:04.179+ 7fabef5a1200 0 monclient(hunting): authenticate timed out after 300 Jun 21 10:11:04 ceph-node bash[2323668]: debug 2023-06-21T08:11:04.174+ 7fabef5a1200 0 monclient(hunting): authenticate timed out after 300 Same messages on all OSDs. We still have some nodes running and did not restart those OSDs. Best, Malte Am 21.06.23 um 09:50 schrieb Eugen Block: Hi, can you share more details what exactly you did? How did you remove the nodes? Hopefully, you waited for the draining to finish? But if the remaining OSDs wait for removed OSDs it sounds like the draining was not finished. Zitat von Malte Stroem : Hello, we removed some nodes from our cluster. This worked without problems. Now, lots of OSDs do not want to join the cluster anymore if we reboot one of the still available nodes. It always runs into timeouts: --> ceph-volume lvm activate successful for osd ID: XX monclient(hunting): authenticate timed out after 300 MONs and MGRs are running fine. Network is working, netcat to the MONs' ports are open. Setting a higher debug level has no effect even if we add it to the ceph.conf file. The PGs are pretty unhappy, e. g.: 7.143 87771 0 0 0 0 314744902235 0 0 10081 10081 down 2023-06-20T09:16:03.546158+ 961275'1395646 961300:9605547 [209,NONE,NONE] 209 [209,NONE,NONE] 209 961231'1395512 2023-06-19T23:46:40.101791+ 961231'1395512 2023-06-19T23:46:40.101791+ PG query wants us to set an OSD lost however I do not want to do this. OSDs are blocked by OSDs from the removed nodes: ceph osd blocked-by osd num_blocked 152 38 244 41 144 54 ... We added the removed hosts again and tried to start the OSDs on this node and they also failed into the timeout mentioned above. This is a containerized cluster running version 16.2.10. Replication is 3, some pools use an erasure coded profile. Best regards, Malte ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: OSDs cannot join cluster anymore
Hi, can you share more details what exactly you did? How did you remove the nodes? Hopefully, you waited for the draining to finish? But if the remaining OSDs wait for removed OSDs it sounds like the draining was not finished. Zitat von Malte Stroem : Hello, we removed some nodes from our cluster. This worked without problems. Now, lots of OSDs do not want to join the cluster anymore if we reboot one of the still available nodes. It always runs into timeouts: --> ceph-volume lvm activate successful for osd ID: XX monclient(hunting): authenticate timed out after 300 MONs and MGRs are running fine. Network is working, netcat to the MONs' ports are open. Setting a higher debug level has no effect even if we add it to the ceph.conf file. The PGs are pretty unhappy, e. g.: 7.143 87771 0 0 00 3147449022350 0 10081 10081 down 2023-06-20T09:16:03.546158+961275'1395646 961300:9605547 [209,NONE,NONE] 209 [209,NONE,NONE] 209961231'1395512 2023-06-19T23:46:40.101791+961231'1395512 2023-06-19T23:46:40.101791+ PG query wants us to set an OSD lost however I do not want to do this. OSDs are blocked by OSDs from the removed nodes: ceph osd blocked-by osd num_blocked 152 38 244 41 144 54 ... We added the removed hosts again and tried to start the OSDs on this node and they also failed into the timeout mentioned above. This is a containerized cluster running version 16.2.10. Replication is 3, some pools use an erasure coded profile. Best regards, Malte ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
