[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

Hi Adrian, I have not tried, but I think it will resolve itself automatically after some minutes. How long have you waited before you do the manual redeploy? Could you also try “ceph mon dump” to see whether mon.node03 is actually removed from monmap when it failed to start? > 在 2021年5月23日，16:

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

So the orchestrator is aware of that mon is stopped, but not tried to bring it up again. What is the placement of mon shown in “ceph orch ls”? I explicitly set it to all host names (e.g. node01;node02;node03), and haven’t experienced this. > 在 2021年5月24日，00:35，Adrian Nicolae 写道： > > Hi, > >

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

Not sure it’s the issue, but it complaina bour msgr not msgr2, do you have the v1 amd v2 adresses in the ceph.conf on that specific osds? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

It's a fresh Pacific install with the default settings on all hosts : root@node01:/home/adrian# ceph config show-with-defaults mon.node03 | grep msgr mon_warn_on_msgr2_not_enabled true default ms_bind_msgr1 true default ms_bind_msgr2 true On 5/23/2021 5:50 PM, Szabo, Istvan (Agoda) wrote: No

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

Hi, I waited for more than a day on the first mon failure, it didn't resolve automatically. I checked with 'ceph status'  and also the ceph.conf on that hosts and the failed mon was removed from the monmap.  The cluster reported only 2 mons (instead of 3) and the third mon was completely rem

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

I think that the orchestrator is trying to bring it up but it's not starting (see the errors from my previous e-mail) - the container is not starting even if I tried to start it manually. the placement is the default one , ceph started the mons automatically on all my hosts because I only have

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

Hi, I wanted to explore the stretch mode in pacific (16.2.4) and see how it behaves with a DC failure. It seems as if I'm hitting the same or at least a similar issue here. To verify if it's the stretch mode I removed the cluster and rebuilt it without stretch mode, three hosts in three D

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

On Tue, May 25, 2021 at 7:17 AM Eugen Block wrote: > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/osd/OSDMap.cc: > In function 'void OSDMap::Incremental::enco

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

Thanks for the confirmation, Greg! I‘ll try with a newer release then. That’s why we’re testing, isn’t it? ;-) Then the OPs issue is probably not resolved yet since he didn’t mention a stretch cluster. Sorry for high-jacking the thread. Zitat von Gregory Farnum : On Tue, May 25, 2021 at 7:1

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

Hi, On my setup I didn't enable a strech cluster. It's just a 3 x VM setup running on the same Proxmox node, all the nodes are using a single unique network. I installed Ceph using the documented cephadm flow. Thanks for the confirmation, Greg! I‘ll try with a newer release then. >That’s wh

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

Hi, We are seeing very similar behavior on 16.2.5, and also have noticed that an undeploy/deploy cycle fixes things. Before we go rummaging through the source code trying to determine the root cause, has anybody else figured this out? It seems odd that a repeatable issue (I've seen other mailing l

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

Wanted to respond to the original thread I saw archived on this topic but I wasn't subscribed to the mailing list yet so don't have the thread in my inbox to reply to. Hopefully, those involved in that thread still see this. This issue looks the same as https://tracker.ceph.com/issues/51027 which

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

Hi, Am 09.08.21 um 20:44 schrieb Adam King: This issue looks the same as https://tracker.ceph.com/issues/51027 which is being worked on. Essentially, it seems that hosts that were being rebooted were temporarily marked as offline and cephadm had an issue where it would try to remove all daemons

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

Good morning Robert, Am 10.08.21 um 09:53 schrieb Robert Sander: Hi, Am 09.08.21 um 20:44 schrieb Adam King: This issue looks the same as https://tracker.ceph.com/issues/51027 which is being worked on. Essentially, it seems that hosts that were being rebooted were temporarily marked as offli

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

Just adding our feedback - this is affecting us as well. We reboot periodically to test durability of the clusters we run, and this is fairly impactful. I could see power loss/other scenarios in which this could end quite poorly for those with less than perfect redundancy in DCs across multiple rac

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

We're seeing the same here with v16.2.5 on CentOS 8.3 Do you know of any progress? Best Greetings André - Am 9. Aug 2021 um 18:15 schrieb David Orman orma...@corenode.com: > Hi, > > We are seeing very similar behavior on 16.2.5, and also have noticed > that an undeploy/deploy cycle fixes t

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

https://github.com/ceph/ceph/pull/42690 looks like it might be a fix, but it's pending review. On Thu, Aug 12, 2021 at 7:46 AM André Gemünd wrote: > > We're seeing the same here with v16.2.5 on CentOS 8.3 > > Do you know of any progress? > > Best Greetings > André > > - Am 9. Aug 2021 um 18:1