hi Wido,
i have now one monitor online. i hve removed the two others from monmap.
how can i procedure, to reset that mon hosts and add them as new monitors to
the monmap?
king regards
Ben
> Wido den Hollander <w...@42on.com> hat am 26. Juli 2018 um 11:52 geschrieben:
>
>
>
>
> On 07/26/2018 11:50 AM, Benjamin Naber wrote:
> > hi Wido,
> >
> > got the folowing outputt since ive changed the debug setting:
> >
>
> This is only debug_ms it seems?
>
> debug_mon = 10
> debug_ms = 10
>
> Those two shoud be set where debug_mon will tell more about the election
> process.
>
> Wido
>
> > 2018-07-26 11:46:21.004490 7f819e968700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46c4a800 :6789 s=STATE_OPEN pgs=71
> > cs=1 l=1)._try_send sent bytes 9 remaining bytes 0
> > 2018-07-26 11:46:21.004520 7f81a196e700 10 -- 10.111.73.1:6789/0
> > dispatch_throttle_release 60 to dispatch throttler 60/104857600
> > 2018-07-26 11:46:23.058057 7f81a4173700 1 -- 10.111.73.1:6789/0 >>
> > 10.111.73.2:0/3994280291 conn(0x55aa46c46000 :6789 s=STATE_OPEN pgs=77
> > cs=1 l=1).mark_down
> > 2018-07-26 11:46:23.058084 7f81a4173700 2 -- 10.111.73.1:6789/0 >>
> > 10.111.73.2:0/3994280291 conn(0x55aa46c46000 :6789 s=STATE_OPEN pgs=77
> > cs=1 l=1)._stop
> > 2018-07-26 11:46:23.058094 7f81a4173700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.2:0/3994280291 conn(0x55aa46c46000 :6789 s=STATE_OPEN pgs=77
> > cs=1 l=1).discard_out_queue started
> > 2018-07-26 11:46:23.058120 7f81a4173700 1 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46c4a800 :6789 s=STATE_OPEN pgs=71
> > cs=1 l=1).mark_down
> > 2018-07-26 11:46:23.058131 7f81a4173700 2 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46c4a800 :6789 s=STATE_OPEN pgs=71
> > cs=1 l=1)._stop
> > 2018-07-26 11:46:23.058143 7f81a4173700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46c4a800 :6789 s=STATE_OPEN pgs=71
> > cs=1 l=1).discard_out_queue started
> > 2018-07-26 11:46:23.962796 7f819d966700 10 Processor -- accept listen_fd=22
> > 2018-07-26 11:46:23.962845 7f819d966700 10 Processor -- accept accepted
> > incoming on sd 23
> > 2018-07-26 11:46:23.962858 7f819d966700 10 -- 10.111.73.1:6789/0 >> -
> > conn(0x55aa46afd800 :-1 s=STATE_NONE pgs=0 cs=0 l=0).accept sd=23
> > 2018-07-26 11:46:23.962929 7f819e167700 1 -- 10.111.73.1:6789/0 >> -
> > conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING pgs=0 cs=0
> > l=0)._process_connection sd=23 -
> > 2018-07-26 11:46:23.963022 7f819e167700 10 -- 10.111.73.1:6789/0 >> -
> > conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._try_send
> > sent bytes 281 remaining bytes 0
> > 2018-07-26 11:46:23.963045 7f819e167700 10 -- 10.111.73.1:6789/0 >> -
> > conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0
> > l=0)._process_connection write banner and addr done: -
> > 2018-07-26 11:46:23.963091 7f819e167700 10 -- 10.111.73.1:6789/0 >> -
> > conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0
> > l=0)._process_connection accept peer addr is 10.111.73.1:0/1745436331
> > 2018-07-26 11:46:23.963190 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> > l=1)._process_connection accept of host_type 8, policy.lossy=1
> > policy.server=1 policy.standby=0 policy.resetcheck=0
> > 2018-07-26 11:46:23.963216 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> > l=1).handle_connect_msg accept my proto 15, their proto 15
> > 2018-07-26 11:46:23.963232 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> > l=1).handle_connect_msg accept setting up session_security.
> > 2018-07-26 11:46:23.963248 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> > l=1).handle_connect_msg accept new session
> > 2018-07-26 11:46:23.963256 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=87 cs=1
> > l=1).handle_connect_msg accept success, connect_seq = 1 in_seq=0,
> > sending READY
> > 2018-07-26 11:46:23.963264 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=87 cs=1
> > l=1).handle_connect_msg accept features 4611087853745930235
> > 2018-07-26 11:46:23.963315 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=87 cs=1 l=1)._try_send sent
> > bytes 34 remaining bytes 0
> > 2018-07-26 11:46:23.963356 7f819e167700 2 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_ACCEPTING_WAIT_SEQ pgs=87 cs=1 l=1).handle_connect_msg accept
> > write reply msg done
> > 2018-07-26 11:46:23.963442 7f819e167700 2 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_ACCEPTING_WAIT_SEQ pgs=87 cs=1 l=1)._process_connection accept
> > get newly_acked_seq 0
> > 2018-07-26 11:46:23.963461 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_ACCEPTING_WAIT_SEQ pgs=87 cs=1 l=1).discard_requeued_up_to 0
> > 2018-07-26 11:46:23.963634 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_OPEN_KEEPALIVE2 pgs=87 cs=1 l=1)._append_keepalive_or_ack
> > 2018-07-26 11:46:23.963658 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=87 cs=1 l=1).process wants 60
> > bytes from policy throttler 120/104857600
> > 2018-07-26 11:46:23.963679 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1).process
> > aborted = 0
> > 2018-07-26 11:46:23.963705 7f819e167700 5 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789
> > s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1). rx
> > client.? seq 1 0x55aa46be4480 auth(proto 0 30 bytes epoch 0) v1
> > 2018-07-26 11:46:23.963750 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN pgs=87
> > cs=1 l=1).handle_write
> > 2018-07-26 11:46:23.963755 7f81a196e700 1 -- 10.111.73.1:6789/0 <==
> > client.? 10.111.73.1:0/1745436331 1 ==== auth(proto 0 30 bytes epoch 0)
> > v1 ==== 60+0+0 (4135352935 0 0) 0x55aa46be4480 con 0x55aa46afd800
> > 2018-07-26 11:46:23.963808 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN pgs=87
> > cs=1 l=1)._try_send sent bytes 9 remaining bytes 0
> > 2018-07-26 11:46:23.963823 7f81a196e700 10 -- 10.111.73.1:6789/0
> > dispatch_throttle_release 60 to dispatch throttler 60/104857600
> > 2018-07-26 11:46:24.003866 7f819d966700 10 Processor -- accept listen_fd=22
> > 2018-07-26 11:46:24.003902 7f819d966700 10 Processor -- accept accepted
> > incoming on sd 26
> > 2018-07-26 11:46:24.003911 7f819d966700 10 -- 10.111.73.1:6789/0 >> -
> > conn(0x55aa46bc1000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).accept sd=26
> > 2018-07-26 11:46:24.004001 7f819e167700 1 -- 10.111.73.1:6789/0 >> -
> > conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING pgs=0 cs=0
> > l=0)._process_connection sd=26 -
> > 2018-07-26 11:46:24.004057 7f819e167700 10 -- 10.111.73.1:6789/0 >> -
> > conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._try_send
> > sent bytes 281 remaining bytes 0
> > 2018-07-26 11:46:24.004071 7f819e167700 10 -- 10.111.73.1:6789/0 >> -
> > conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0
> > l=0)._process_connection write banner and addr done: -
> > 2018-07-26 11:46:24.004199 7f819e167700 10 -- 10.111.73.1:6789/0 >> -
> > conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0
> > l=0)._process_connection accept peer addr is 10.111.73.3:0/1033315403
> > 2018-07-26 11:46:24.004286 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> > l=1)._process_connection accept of host_type 8, policy.lossy=1
> > policy.server=1 policy.standby=0 policy.resetcheck=0
> > 2018-07-26 11:46:24.004304 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> > l=1).handle_connect_msg accept my proto 15, their proto 15
> > 2018-07-26 11:46:24.004319 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> > l=1).handle_connect_msg accept setting up session_security.
> > 2018-07-26 11:46:24.004338 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
> > l=1).handle_connect_msg accept new session
> > 2018-07-26 11:46:24.004351 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=74 cs=1
> > l=1).handle_connect_msg accept success, connect_seq = 1 in_seq=0,
> > sending READY
> > 2018-07-26 11:46:24.004365 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=74 cs=1
> > l=1).handle_connect_msg accept features 4611087853745930235
> > 2018-07-26 11:46:24.004463 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=74 cs=1 l=1)._try_send sent
> > bytes 34 remaining bytes 0
> > 2018-07-26 11:46:24.004489 7f819e167700 2 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_ACCEPTING_WAIT_SEQ pgs=74 cs=1 l=1).handle_connect_msg accept
> > write reply msg done
> > 2018-07-26 11:46:24.004634 7f819e167700 2 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_ACCEPTING_WAIT_SEQ pgs=74 cs=1 l=1)._process_connection accept
> > get newly_acked_seq 0
> > 2018-07-26 11:46:24.004650 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_ACCEPTING_WAIT_SEQ pgs=74 cs=1 l=1).discard_requeued_up_to 0
> > 2018-07-26 11:46:24.004807 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_OPEN_KEEPALIVE2 pgs=74 cs=1 l=1)._append_keepalive_or_ack
> > 2018-07-26 11:46:24.004828 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=74 cs=1 l=1).process wants 60
> > bytes from policy throttler 180/104857600
> > 2018-07-26 11:46:24.004847 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74 cs=1 l=1).process
> > aborted = 0
> > 2018-07-26 11:46:24.004873 7f819e167700 5 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789
> > s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74 cs=1 l=1). rx
> > client.? seq 1 0x55aa46be4fc0 auth(proto 0 30 bytes epoch 0) v1
> > 2018-07-26 11:46:24.004914 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN pgs=74
> > cs=1 l=1).handle_write
> > 2018-07-26 11:46:24.004921 7f81a196e700 1 -- 10.111.73.1:6789/0 <==
> > client.? 10.111.73.3:0/1033315403 1 ==== auth(proto 0 30 bytes epoch 0)
> > v1 ==== 60+0+0 (2547518125 0 0) 0x55aa46be4fc0 con 0x55aa46bc1000
> > 2018-07-26 11:46:24.004954 7f819e167700 10 -- 10.111.73.1:6789/0 >>
> > 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN pgs=74
> > cs=1 l=1)._try_send sent bytes 9 remaining bytes 0
> > 2018-07-26 11:46:24.004965 7f81a196e700 10 -- 10.111.73.1:6789/0
> > dispatch_throttle_release 60 to dispatch throttler 60/104857600
> >
> > kind regards
> >
> > ben
> >
> >> Wido den Hollander <w...@42on.com> hat am 26. Juli 2018 um 11:07
> > geschrieben:
> >>
> >>
> >>
> >>
> >> On 07/26/2018 10:33 AM, Benjamin Naber wrote:
> >> > hi Wido,
> >> >
> >> > thx for your reply.
> >> > time is also in sync. i forced time sync again to be sure.
> >> >
> >>
> >> Try setting debug_mon to 10 or even 20 and check the logs about what the
> >> MONs are saying.
> >>
> >> debug_ms = 10 might also help to get some more information about the
> >> Messenger Traffic.
> >>
> >> Wido
> >>
> >> > kind regards
> >> >
> >> > Ben
> >> >
> >> >> Wido den Hollander <w...@42on.com> hat am 26. Juli 2018 um 10:18
> >> > geschrieben:
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 07/26/2018 10:12 AM, Benjamin Naber wrote:
> >> >> > Hi together,
> >> >> >
> >> >> > we currently have some problems with monitor quorum after shutting
> >> > down all cluster nodes for migration to another location.
> >> >> >
> >> >> > mon_status gives uns the following outputt:
> >> >> >
> >> >> > {
> >> >> > "name": "mon01",
> >> >> > "rank": 0,
> >> >> > "state": "electing",
> >> >> > "election_epoch": 20345,
> >> >> > "quorum": [],
> >> >> > "features": {
> >> >> > "required_con": "153140804152475648",
> >> >> > "required_mon": [
> >> >> > "kraken",
> >> >> > "luminous"
> >> >> > ],
> >> >> > "quorum_con": "0",
> >> >> > "quorum_mon": []
> >> >> > },
> >> >> > "outside_quorum": [],
> >> >> > "extra_probe_peers": [],
> >> >> > "sync_provider": [],
> >> >> > "monmap": {
> >> >> > "epoch": 1,
> >> >> > "fsid": "c1e3c489-67a4-47a2-a3ca-98816d1c9d44",
> >> >> > "modified": "2018-06-21 13:48:58.796939",
> >> >> > "created": "2018-06-21 13:48:58.796939",
> >> >> > "features": {
> >> >> > "persistent": [
> >> >> > "kraken",
> >> >> > "luminous"
> >> >> > ],
> >> >> > "optional": []
> >> >> > },
> >> >> > "mons": [
> >> >> > {
> >> >> > "rank": 0,
> >> >> > "name": "mon01",
> >> >> > "addr": "10.111.73.1:6789/0",
> >> >> > "public_addr": "10.111.73.1:6789/0"
> >> >> > },
> >> >> > {
> >> >> > "rank": 1,
> >> >> > "name": "mon02",
> >> >> > "addr": "10.111.73.2:6789/0",
> >> >> > "public_addr": "10.111.73.2:6789/0"
> >> >> > },
> >> >> > {
> >> >> > "rank": 2,
> >> >> > "name": "mon03",
> >> >> > "addr": "10.111.73.3:6789/0",
> >> >> > "public_addr": "10.111.73.3:6789/0"
> >> >> > }
> >> >> > ]
> >> >> > },
> >> >> > "feature_map": {
> >> >> > "mon": {
> >> >> > "group": {
> >> >> > "features": "0x3ffddff8eea4fffb",
> >> >> > "release": "luminous",
> >> >> > "num": 1
> >> >> > }
> >> >> > }
> >> >> > }
> >> >> > }
> >> >> >
> >> >> > ceph ping mon.id gives us also just dosent work. monitoring nodes
> >> > have full network connectivity. firewall rules are also ok.
> >> >> >
> >> >> > what cloud be the reson for stucking quorum election ?
> >> >> >
> >> >>
> >> >> Is the time in sync between the nodes?
> >> >>
> >> >> Wido
> >> >>
> >> >> > kind regards
> >> >> >
> >> >> > Ben
> >> >> > _______________________________________________
> >> >> > ceph-users mailing list
> >> >> > ceph-users@lists.ceph.com
> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >> >
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
