hi Wido, got the folowing outputt since ive changed the debug setting:
2018-07-26 11:46:21.004490 7f819e968700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46c4a800 :6789 s=STATE_OPEN pgs=71 cs=1 l=1)._try_send sent bytes 9 remaining bytes 0 2018-07-26 11:46:21.004520 7f81a196e700 10 -- 10.111.73.1:6789/0 dispatch_throttle_release 60 to dispatch throttler 60/104857600 2018-07-26 11:46:23.058057 7f81a4173700 1 -- 10.111.73.1:6789/0 >> 10.111.73.2:0/3994280291 conn(0x55aa46c46000 :6789 s=STATE_OPEN pgs=77 cs=1 l=1).mark_down 2018-07-26 11:46:23.058084 7f81a4173700 2 -- 10.111.73.1:6789/0 >> 10.111.73.2:0/3994280291 conn(0x55aa46c46000 :6789 s=STATE_OPEN pgs=77 cs=1 l=1)._stop 2018-07-26 11:46:23.058094 7f81a4173700 10 -- 10.111.73.1:6789/0 >> 10.111.73.2:0/3994280291 conn(0x55aa46c46000 :6789 s=STATE_OPEN pgs=77 cs=1 l=1).discard_out_queue started 2018-07-26 11:46:23.058120 7f81a4173700 1 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46c4a800 :6789 s=STATE_OPEN pgs=71 cs=1 l=1).mark_down 2018-07-26 11:46:23.058131 7f81a4173700 2 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46c4a800 :6789 s=STATE_OPEN pgs=71 cs=1 l=1)._stop 2018-07-26 11:46:23.058143 7f81a4173700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46c4a800 :6789 s=STATE_OPEN pgs=71 cs=1 l=1).discard_out_queue started 2018-07-26 11:46:23.962796 7f819d966700 10 Processor -- accept listen_fd=22 2018-07-26 11:46:23.962845 7f819d966700 10 Processor -- accept accepted incoming on sd 23 2018-07-26 11:46:23.962858 7f819d966700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46afd800 :-1 s=STATE_NONE pgs=0 cs=0 l=0).accept sd=23 2018-07-26 11:46:23.962929 7f819e167700 1 -- 10.111.73.1:6789/0 >> - conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=23 - 2018-07-26 11:46:23.963022 7f819e167700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._try_send sent bytes 281 remaining bytes 0 2018-07-26 11:46:23.963045 7f819e167700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection write banner and addr done: - 2018-07-26 11:46:23.963091 7f819e167700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection accept peer addr is 10.111.73.1:0/1745436331 2018-07-26 11:46:23.963190 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1)._process_connection accept of host_type 8, policy.lossy=1 policy.server=1 policy.standby=0 policy.resetcheck=0 2018-07-26 11:46:23.963216 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept my proto 15, their proto 15 2018-07-26 11:46:23.963232 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept setting up session_security. 2018-07-26 11:46:23.963248 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept new session 2018-07-26 11:46:23.963256 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=87 cs=1 l=1).handle_connect_msg accept success, connect_seq = 1 in_seq=0, sending READY 2018-07-26 11:46:23.963264 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=87 cs=1 l=1).handle_connect_msg accept features 4611087853745930235 2018-07-26 11:46:23.963315 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=87 cs=1 l=1)._try_send sent bytes 34 remaining bytes 0 2018-07-26 11:46:23.963356 7f819e167700 2 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=87 cs=1 l=1).handle_connect_msg accept write reply msg done 2018-07-26 11:46:23.963442 7f819e167700 2 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=87 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2018-07-26 11:46:23.963461 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=87 cs=1 l=1).discard_requeued_up_to 0 2018-07-26 11:46:23.963634 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN_KEEPALIVE2 pgs=87 cs=1 l=1)._append_keepalive_or_ack 2018-07-26 11:46:23.963658 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=87 cs=1 l=1).process wants 60 bytes from policy throttler 120/104857600 2018-07-26 11:46:23.963679 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1).process aborted = 0 2018-07-26 11:46:23.963705 7f819e167700 5 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1). rx client.? seq 1 0x55aa46be4480 auth(proto 0 30 bytes epoch 0) v1 2018-07-26 11:46:23.963750 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN pgs=87 cs=1 l=1).handle_write 2018-07-26 11:46:23.963755 7f81a196e700 1 -- 10.111.73.1:6789/0 <== client.? 10.111.73.1:0/1745436331 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (4135352935 0 0) 0x55aa46be4480 con 0x55aa46afd800 2018-07-26 11:46:23.963808 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.1:0/1745436331 conn(0x55aa46afd800 :6789 s=STATE_OPEN pgs=87 cs=1 l=1)._try_send sent bytes 9 remaining bytes 0 2018-07-26 11:46:23.963823 7f81a196e700 10 -- 10.111.73.1:6789/0 dispatch_throttle_release 60 to dispatch throttler 60/104857600 2018-07-26 11:46:24.003866 7f819d966700 10 Processor -- accept listen_fd=22 2018-07-26 11:46:24.003902 7f819d966700 10 Processor -- accept accepted incoming on sd 26 2018-07-26 11:46:24.003911 7f819d966700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46bc1000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).accept sd=26 2018-07-26 11:46:24.004001 7f819e167700 1 -- 10.111.73.1:6789/0 >> - conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=26 - 2018-07-26 11:46:24.004057 7f819e167700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._try_send sent bytes 281 remaining bytes 0 2018-07-26 11:46:24.004071 7f819e167700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection write banner and addr done: - 2018-07-26 11:46:24.004199 7f819e167700 10 -- 10.111.73.1:6789/0 >> - conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 cs=0 l=0)._process_connection accept peer addr is 10.111.73.3:0/1033315403 2018-07-26 11:46:24.004286 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1)._process_connection accept of host_type 8, policy.lossy=1 policy.server=1 policy.standby=0 policy.resetcheck=0 2018-07-26 11:46:24.004304 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept my proto 15, their proto 15 2018-07-26 11:46:24.004319 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept setting up session_security. 2018-07-26 11:46:24.004338 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept new session 2018-07-26 11:46:24.004351 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=74 cs=1 l=1).handle_connect_msg accept success, connect_seq = 1 in_seq=0, sending READY 2018-07-26 11:46:24.004365 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=74 cs=1 l=1).handle_connect_msg accept features 4611087853745930235 2018-07-26 11:46:24.004463 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=74 cs=1 l=1)._try_send sent bytes 34 remaining bytes 0 2018-07-26 11:46:24.004489 7f819e167700 2 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=74 cs=1 l=1).handle_connect_msg accept write reply msg done 2018-07-26 11:46:24.004634 7f819e167700 2 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=74 cs=1 l=1)._process_connection accept get newly_acked_seq 0 2018-07-26 11:46:24.004650 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_ACCEPTING_WAIT_SEQ pgs=74 cs=1 l=1).discard_requeued_up_to 0 2018-07-26 11:46:24.004807 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN_KEEPALIVE2 pgs=74 cs=1 l=1)._append_keepalive_or_ack 2018-07-26 11:46:24.004828 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=74 cs=1 l=1).process wants 60 bytes from policy throttler 180/104857600 2018-07-26 11:46:24.004847 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74 cs=1 l=1).process aborted = 0 2018-07-26 11:46:24.004873 7f819e167700 5 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74 cs=1 l=1). rx client.? seq 1 0x55aa46be4fc0 auth(proto 0 30 bytes epoch 0) v1 2018-07-26 11:46:24.004914 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN pgs=74 cs=1 l=1).handle_write 2018-07-26 11:46:24.004921 7f81a196e700 1 -- 10.111.73.1:6789/0 <== client.? 10.111.73.3:0/1033315403 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (2547518125 0 0) 0x55aa46be4fc0 con 0x55aa46bc1000 2018-07-26 11:46:24.004954 7f819e167700 10 -- 10.111.73.1:6789/0 >> 10.111.73.3:0/1033315403 conn(0x55aa46bc1000 :6789 s=STATE_OPEN pgs=74 cs=1 l=1)._try_send sent bytes 9 remaining bytes 0 2018-07-26 11:46:24.004965 7f81a196e700 10 -- 10.111.73.1:6789/0 dispatch_throttle_release 60 to dispatch throttler 60/104857600 kind regards ben > Wido den Hollander <w...@42on.com> hat am 26. Juli 2018 um 11:07 geschrieben: > > > > > On 07/26/2018 10:33 AM, Benjamin Naber wrote: > > hi Wido, > > > > thx for your reply. > > time is also in sync. i forced time sync again to be sure. > > > > Try setting debug_mon to 10 or even 20 and check the logs about what the > MONs are saying. > > debug_ms = 10 might also help to get some more information about the > Messenger Traffic. > > Wido > > > kind regards > > > > Ben > > > >> Wido den Hollander <w...@42on.com> hat am 26. Juli 2018 um 10:18 > > geschrieben: > >> > >> > >> > >> > >> On 07/26/2018 10:12 AM, Benjamin Naber wrote: > >> > Hi together, > >> > > >> > we currently have some problems with monitor quorum after shutting > > down all cluster nodes for migration to another location. > >> > > >> > mon_status gives uns the following outputt: > >> > > >> > { > >> > "name": "mon01", > >> > "rank": 0, > >> > "state": "electing", > >> > "election_epoch": 20345, > >> > "quorum": [], > >> > "features": { > >> > "required_con": "153140804152475648", > >> > "required_mon": [ > >> > "kraken", > >> > "luminous" > >> > ], > >> > "quorum_con": "0", > >> > "quorum_mon": [] > >> > }, > >> > "outside_quorum": [], > >> > "extra_probe_peers": [], > >> > "sync_provider": [], > >> > "monmap": { > >> > "epoch": 1, > >> > "fsid": "c1e3c489-67a4-47a2-a3ca-98816d1c9d44", > >> > "modified": "2018-06-21 13:48:58.796939", > >> > "created": "2018-06-21 13:48:58.796939", > >> > "features": { > >> > "persistent": [ > >> > "kraken", > >> > "luminous" > >> > ], > >> > "optional": [] > >> > }, > >> > "mons": [ > >> > { > >> > "rank": 0, > >> > "name": "mon01", > >> > "addr": "10.111.73.1:6789/0", > >> > "public_addr": "10.111.73.1:6789/0" > >> > }, > >> > { > >> > "rank": 1, > >> > "name": "mon02", > >> > "addr": "10.111.73.2:6789/0", > >> > "public_addr": "10.111.73.2:6789/0" > >> > }, > >> > { > >> > "rank": 2, > >> > "name": "mon03", > >> > "addr": "10.111.73.3:6789/0", > >> > "public_addr": "10.111.73.3:6789/0" > >> > } > >> > ] > >> > }, > >> > "feature_map": { > >> > "mon": { > >> > "group": { > >> > "features": "0x3ffddff8eea4fffb", > >> > "release": "luminous", > >> > "num": 1 > >> > } > >> > } > >> > } > >> > } > >> > > >> > ceph ping mon.id gives us also just dosent work. monitoring nodes > > have full network connectivity. firewall rules are also ok. > >> > > >> > what cloud be the reson for stucking quorum election ? > >> > > >> > >> Is the time in sync between the nodes? > >> > >> Wido > >> > >> > kind regards > >> > > >> > Ben > >> > _______________________________________________ > >> > ceph-users mailing list > >> > ceph-users@lists.ceph.com > >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com