[ https://issues.apache.org/jira/browse/KUDU-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719815#comment-16719815 ]
jiaqiyang edited comment on KUDU-2638 at 12/13/18 6:50 AM: ----------------------------------------------------------- {code:java} // code placeholder I1121 17:04:53.100796 165214 ts_tablet_manager.cc:909] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Loading tablet metadata I1121 17:07:06.116400 165214 ts_tablet_manager.cc:1082] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Registered tablet (data state: TABLET_DATA_READY) I1121 17:15:29.870625 168167 ts_tablet_manager.cc:932] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrapping tablet I1121 17:15:29.870635 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap starting. I1121 17:16:57.754650 168167 tablet_bootstrap.cc:616] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Time spent opening tablet: real 87.881s user 0.908s sys 0.340s I1121 17:16:59.455792 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:16:59.455893 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 1/14 log segments. Stats: ops{read=1614 overwritten=0 applied=1613 ignored=1476} inserts{seen=65 ignored=0} mutations{seen=423 ignored=0} orphaned_commits=1. Pending: 1 replicates I1121 17:16:59.456018 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000002 I1121 17:17:02.018604 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:17:02.018836 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000003 I1121 17:17:02.018995 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 2/14 log segments. Stats: ops{read=3892 overwritten=0 applied=3891 ignored=3256} inserts{seen=718 ignored=0} mutations{seen=2327 ignored=0} orphaned_commits=1. Pending: 1 replicates I1121 17:17:03.023664 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replaying log segment 3/14 (2.46M/7.99M this segment, stats: ops{read=4487 overwritten=0 applied=4487 ignored=3705} inserts{seen=881 ignored=0} mutations{seen=2898 ignored=0} orphaned_commits=1) I1121 17:17:04.888808 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:17:04.889019 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000004 I1121 17:17:04.889173 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 3/14 log segments. Stats: ops{read=5397 overwritten=0 applied=5396 ignored=4259} inserts{seen=1392 ignored=0} mutations{seen=4399 ignored=0} orphaned_commits=1. Pending: 1 replicates I1121 17:17:07.373458 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:17:07.373601 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 4/14 log segments. Stats: ops{read=7365 overwritten=0 applied=7364 ignored=5769} inserts{seen=1779 ignored=0} mutations{seen=6078 ignored=0} orphaned_commits=1. Pending: 1 replicates I1121 17:17:07.373723 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000005 I1121 17:17:08.071877 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replaying log segment 5/14 (2.36M/8.00M this segment, stats: ops{read=7680 overwritten=0 applied=7680 ignored=5940} inserts{seen=1972 ignored=0} mutations{seen=6778 ignored=0} orphaned_commits=1) I1121 17:17:09.348376 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:17:09.348531 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 5/14 log segments. Stats: ops{read=8550 overwritten=0 applied=8549 ignored=6513} inserts{seen=2274 ignored=0} mutations{seen=8124 ignored=0} orphaned_commits=1. Pending: 1 replicates I1121 17:17:09.348651 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000006 I1121 17:17:11.825079 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:17:11.825600 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000007 I1121 17:17:11.825631 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 6/14 log segments. Stats: ops{read=10643 overwritten=0 applied=10643 ignored=8051} inserts{seen=2796 ignored=0} mutations{seen=10250 ignored=0} orphaned_commits=1. Pending: 0 replicates I1121 17:17:13.076170 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replaying log segment 7/14 (6.40M/7.99M this segment, stats: ops{read=11683 overwritten=0 applied=11683 ignored=8705} inserts{seen=3136 ignored=0} mutations{seen=11879 ignored=0} orphaned_commits=1) I1121 17:17:13.955246 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:17:13.955353 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 7/14 log segments. Stats: ops{read=12097 overwritten=0 applied=12096 ignored=8966} inserts{seen=3223 ignored=0} mutations{seen=12355 ignored=0} orphaned_commits=1. Pending: 1 replicates I1121 17:17:13.955492 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000008 I1121 17:17:17.418372 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:17:17.418529 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 8/14 log segments. Stats: ops{read=14130 overwritten=0 applied=14129 ignored=10293} inserts{seen=3491 ignored=0} mutations{seen=14445 ignored=0} orphaned_commits=1. Pending: 1 replicates I1121 17:17:17.418648 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000009 I1121 17:17:18.142529 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replaying log segment 9/14 (1.20M/7.98M this segment, stats: ops{read=14463 overwritten=0 applied=14463 ignored=10519} inserts{seen=3516 ignored=0} mutations{seen=14705 ignored=0} orphaned_commits=1) I1121 17:17:21.280908 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:17:21.281145 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 9/14 log segments. Stats: ops{read=16094 overwritten=0 applied=16093 ignored=11561} inserts{seen=3744 ignored=0} mutations{seen=16403 ignored=0} orphaned_commits=1. Pending: 1 replicates I1121 17:17:21.281450 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000010 I1121 17:17:23.187130 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replaying log segment 10/14 (6.89M/7.99M this segment, stats: ops{read=17544 overwritten=0 applied=17544 ignored=12520} inserts{seen=4115 ignored=0} mutations{seen=18210 ignored=0} orphaned_commits=1) I1121 17:17:23.543048 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:17:23.543221 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 10/14 log segments. Stats: ops{read=17734 overwritten=0 applied=17733 ignored=12621} inserts{seen=4193 ignored=0} mutations{seen=18556 ignored=0} orphaned_commits=1. Pending: 1 replicates I1121 17:17:23.543499 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000011 I1121 17:17:25.603638 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:17:25.603852 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 11/14 log segments. Stats: ops{read=19220 overwritten=0 applied=19219 ignored=13516} inserts{seen=4522 ignored=0} mutations{seen=20514 ignored=0} orphaned_commits=1. Pending: 1 replicates I1121 17:17:25.603987 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000012 I1121 17:17:26.872901 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:17:26.873162 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000013 I1121 17:17:26.873526 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 12/14 log segments. Stats: ops{read=20455 overwritten=0 applied=20455 ignored=14196} inserts{seen=5022 ignored=0} mutations{seen=22764 ignored=0} orphaned_commits=1. Pending: 0 replicates I1121 17:17:28.198773 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replaying log segment 13/14 (7.64M/7.98M this segment, stats: ops{read=21983 overwritten=0 applied=21983 ignored=15140} inserts{seen=5350 ignored=0} mutations{seen=24568 ignored=0} orphaned_commits=1) I1121 17:17:28.335067 168167 log.cc:644] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Max segment size reached. Starting new segment allocation I1121 17:17:28.335395 168167 log.cc:571] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Rolled over to a new log segment at /data/data/kudu/tserver-new/wals/5aae5dc9e6f4468aaf00c060152d4fed/wal-000000014 I1121 17:17:28.503533 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 13/14 log segments. Stats: ops{read=22058 overwritten=0 applied=22057 ignored=15186} inserts{seen=5361 ignored=0} mutations{seen=24643 ignored=0} orphaned_commits=1. Pending: 1 replicates I1121 17:17:28.547025 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap replayed 14/14 log segments. Stats: ops{read=22298 overwritten=0 applied=22298 ignored=15321} inserts{seen=5426 ignored=0} mutations{seen=25058 ignored=0} orphaned_commits=1. Pending: 0 replicates I1121 17:17:28.558323 168167 tablet_bootstrap.cc:437] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Bootstrap complete. I1121 17:17:28.649260 168167 ts_tablet_manager.cc:957] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Time spent bootstrapping tablet: real 118.779s user 2.844s sys 0.760s I1121 17:17:28.649363 168167 raft_consensus.cc:297] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 195669 OBSOLETE_local: false peers { permanent_uuid: "510015b8e3d2462e9d52965cfa306af7" member_type: VOTER last_known_addr { host: "kudu16." port: 7050 } } peers { permanent_uuid: "227b096936f9442a875981a7e418d402" member_type: VOTER last_known_addr { host: "kudu12." port: 7050 } } I1121 17:17:28.649374 168167 raft_consensus.cc:598] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510015b8e3d2462e9d52965cfa306af7, State: Initialized, Role: FOLLOWER I1121 17:17:28.649402 168167 consensus_queue.cc:217] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 195669, Last appended: 4.195669, Last appended by leader: 195669, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 195669 OBSOLETE_local: false peers { permanent_uuid: "510015b8e3d2462e9d52965cfa306af7" member_type: VOTER last_known_addr { host: "kudu16." port: 7050 } } peers { permanent_uuid: "227b096936f9442a875981a7e418d402" member_type: VOTER last_known_addr { host: "kudu12." port: 7050 } } I1121 17:17:28.649459 168167 ts_tablet_manager.cc:985] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 17:17:30.138490 168926 raft_consensus.cc:429] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 17:17:30.138504 168926 raft_consensus.cc:2697] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Snoozing failure detection for 1.642s (starting election) I1121 17:17:30.138541 168926 raft_consensus.cc:451] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 195669 OBSOLETE_local: false peers { permanent_uuid: "510015b8e3d2462e9d52965cfa306af7" member_type: VOTER last_known_addr { host: "kudu16." port: 7050 } } peers { permanent_uuid: "227b096936f9442a875981a7e418d402" member_type: VOTER last_known_addr { host: "kudu12." port: 7050 } } I1121 17:17:30.138797 168926 leader_election.cc:231] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 227b096936f9442a875981a7e418d402 W1121 17:17:30.138998 166209 leader_election.cc:293] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 227b096936f9442a875981a7e418d402: Illegal state: must be running to vote when last-logged opid is not known I1121 17:17:30.139014 166209 leader_election.cc:258] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I1121 17:17:30.139045 168934 raft_consensus.cc:2697] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Snoozing failure detection for 2.227s (election complete) I1121 17:17:30.139055 168934 raft_consensus.cc:2441] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I1121 17:17:32.366122 168926 raft_consensus.cc:429] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 17:17:32.366138 168926 raft_consensus.cc:2697] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Snoozing failure detection for 2.648s (starting election) I1121 17:17:32.366163 168926 raft_consensus.cc:451] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 195669 OBSOLETE_local: false peers { permanent_uuid: "510015b8e3d2462e9d52965cfa306af7" member_type: VOTER last_known_addr { host: "kudu16." port: 7050 } } peers { permanent_uuid: "227b096936f9442a875981a7e418d402" member_type: VOTER last_known_addr { host: "kudu12." port: 7050 } } I1121 17:17:32.366503 168926 leader_election.cc:231] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 227b096936f9442a875981a7e418d402 W1121 17:17:32.366747 166209 leader_election.cc:293] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 227b096936f9442a875981a7e418d402: Illegal state: must be running to vote when last-logged opid is not known I1121 17:17:32.366765 166209 leader_election.cc:258] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I1121 17:17:32.366792 169268 raft_consensus.cc:2697] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Snoozing failure detection for 1.748s (election complete) I1121 17:17:32.366801 169268 raft_consensus.cc:2441] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I1121 17:17:33.888212 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(26926): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:34.114876 168934 raft_consensus.cc:429] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 17:17:34.114899 168934 raft_consensus.cc:2697] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Snoozing failure detection for 5.060s (starting election) I1121 17:17:34.114933 168934 raft_consensus.cc:451] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 195669 OBSOLETE_local: false peers { permanent_uuid: "510015b8e3d2462e9d52965cfa306af7" member_type: VOTER last_known_addr { host: "kudu16." port: 7050 } } peers { permanent_uuid: "227b096936f9442a875981a7e418d402" member_type: VOTER last_known_addr { host: "kudu12." port: 7050 } } I1121 17:17:34.115311 168934 leader_election.cc:231] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 227b096936f9442a875981a7e418d402 W1121 17:17:34.115607 166209 leader_election.cc:293] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 227b096936f9442a875981a7e418d402: Illegal state: must be running to vote when last-logged opid is not known I1121 17:17:34.115624 166209 leader_election.cc:258] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I1121 17:17:34.115649 168926 raft_consensus.cc:2697] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Snoozing failure detection for 4.142s (election complete) I1121 17:17:34.115660 168926 raft_consensus.cc:2441] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I1121 17:17:34.600126 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(27363): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:34.874528 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(27134): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:35.125910 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(27436): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:35.400064 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(27232): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:35.665071 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(25265): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:35.950384 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(22656): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:36.289477 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(23099): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:36.560333 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(28438): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:36.842886 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(24810): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:37.100394 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(26906): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:37.432247 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(26915): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:37.713157 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(26447): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:38.077538 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(26988): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:38.257942 168926 raft_consensus.cc:429] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 17:17:38.257966 168926 raft_consensus.cc:2697] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Snoozing failure detection for 6.528s (starting election) I1121 17:17:38.258010 168926 raft_consensus.cc:451] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 195669 OBSOLETE_local: false peers { permanent_uuid: "510015b8e3d2462e9d52965cfa306af7" member_type: VOTER last_known_addr { host: "kudu16." port: 7050 } } peers { permanent_uuid: "227b096936f9442a875981a7e418d402" member_type: VOTER last_known_addr { host: "kudu12." port: 7050 } } I1121 17:17:38.258285 168926 leader_election.cc:231] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 227b096936f9442a875981a7e418d402 W1121 17:17:38.258615 166209 leader_election.cc:293] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 227b096936f9442a875981a7e418d402: Illegal state: must be running to vote when last-logged opid is not known I1121 17:17:38.258639 166209 leader_election.cc:258] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I1121 17:17:38.258672 168934 raft_consensus.cc:2697] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Snoozing failure detection for 5.281s (election complete) I1121 17:17:38.258688 168934 raft_consensus.cc:2441] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I1121 17:17:38.375011 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(126): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:39.421612 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(25435): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:39.706590 165222 diskrowset.cc:564] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: RowSet(26994): Major compacting REDO delta stores (cols: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17) I1121 17:17:39.981084 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000023452936 (ts range=[6285524667418599424, 6298841746662703104], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:102,9:102,8:102,7:102,6:102,5:102,4:102,17:102,16:102,15:102,14:102,3:102,13:102,2:102,12:102,1:102,11:102)) I1121 17:17:39.981107 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000023989372 (ts range=[6299672578691997696, 6299875741096427520], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:5,9:5,8:5,7:5,6:5,5:5,4:5,17:5,16:5,15:5,14:5,3:5,13:5,2:5,12:5,1:5,11:5)) I1121 17:17:39.981117 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000025186188 (ts range=[6300593933888159744, 6300593933888159744], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:2,9:2,8:2,7:2,6:2,5:2,4:2,17:2,16:2,15:2,14:2,3:2,13:2,2:2,12:2,1:2,11:2)) I1121 17:17:39.981125 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000025971050 (ts range=[6301639806444765184, 6302216768460619776], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:6,9:6,8:6,7:6,6:6,5:6,4:6,17:6,16:6,15:6,14:6,3:6,13:6,2:6,12:6,1:6,11:6)) I1121 17:17:39.981135 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000026602108 (ts range=[6302514629513302016, 6303340137793593344], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:12,9:12,8:12,7:12,6:12,5:12,4:12,17:12,16:12,15:12,14:12,3:12,13:12,2:12,12:12,1:12,11:12)) I1121 17:17:39.981144 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000027514996 (ts range=[6303715663678509056, 6304755142271217664], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:15,9:15,8:15,7:15,6:15,5:15,4:15,17:15,16:15,15:15,14:15,3:15,13:15,2:15,12:15,1:15,11:15)) I1121 17:17:39.981151 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000028572396 (ts range=[6304973785559764992, 6306207747436335104], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:25,9:25,8:25,7:25,6:25,5:25,4:25,17:25,16:25,15:25,14:25,3:25,13:25,2:25,12:25,1:25,11:25)) I1121 17:17:39.981158 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000029615429 (ts range=[6306408860471484416, 6307621768480038912], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:12,9:12,8:12,7:12,6:12,5:12,4:12,17:12,16:12,15:12,14:12,3:12,13:12,2:12,12:12,1:12,11:12)) I1121 17:17:39.981163 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000033407572 (ts range=[6308014329249062912, 6309722443507134464], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:23,9:23,8:23,7:23,6:23,5:23,4:23,17:23,16:23,15:23,14:23,3:23,13:23,2:23,12:23,1:23,11:23)) I1121 17:17:39.981176 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000038792663 (ts range=[6313572028185088000, 6313608155229949952], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:4,9:4,8:4,7:4,6:4,5:4,4:4,17:4,16:4,15:4,14:4,3:4,13:4,2:4,12:4,1:4,11:4)) I1121 17:17:39.981194 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000038902521 (ts range=[6313616264792768512, 6313720303352397824], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:6,9:6,8:6,7:6,6:6,5:6,4:6,17:6,16:6,15:6,14:6,3:6,13:6,2:6,12:6,1:6,11:6)) I1121 17:17:39.981201 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000039522833 (ts range=[6313967296324382720, 6314406628765077504], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:3,9:3,8:3,7:3,6:3,5:3,4:3,17:3,16:3,15:3,14:3,3:3,13:3,2:3,12:3,1:3,11:3)) I1121 17:17:39.981209 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000040713796 (ts range=[6314719976939122688, 6315618798758551552], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:23,9:23,8:23,7:23,6:23,5:23,4:23,17:23,16:23,15:23,14:23,3:23,13:23,2:23,12:23,1:23,11:23)) I1121 17:17:39.981217 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000042650190 (ts range=[6316317903927185408, 6319085980321599488], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:36,9:36,8:36,7:36,6:36,5:36,4:36,17:36,16:36,15:36,14:36,3:36,13:36,2:36,12:36,1:36,11:36)) I1121 17:17:39.981237 165222 delta_tracker.cc:532] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Compacting 14 delta files. I1121 17:17:40.698355 165222 delta_tracker.cc:539] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Succesfully compacted the specified delta files. I1121 17:17:40.698376 165222 delta_tracker.cc:402] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Flushing compaction of redo delta blocks { 0000000023452936 0000000023989372 0000000025186188 0000000025971050 0000000026602108 0000000027514996 0000000028572396 0000000029615429 0000000033407572 0000000038792663 0000000038902521 0000000039522833 0000000040713796 0000000042650190 } into block 0000000042793690 I1121 17:17:40.851992 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000023739641 (ts range=[6285028314515738624, 6299249121620205568], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:174,9:174,8:174,7:174,6:174,5:174,4:174,17:174,16:174,15:174,14:174,3:174,13:174,2:174,12:174,1:174,11:174)) I1121 17:17:40.852017 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000023960401 (ts range=[6299491454344941568, 6299860831583752192], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:10,9:10,8:10,7:10,6:10,5:10,4:10,17:10,16:10,15:10,14:10,3:10,13:10,2:10,12:10,1:10,11:10)) I1121 17:17:40.852025 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000024502003 (ts range=[6300089961620824064, 6300390935577907200], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:7,9:7,8:7,7:7,6:7,5:7,4:7,17:7,16:7,15:7,14:7,3:7,13:7,2:7,12:7,1:7,11:7)) I1121 17:17:40.852033 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000025225142 (ts range=[6300826750502981632, 6301489155076800512], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:6,9:6,8:6,7:6,6:6,5:6,4:6,17:6,16:6,15:6,14:6,3:6,13:6,2:6,12:6,1:6,11:6)) I1121 17:17:40.852041 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000025682884 (ts range=[6301493906525007872, 6302038428513263616], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:12,9:12,8:12,7:12,6:12,5:12,4:12,17:12,16:12,15:12,14:12,3:12,13:12,2:12,12:12,1:12,11:12)) I1121 17:17:40.852047 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000026423496 (ts range=[6302345874789007360, 6303071685895675904], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:5,9:5,8:5,7:5,6:5,5:5,4:5,17:5,16:5,15:5,14:5,3:5,13:5,2:5,12:5,1:5,11:5)) I1121 17:17:40.852056 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000027003334 (ts range=[6303279516791078912, 6303771529024950272], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:4,9:4,8:4,7:4,6:4,5:4,4:4,17:4,16:4,15:4,14:4,3:4,13:4,2:4,12:4,1:4,11:4)) I1121 17:17:40.852062 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000027938574 (ts range=[6304369217132531712, 6305187024140967936], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:40,9:40,8:40,7:40,6:40,5:40,4:40,17:40,16:40,15:40,14:40,3:40,13:40,2:40,12:40,1:40,11:40)) I1121 17:17:40.852071 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000029012344 (ts range=[6305412631528894464, 6306927823983763456], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:18,9:18,8:18,7:18,6:18,5:18,4:18,17:18,16:18,15:18,14:18,3:18,13:18,2:18,12:18,1:18,11:18)) I1121 17:17:40.852078 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000030507842 (ts range=[6307221507768320000, 6309088792123260928], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:31,9:31,8:31,7:31,6:31,5:31,4:31,17:31,16:31,15:31,14:31,3:31,13:31,2:31,12:31,1:31,11:31)) I1121 17:17:40.852087 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000036326468 (ts range=[6309322837677707264, 6309780525484576768], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:13,9:13,8:13,7:13,6:13,5:13,4:13,17:13,16:13,15:13,14:13,3:13,13:13,2:13,12:13,1:13,11:13)) I1121 17:17:40.852093 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000038828967 (ts range=[6313605453234503680, 6313619459664740352], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:8,9:8,8:8,7:8,6:8,5:8,4:8,17:8,16:8,15:8,14:8,3:8,13:8,2:8,12:8,1:8,11:8)) I1121 17:17:40.852100 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000040529736 (ts range=[6314392539203596288, 6315957538107301888], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:10,9:10,8:10,7:10,6:10,5:10,4:10,17:10,16:10,15:10,14:10,3:10,13:10,2:10,12:10,1:10,11:10)) I1121 17:17:40.852105 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000042414430 (ts range=[6316043063334801408, 6318675643714973696], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:29,9:29,8:29,7:29,6:29,5:29,4:29,17:29,16:29,15:29,14:29,3:19,13:29,2:29,12:29,1:29,11:29)) I1121 17:17:40.852120 165222 delta_tracker.cc:532] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Compacting 14 delta files. I1121 17:17:41.222306 165222 delta_tracker.cc:539] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Succesfully compacted the specified delta files. I1121 17:17:41.222323 165222 delta_tracker.cc:402] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Flushing compaction of redo delta blocks { 0000000023739641 0000000023960401 0000000024502003 0000000025225142 0000000025682884 0000000026423496 0000000027003334 0000000027938574 0000000029012344 0000000030507842 0000000036326468 0000000038828967 0000000040529736 0000000042414430 } into block 0000000042793691 I1121 17:17:41.373366 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000023453043 (ts range=[6284980565251276800, 6299034668172173312], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:385,9:385,8:385,7:385,6:385,5:385,4:385,17:385,16:385,15:385,14:385,3:385,13:385,2:385,12:385,1:385,11:385)) I1121 17:17:41.373390 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000023777043 (ts range=[6299252984390647808, 6299372261325381632], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:3,9:3,8:3,7:3,6:3,5:3,4:3,17:3,16:3,15:3,14:3,3:3,13:3,2:3,12:3,1:3,11:3)) I1121 17:17:41.373400 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000024440932 (ts range=[6299850919132803072, 6300412972627501056], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:8,9:8,8:8,7:8,6:8,5:8,4:8,17:8,16:8,15:8,14:8,3:8,13:8,2:8,12:8,1:8,11:8)) I1121 17:17:41.373411 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000024968441 (ts range=[6300497597646741504, 6300903590836076544], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:19,9:19,8:19,7:19,6:19,5:19,4:19,17:19,16:19,15:19,14:19,3:19,13:19,2:19,12:19,1:19,11:19)) I1121 17:17:41.373420 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000025777928 (ts range=[6301306719738744832, 6302190800007405568], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:9,9:9,8:9,7:9,6:9,5:9,4:9,17:9,16:9,15:9,14:9,3:9,13:9,2:9,12:9,1:9,11:9)) I1121 17:17:41.373427 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000026718727 (ts range=[6302516841199108096, 6303342021819502592], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:5,9:5,8:5,7:5,6:5,5:5,4:5,17:5,16:5,15:5,14:5,3:5,13:5,2:5,12:5,1:5,11:5)) I1121 17:17:41.373436 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000027258877 (ts range=[6303615635301699584, 6304290245710598144], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:13,9:13,8:13,7:13,6:13,5:13,4:13,17:13,16:13,15:13,14:13,3:13,13:13,2:13,12:13,1:13,11:13)) I1121 17:17:41.373445 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000028181825 (ts range=[6304442215901667328, 6305713278230274048], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:37,9:37,8:37,7:37,6:37,5:37,4:37,17:37,16:37,15:37,14:37,3:37,13:37,2:37,12:37,1:37,11:37)) I1121 17:17:41.373451 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000029200005 (ts range=[6305759972572561408, 6307221098111090688], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:34,9:34,8:34,7:34,6:34,5:34,4:34,17:34,16:34,15:34,14:34,3:34,13:34,2:34,12:34,1:34,11:34)) I1121 17:17:41.373459 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000030532274 (ts range=[6307276230065197056, 6309295148360593408], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:21,9:21,8:21,7:21,6:21,5:21,4:21,17:21,16:21,15:21,14:21,3:21,13:21,2:21,12:21,1:21,11:21)) I1121 17:17:41.373466 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000036387613 (ts range=[6309441626499162112, 6309720395707756544], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:3,9:3,8:3,7:3,6:3,5:3,4:3,17:3,16:3,15:3,14:3,3:3,13:3,2:3,12:3,1:3,11:3)) I1121 17:17:41.373474 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000039019852 (ts range=[6313678032633446400, 6313926906097750016], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:4,9:4,8:4,7:4,6:4,5:4,4:4,17:4,16:4,15:4,14:4,3:4,13:4,2:4,12:4,1:4,11:4)) I1121 17:17:41.373481 165222 delta_tracker.cc:180] T 5aae5dc9e6f4468aaf00c060152d4fed P 510015b8e3d2462e9d52965cfa306af7: Preparing to minor compact delta file: 0000000039777124 (ts range=[6314044216133554176, 6314634284715159552], delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[10:10,9:10,8:10,7:10,6:10,5:10,4:10,17:10,16:10,15:10,14:10,3:10,13:10,2:10,12:10,1:10,11:10)) {code} was (Author: jiaqiyang): {code:java} // code placeholder {code} > kudu cluster restart very long time to reused > --------------------------------------------- > > Key: KUDU-2638 > URL: https://issues.apache.org/jira/browse/KUDU-2638 > Project: Kudu > Issue Type: Improvement > Reporter: jiaqiyang > Priority: Major > > when restart my kudu cluster ;all tablet not avalible: > run kudu cluster ksck show that: > Table Summary > > > Name | Status | Total Tablets | Healthy | Under-replicated | Unavailable > --------------------------------------------------------------------------------+------------ > t1 | HEALTHY | 1 | 1 | 0 | 0 > t2 | UNAVAILABLE | 5 | 0 | 1 | 4 > t3 | UNAVAILABLE | 6 | 2 | 0 | 4 > t3 | UNAVAILABLE | 3 | 0 | 0 | 3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)