[
https://issues.apache.org/jira/browse/KUDU-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bakai Ádám updated KUDU-3634:
-----------------------------
Description:
{code:java}
F20241218 13:05:04.229065 4953 ksck_results.cc:160] Check failed:
servers.size() <= server_count (6 vs. 3)
*** Check failure stack trace: ***
*** Aborted at 1734527104 (unix time) try "date -d @1734527104" if you are
using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800001359) received by PID 4953 (TID 0x7f5270649880) from PID
4953; stack trace: ***
@ 0x2d8140 __tsan::CallUserSignalHandler() at
/root/tmp/test123/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2da623 rtl_sigaction() at
/root/tmp/test123/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20241218 13:05:04.427536 14086 raft_consensus.cc:1270] T
5d3037cff4d843b0a7dbc9ff78a339ba P 3e6bbc8984eb4fbfbefd27d5036db51a [term 5
FOLLOWER]: Refusing update from remote peer 1a37ef9543954c8e823eeae2d8592107:
Log matching property violated. Preceding OpId in replica: term: 4 index: 3.
Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20241218 13:05:04.429109 14549 consensus_queue.cc:1035] T
5d3037cff4d843b0a7dbc9ff78a339ba P 1a37ef9543954c8e823eeae2d8592107 [LEADER]:
Connected to new peer: Peer: permanent_uuid: "3e6bbc8984eb4fbfbefd27d5036db51a"
member_type: VOTER last_known_addr { host: "127.4.214.67" port: 37251 },
Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed
idx: 3, Time since last communication: 0.000s
@ 0x7f527808f980 (unknown) at ??:0
I20241218 13:05:04.509820 14545 raft_consensus.cc:491] T
5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5
FOLLOWER]: Starting pre-election (no leader contacted us within the election
timeout)
I20241218 13:05:04.510313 14545 raft_consensus.cc:513] T
5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5
FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local:
false peers { permanent_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" member_type:
VOTER last_known_addr { host: "127.4.214.65" port: 36387 } } peers {
permanent_uuid: "1a37ef9543954c8e823eeae2d8592107" member_type: VOTER
last_known_addr { host: "127.4.214.66" port: 36001 } } peers { permanent_uuid:
"3e6bbc8984eb4fbfbefd27d5036db51a" member_type: VOTER last_known_addr { host:
"127.4.214.67" port: 37251 } }
I20241218 13:05:04.513247 14001 tablet_service.cc:1812] Received
RequestConsensusVote() RPC: tablet_id: "5d3037cff4d843b0a7dbc9ff78a339ba"
candidate_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" candidate_term: 6
candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader:
false dest_uuid: "1a37ef9543954c8e823eeae2d8592107" is_pre_election: true
I20241218 13:05:04.516223 14089 tablet_service.cc:1812] Received
RequestConsensusVote() RPC: tablet_id: "5d3037cff4d843b0a7dbc9ff78a339ba"
candidate_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" candidate_term: 6
candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader:
false dest_uuid: "3e6bbc8984eb4fbfbefd27d5036db51a" is_pre_election: true
I20241218 13:05:04.517762 13891 leader_election.cc:304] T
5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a
[CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost.
Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no
votes. yes voters: 0bfce15bbb5f4f0aa5a9e60872491f3a; no voters:
1a37ef9543954c8e823eeae2d8592107, 3e6bbc8984eb4fbfbefd27d5036db51a
I20241218 13:05:04.518554 14506 raft_consensus.cc:2743] T
5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5
FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve
majority
I20241218 13:05:04.518522 14545 leader_election.cc:290] T
5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a
[CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers
1a37ef9543954c8e823eeae2d8592107 (127.4.214.66:36001),
3e6bbc8984eb4fbfbefd27d5036db51a (127.4.214.67:37251)
@ 0x7f5271309fb7 gsignal at ??:0
I20241218 13:05:04.633380 13919 raft_consensus.cc:1270] T
5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5
FOLLOWER]: Refusing update from remote peer 1a37ef9543954c8e823eeae2d8592107:
Log matching property violated. Preceding OpId in replica: term: 3 index: 2.
Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20241218 13:05:04.634791 14549 consensus_queue.cc:1035] T
5d3037cff4d843b0a7dbc9ff78a339ba P 1a37ef9543954c8e823eeae2d8592107 [LEADER]:
Connected to new peer: Peer: permanent_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a"
member_type: VOTER last_known_addr { host: "127.4.214.65" port: 36387 },
Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed
idx: 2, Time since last communication: 0.000s
@ 0x7f527130b921 abort at ??:0
@ 0x2de136 abort at
/root/tmp/test123/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f5273d37d64 google::LogMessage::SendToLog() at ??:0
@ 0x7f5273d38910 google::LogMessage::Flush() at ??:0
@ 0x7f5273d3da4b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f527ec9165b kudu::tools::(anonymous namespace)::ServerCsv() at ??:0
@ 0x7f527ec89f4e kudu::tools::PrintTaggedFlagTable() at ??:0
@ 0x7f527ec8415e kudu::tools::KsckResults::PrintTo() at ??:0
@ 0x7f527eb8e0f9 kudu::tools::Ksck::PrintResults() at ??:0
@ 0x7f527eb8e1a1 kudu::tools::Ksck::RunAndPrintResults() at ??:0
@ 0x388cd8
kudu::tools::RemoteKsckTest_TestNonMatchingTableFilter_Test::TestBody()::$_10::operator()()
at /root/tmp/test123/kudu/src/kudu/tools/ksck_remote-test.cc:671 @
0x388c01
_ZNSt3__18__invokeIRZN4kudu5tools46RemoteKsckTest_TestNonMatchingTableFilter_Test8TestBodyEvE4$_10JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_
at
/root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899
@ 0x388b61 std::__1::__invoke_void_return_wrapper<>::__call<>()
at
/root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348
@ 0x388b01 std::__1::__function::__alloc_func<>::operator()() at
/root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557
@ 0x386e30 std::__1::__function::__func<>::operator()() at
/root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731
@ 0x7f527f277417 std::__1::__function::__value_func<>::operator()() at
??:0
@ 0x7f527f277129 std::__1::function<>::operator()() at ??:0
W20241218 13:05:06.008596 14053 outbound_call.cc:321] RPC callback for RPC call
kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.214.66:36001,
user_credentials={real_user=slave}} blocked reactor thread for 63952.8us
@ 0x7f527f1a4920 kudu::AssertEventually() at ??:0
@ 0x36b1a5
kudu::tools::RemoteKsckTest_TestNonMatchingTableFilter_Test::TestBody() at
/root/tmp/test123/kudu/src/kudu/tools/ksck_remote-test.cc:671
@ 0x7f527ed34df0
testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f527ed129f2 testing::Test::Run() at ??:0
@ 0x7f527ed13d09 testing::TestInfo::Run() at ??:0
@ 0x7f527ed14ce5 testing::TestSuite::Run() at ??:0
@ 0x7f527ed297d5 testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f527ed35cb0
testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f527ed28d8d testing::UnitTest::Run() at ??:0
@ 0x3ca0b8 RUN_ALL_TESTS() at
/root/tmp/test123/kudu/thirdparty/installed/tsan/include/gtest/gtest.h:2293
@ 0x3c7f4b main at
/root/tmp/test123/kudu/src/kudu/util/test_main.cc:109
{code}
> check error in ksck_results.cc in TSAN build
> --------------------------------------------
>
> Key: KUDU-3634
> URL: https://issues.apache.org/jira/browse/KUDU-3634
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Bakai Ádám
> Priority: Major
>
> {code:java}
> F20241218 13:05:04.229065 4953 ksck_results.cc:160] Check failed:
> servers.size() <= server_count (6 vs. 3)
> *** Check failure stack trace: ***
> *** Aborted at 1734527104 (unix time) try "date -d @1734527104" if you are
> using GNU date ***
> PC: @ 0x0 (unknown)
> *** SIGABRT (@0x3e800001359) received by PID 4953 (TID 0x7f5270649880) from
> PID 4953; stack trace: ***
> @ 0x2d8140 __tsan::CallUserSignalHandler() at
> /root/tmp/test123/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
> @ 0x2da623 rtl_sigaction() at
> /root/tmp/test123/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
> I20241218 13:05:04.427536 14086 raft_consensus.cc:1270] T
> 5d3037cff4d843b0a7dbc9ff78a339ba P 3e6bbc8984eb4fbfbefd27d5036db51a [term 5
> FOLLOWER]: Refusing update from remote peer 1a37ef9543954c8e823eeae2d8592107:
> Log matching property violated. Preceding OpId in replica: term: 4 index: 3.
> Preceding OpId from leader: term: 5 index: 4. (index mismatch)
> I20241218 13:05:04.429109 14549 consensus_queue.cc:1035] T
> 5d3037cff4d843b0a7dbc9ff78a339ba P 1a37ef9543954c8e823eeae2d8592107 [LEADER]:
> Connected to new peer: Peer: permanent_uuid:
> "3e6bbc8984eb4fbfbefd27d5036db51a" member_type: VOTER last_known_addr { host:
> "127.4.214.67" port: 37251 }, Status: LMP_MISMATCH, Last received: 0.0, Next
> index: 4, Last known committed idx: 3, Time since last communication: 0.000s
> @ 0x7f527808f980 (unknown) at ??:0
> I20241218 13:05:04.509820 14545 raft_consensus.cc:491] T
> 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5
> FOLLOWER]: Starting pre-election (no leader contacted us within the election
> timeout)
> I20241218 13:05:04.510313 14545 raft_consensus.cc:513] T
> 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5
> FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local:
> false peers { permanent_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" member_type:
> VOTER last_known_addr { host: "127.4.214.65" port: 36387 } } peers {
> permanent_uuid: "1a37ef9543954c8e823eeae2d8592107" member_type: VOTER
> last_known_addr { host: "127.4.214.66" port: 36001 } } peers {
> permanent_uuid: "3e6bbc8984eb4fbfbefd27d5036db51a" member_type: VOTER
> last_known_addr { host: "127.4.214.67" port: 37251 } }
> I20241218 13:05:04.513247 14001 tablet_service.cc:1812] Received
> RequestConsensusVote() RPC: tablet_id: "5d3037cff4d843b0a7dbc9ff78a339ba"
> candidate_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" candidate_term: 6
> candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader:
> false dest_uuid: "1a37ef9543954c8e823eeae2d8592107" is_pre_election: true
> I20241218 13:05:04.516223 14089 tablet_service.cc:1812] Received
> RequestConsensusVote() RPC: tablet_id: "5d3037cff4d843b0a7dbc9ff78a339ba"
> candidate_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" candidate_term: 6
> candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader:
> false dest_uuid: "3e6bbc8984eb4fbfbefd27d5036db51a" is_pre_election: true
> I20241218 13:05:04.517762 13891 leader_election.cc:304] T
> 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a
> [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost.
> Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no
> votes. yes voters: 0bfce15bbb5f4f0aa5a9e60872491f3a; no voters:
> 1a37ef9543954c8e823eeae2d8592107, 3e6bbc8984eb4fbfbefd27d5036db51a
> I20241218 13:05:04.518554 14506 raft_consensus.cc:2743] T
> 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5
> FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve
> majority
> I20241218 13:05:04.518522 14545 leader_election.cc:290] T
> 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a
> [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers
> 1a37ef9543954c8e823eeae2d8592107 (127.4.214.66:36001),
> 3e6bbc8984eb4fbfbefd27d5036db51a (127.4.214.67:37251)
> @ 0x7f5271309fb7 gsignal at ??:0
> I20241218 13:05:04.633380 13919 raft_consensus.cc:1270] T
> 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5
> FOLLOWER]: Refusing update from remote peer 1a37ef9543954c8e823eeae2d8592107:
> Log matching property violated. Preceding OpId in replica: term: 3 index: 2.
> Preceding OpId from leader: term: 5 index: 4. (index mismatch)
> I20241218 13:05:04.634791 14549 consensus_queue.cc:1035] T
> 5d3037cff4d843b0a7dbc9ff78a339ba P 1a37ef9543954c8e823eeae2d8592107 [LEADER]:
> Connected to new peer: Peer: permanent_uuid:
> "0bfce15bbb5f4f0aa5a9e60872491f3a" member_type: VOTER last_known_addr { host:
> "127.4.214.65" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next
> index: 4, Last known committed idx: 2, Time since last communication: 0.000s
> @ 0x7f527130b921 abort at ??:0
> @ 0x2de136 abort at
> /root/tmp/test123/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
> @ 0x7f5273d37d64 google::LogMessage::SendToLog() at ??:0
> @ 0x7f5273d38910 google::LogMessage::Flush() at ??:0
> @ 0x7f5273d3da4b google::LogMessageFatal::~LogMessageFatal() at ??:0
> @ 0x7f527ec9165b kudu::tools::(anonymous namespace)::ServerCsv() at
> ??:0
> @ 0x7f527ec89f4e kudu::tools::PrintTaggedFlagTable() at ??:0
> @ 0x7f527ec8415e kudu::tools::KsckResults::PrintTo() at ??:0
> @ 0x7f527eb8e0f9 kudu::tools::Ksck::PrintResults() at ??:0
> @ 0x7f527eb8e1a1 kudu::tools::Ksck::RunAndPrintResults() at ??:0
> @ 0x388cd8
> kudu::tools::RemoteKsckTest_TestNonMatchingTableFilter_Test::TestBody()::$_10::operator()()
> at /root/tmp/test123/kudu/src/kudu/tools/ksck_remote-test.cc:671 @
> 0x388c01
> _ZNSt3__18__invokeIRZN4kudu5tools46RemoteKsckTest_TestNonMatchingTableFilter_Test8TestBodyEvE4$_10JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_
> at
> /root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899
> @ 0x388b61 std::__1::__invoke_void_return_wrapper<>::__call<>()
> at
> /root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348
> @ 0x388b01 std::__1::__function::__alloc_func<>::operator()()
> at
> /root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557
> @ 0x386e30 std::__1::__function::__func<>::operator()() at
> /root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731
> @ 0x7f527f277417 std::__1::__function::__value_func<>::operator()()
> at ??:0
> @ 0x7f527f277129 std::__1::function<>::operator()() at ??:0
> W20241218 13:05:06.008596 14053 outbound_call.cc:321] RPC callback for RPC
> call kudu.consensus.ConsensusService.UpdateConsensus ->
> {remote=127.4.214.66:36001, user_credentials={real_user=slave}} blocked
> reactor thread for 63952.8us
> @ 0x7f527f1a4920 kudu::AssertEventually() at ??:0
> @ 0x36b1a5
> kudu::tools::RemoteKsckTest_TestNonMatchingTableFilter_Test::TestBody() at
> /root/tmp/test123/kudu/src/kudu/tools/ksck_remote-test.cc:671
> @ 0x7f527ed34df0
> testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
> @ 0x7f527ed129f2 testing::Test::Run() at ??:0
> @ 0x7f527ed13d09 testing::TestInfo::Run() at ??:0
> @ 0x7f527ed14ce5 testing::TestSuite::Run() at ??:0
> @ 0x7f527ed297d5 testing::internal::UnitTestImpl::RunAllTests() at
> ??:0
> @ 0x7f527ed35cb0
> testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
> @ 0x7f527ed28d8d testing::UnitTest::Run() at ??:0
> @ 0x3ca0b8 RUN_ALL_TESTS() at
> /root/tmp/test123/kudu/thirdparty/installed/tsan/include/gtest/gtest.h:2293
> @ 0x3c7f4b main at
> /root/tmp/test123/kudu/src/kudu/util/test_main.cc:109
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
