Hello Matthew Jacobs, Adar Dembo, Alexey Serbin, I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/5332 to review the following change. Change subject: client: throttle some warning messages when lookup RPCs fail ...................................................................... client: throttle some warning messages when lookup RPCs fail In an Impala stress workload, we found that LookupRpc::SendRpcCb was spewing warnings at an alarming rate. The warnings themselves were causing so much load (due to Impala's glog lock contention) that it produced a vicious cycle: the log lock contention caused more timeouts, which caused more warnings, which caused more lock contention, etc... I figured out the address of the glog mutex in an impalad process and used perf to trace the stack trace of its lock acquisitions over a 5-second period: [todd@ve1120 kudu]$ sudo perf record -a -e syscalls:sys_enter_futex --filter 'uaddr == "0x7f6950a7714c"' -g sleep 5 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 4.191 MB perf.data (~183118 samples) ] [todd@ve1120 kudu]$ sudo perf script | grep -A1 'LogMess' | grep client:: | sort | uniq -c 272 7f6950477224 kudu::client::internal::WriteRpc::Finish(kudu::Status const&) 92 7f695049be85 kudu::client::internal::RemoteTablet::MarkReplicaFailed(kudu::client::internal::RemoteTabletServer*, kudu::Status const&) 14453 7f69504a1bc5 kudu::client::internal::LookupRpc::SendRpcCb(kudu::Status const&) Here we can see the vast majority came from LookupRpc::SendRpcCb and WriteRpc::Finish, so this patch just addresses those warnings with a 1-second throttle. Change-Id: I840bd57976ccefba4453667f82c7aa32756922d3 --- M src/kudu/client/batcher.cc M src/kudu/client/meta_cache.cc 2 files changed, 6 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/32/5332/1 -- To view, visit http://gerrit.cloudera.org:8080/5332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I840bd57976ccefba4453667f82c7aa32756922d3 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com> Gerrit-Reviewer: Matthew Jacobs <m...@cloudera.com>