Hello Matthew Jacobs, Adar Dembo, Alexey Serbin,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/5332
to review the following change.
Change subject: client: throttle some warning messages when lookup RPCs fail
......................................................................
client: throttle some warning messages when lookup RPCs fail
In an Impala stress workload, we found that LookupRpc::SendRpcCb was
spewing warnings at an alarming rate. The warnings themselves were
causing so much load (due to Impala's glog lock contention) that it
produced a vicious cycle: the log lock contention caused more timeouts,
which caused more warnings, which caused more lock contention, etc...
I figured out the address of the glog mutex in an impalad process and
used perf to trace the stack trace of its lock acquisitions over a
5-second period:
[todd@ve1120 kudu]$ sudo perf record -a -e syscalls:sys_enter_futex --filter
'uaddr == "0x7f6950a7714c"' -g sleep 5
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 4.191 MB perf.data (~183118 samples) ]
[todd@ve1120 kudu]$ sudo perf script | grep -A1 'LogMess' | grep client:: |
sort | uniq -c
272 7f6950477224
kudu::client::internal::WriteRpc::Finish(kudu::Status const&)
92 7f695049be85
kudu::client::internal::RemoteTablet::MarkReplicaFailed(kudu::client::internal::RemoteTabletServer*,
kudu::Status const&)
14453 7f69504a1bc5
kudu::client::internal::LookupRpc::SendRpcCb(kudu::Status const&)
Here we can see the vast majority came from LookupRpc::SendRpcCb and
WriteRpc::Finish, so this patch just addresses those warnings with a 1-second
throttle.
Change-Id: I840bd57976ccefba4453667f82c7aa32756922d3
---
M src/kudu/client/batcher.cc
M src/kudu/client/meta_cache.cc
2 files changed, 6 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/32/5332/1
--
To view, visit http://gerrit.cloudera.org:8080/5332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I840bd57976ccefba4453667f82c7aa32756922d3
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <m...@cloudera.com>
