Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9375 )
Change subject: KUDU-2297 (part 5): dump stacks into diagnostics log on service queue overflow ...................................................................... KUDU-2297 (part 5): dump stacks into diagnostics log on service queue overflow In the past we've had several bugs and performance issues which manifest themselves as service queue overflows. It would be easier to pinpoint the root cause (eg slow IO vs lock contention) if we had a process-wide stack snapshot at the time of the overflow. This patch does some plumbing to trigger such a stack trace into the diagnostics log when a service queue overflow occurs. The logging is throttled to once every 5 seconds so that we don't contribute too much to the load itself. The plumbing is a bit ugly since the diagnostics log is in the server/ module whereas the overflows surface in the rpc/ module. Having rpc/ call back into server/ would be a cyclic dependency, so instead the rpc/ module exposes a std::function<> style callback for the server to hook into. Change-Id: I62019c71121d7ca4037cab86a7c2ea048a261ad8 Reviewed-on: http://gerrit.cloudera.org:8080/9375 Tested-by: Kudu Jenkins Reviewed-by: Adar Dembo <a...@cloudera.com> --- M src/kudu/master/master-test.cc M src/kudu/master/mini_master.h M src/kudu/rpc/service_pool.cc M src/kudu/rpc/service_pool.h M src/kudu/server/diagnostics_log.cc M src/kudu/server/diagnostics_log.h M src/kudu/server/rpc_server.cc M src/kudu/server/rpc_server.h M src/kudu/server/server_base.cc M src/kudu/server/server_base.h 10 files changed, 185 insertions(+), 19 deletions(-) Approvals: Kudu Jenkins: Verified Adar Dembo: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/9375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I62019c71121d7ca4037cab86a7c2ea048a261ad8 Gerrit-Change-Number: 9375 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy <mpe...@apache.org> Gerrit-Reviewer: Sailesh Mukil <sail...@cloudera.com> Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <t...@apache.org>