Hello Lars Volker, Tim Armstrong, Dan Hecht, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/9282 to look at the new patch set (#2). Change subject: IMPALA-6116: Bound memory usage of DataStreamSevice's service queue ...................................................................... IMPALA-6116: Bound memory usage of DataStreamSevice's service queue The fix for IMPALA-6193 added a memory tracker for the memory consumed by the payloads in the service queue of DataStreamService. This change extends it by introducing a bound on the memory usage for that service queue. In addition, it deprecates FLAGS_datastream_service_queue_depth and replaces it with FLAGS_datastream_service_queue_max_mem. These flags only take effect when KRPC is in use and KRPC was never enabled in any previous releases so it seems safe to do this flag replacement. The new flag FLAGS_datastream_service_queue_max_mem directly dictates the amount of memory which can be consumed by the service queue of DataStreamService. This allows a more direct control over the memory usage of the queue instead of inferring via the number of entries in the queue. The default value of this flag is left at 0, in which case it will be set to 20% of process memory limit. Testing done: exhaustive debug builds. Updated data-stream-test to exercise the case in which the payload is larger than the limit. Change-Id: Idea4262dfb0e0aa8d58ff6ea6a8aaaa248e880b9 --- M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-test-base.h M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-mgr.h M be/src/runtime/mem-tracker.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M tests/custom_cluster/test_krpc_mem_usage.py 15 files changed, 282 insertions(+), 164 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/9282/2 -- To view, visit http://gerrit.cloudera.org:8080/9282 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idea4262dfb0e0aa8d58ff6ea6a8aaaa248e880b9 Gerrit-Change-Number: 9282 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho <k...@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Lars Volker <l...@cloudera.com> Gerrit-Reviewer: Michael Ho <k...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>