Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10573
Change subject: IMPALA-7078: Part 2: reduce queue size based on num_scanner_threads ...................................................................... IMPALA-7078: Part 2: reduce queue size based on num_scanner_threads This gives a tighter bound on memory consumption when running with a lower num_scanner_threads value. Cap the maximum row batch queue size at 5 * max_num_scanner_threads_ so that we guarantee lower amounts of memory in the row batch queue when num_scanner_threads is set, rather than just achieving it statistically because of the producer running slower relative to consumer. It does not reduce the default significantly on typical server configurations that would have 24+ cores except under high concurrency or low memory environments where the number of scanner threads is limited. We should evaluate reducing the default further or otherwise better controlling memory consumption in a follow-up, based on experiments. Testing: Tested along with Part 1. Change-Id: I8c0bd14e6bcd9fc1655e344a5307ea0eb4600e8b --- M be/src/exec/hdfs-scan-node.cc 1 file changed, 19 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/10573/1 -- To view, visit http://gerrit.cloudera.org:8080/10573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8c0bd14e6bcd9fc1655e344a5307ea0eb4600e8b Gerrit-Change-Number: 10573 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com>
