Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13883 )
Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl ...................................................................... IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl Improves the encapsulation of RowBatchQueue by the doing the following re-factoring: * Renames RowBatchQueue to BlockingRowBatchQueue which is more indicitive of what the queue does * Re-factors the timers managed by the scan-node into the BlockingRowBatchQueue implementation * Favors composition over inheritance by re-factoring BlockingRowBatchQueue to own a BlockingQueue rather than extending one The re-factoring lays the groundwork for introducing a generic RowBatchQueue that all RowBatch queues inherit from. Adds a new DequeRowBatchQueue which is a simple wrapper around a std::deque that (1) stores unique_ptr to queued RowBatch-es and (2) has a maximum capacity. Implements BufferedPlanRootSink using the new DequeRowBatchQueue. DequeRowBatchQueue is generic enough that replacing it with a SpillableQueue (queue backed by a BufferedTupleStream) should be straightforward. BufferedPlanRootSink is synchronized to protect access to DequeRowBatchQueue since the queue is not thread safe. BufferedPlanRootSink FlushFinal blocks until the consumer thread has processed all RowBatches. This ensures that the coordinator fragment stays alive until all results are fetched, but allows all other fragments to be shutdown immediately. Testing: * Running core tests * Updated tests/query_test/test_result_spooling.py Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be Reviewed-on: http://gerrit.cloudera.org:8080/13883 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/blocking-plan-root-sink.h M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.h M be/src/exec/data-sink.cc M be/src/exec/hdfs-scan-node.cc M be/src/exec/kudu-scan-node.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/exec/scan-node.cc M be/src/exec/scan-node.h M be/src/exec/scanner-context.cc M be/src/runtime/CMakeLists.txt A be/src/runtime/blocking-row-batch-queue.cc A be/src/runtime/blocking-row-batch-queue.h A be/src/runtime/deque-row-batch-queue.cc A be/src/runtime/deque-row-batch-queue.h D be/src/runtime/row-batch-queue.cc D be/src/runtime/row-batch-queue.h M be/src/util/blocking-queue.h M tests/query_test/test_result_spooling.py 21 files changed, 559 insertions(+), 205 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be Gerrit-Change-Number: 13883 Gerrit-PatchSet: 20 Gerrit-Owner: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Michael Ho <k...@cloudera.com> Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>