Michael Ho has uploaded a new patch set (#4). Change subject: IMPALA-4026: Implement double-buffering for BlockingQueue ......................................................................
IMPALA-4026: Implement double-buffering for BlockingQueue With recent changes to improve the parquet scanner's efficency, row batches are produced more quickly, leading to higher contention in the blocking queue shared between scanner threads and the scan node. The contention happens between different producers (i.e. the scanner threads) and also to a lesser extent, between the scanner threads and the scan node. This change addresses the contention between the scanner threads and the scan node by splitting the queue into a 'read_list_' and a 'write_list_'. The consumers will consume from 'read_list_' until it's exhausted while the producers will enqueue into 'write_list_' until it's full. When 'read_list_' is exhausted, the consumer will atomically swap the 'read_list_' with 'write_list_'. This reduces the contention/overhead: 'read_list_' and 'write_list_' are protected by two different locks so consumer only contends for the write lock when 'read_list_' is empty. Change-Id: Ib9f4cf351455efefb0f3bb791cf9bc82d1421d54 --- M be/src/common/compiler-util.h M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/util/blocking-queue.h A be/src/util/condition-variable.h 5 files changed, 184 insertions(+), 61 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/4350/4 -- To view, visit http://gerrit.cloudera.org:8080/4350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib9f4cf351455efefb0f3bb791cf9bc82d1421d54 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Michael Ho <k...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Chen Huang <paulhuan...@utexas.edu> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Michael Ho <k...@cloudera.com> Gerrit-Reviewer: Mostafa Mokhtar <mmokh...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>