Hello Kurt Deschler, Daniel Becker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20462 to look at the new patch set (#6). Change subject: IMPALA-12430: Skip compression when sending row batches within same process ...................................................................... IMPALA-12430: Skip compression when sending row batches within same process LZ4 compression doesn't seem useful when the RowBatch is sent to a fragment instance within the same process instead of a remote host. After this change KrpcDataStreamSender skips compression for channels where the destination is in the same process. Other changes: - OutboundRowBatch is moved to a separate file to make the commonly included row-batch.h lighter. - TestObservability.test_global_exchange_counters had to be changed as skipping compression changed metric ExchangeScanRatio. Also added a sleep to the test query because it was flaky on my machine (it doesn't seem flaky in jenkins runs, probably my CPU is faster). See the Jira for more details on tasks that could be skipped in intra process RowBatch transfer. From these compression is both the most expensive and easiest to avoid. Note that it may also make sense to skip compression if the target is not the in same process but resides on the same host. This setup is not typical in production environment AFAIK and it would complicate testing compression as impalad processes often run on the same host during tests. For these reasons it seems better to only implement this if both the host and port are the same. TPCH benchmark shows significant improvement but it uses only 3 impalad processes so 1/3 of exchanges are affected - in bigger clusters the change should be much smaller. +----------+-----------------------+---------+------------+------------+----------------+ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +----------+-----------------------+---------+------------+------------+----------------+ | TPCH(42) | parquet / none / none | 3.59 | -4.95% | 2.37 | -2.51% | +----------+-----------------------+---------+------------+------------+----------------+ Change-Id: I7ea23fd1f0f10f72f3dbd8594f3def3ee190230a --- M be/src/benchmarks/row-batch-serialize-benchmark.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h A be/src/runtime/outbound-row-batch.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h M tests/query_test/test_observability.py 8 files changed, 128 insertions(+), 65 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/20462/6 -- To view, visit http://gerrit.cloudera.org:8080/20462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7ea23fd1f0f10f72f3dbd8594f3def3ee190230a Gerrit-Change-Number: 20462 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com>