Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/14717 )
Change subject: IMPALA-9134: pipeline data stream sender flush
......................................................................
IMPALA-9134: pipeline data stream sender flush
This changes the logic so that the last TransmitData
and EndDataStream RPCs can be sent to receivers in
parallel, instead of them being serialized in FlushAndSendEos().
This was a problem because each call to FlushAndSendEos() would
wait for one or two network round-trips, so previously the flush
could easily take sum(2 * RTT). After this change the sending
and waiting for RPCs is pipeline so the time taken should be
closer to 2 * RTT.
Testing:
This code is exercised by existing tests. Ran exhaustive tests.
Perf:
No measurable change on local TPC-H (where network latency isn't a
significant factor).
+----------+-----------------------+---------+------------+------------+----------------+
| Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) |
Delta(GeoMean) |
+----------+-----------------------+---------+------------+------------+----------------+
| TPCH(30) | parquet / none / none | 7.64 | -1.37% | 5.00 | +0.29%
|
+----------+-----------------------+---------+------------+------------+----------------+
+----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+
| Workload | Query | File Format | Avg(s) | Base Avg(s) |
Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval |
Tval |
+----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+
| TPCH(30) | TPCH-Q14 | parquet / none / none | 3.75 | 3.39 |
+10.48% | * 19.43% * | 8.88% | 7 | +7.26% | 0.48 |
1.19 |
| TPCH(30) | TPCH-Q13 | parquet / none / none | 13.55 | 12.78 | +6.05%
| 5.43% | 4.75% | 7 | +5.09% | 1.76 | 2.14 |
| TPCH(30) | TPCH-Q2 | parquet / none / none | 1.30 | 1.26 | +3.63%
| 5.53% | 3.27% | 7 | +3.93% | 1.28 | 1.46 |
| TPCH(30) | TPCH-Q12 | parquet / none / none | 2.65 | 2.56 | +3.44%
| 4.23% | 2.92% | 7 | +3.68% | 1.28 | 1.73 |
| TPCH(30) | TPCH-Q15 | parquet / none / none | 3.80 | 3.76 | +0.91%
| 0.81% | 1.80% | 7 | +1.42% | 1.12 | 1.22 |
| TPCH(30) | TPCH-Q8 | parquet / none / none | 5.23 | 5.13 | +2.11%
| 5.41% | 0.48% | 7 | +0.04% | 0.16 | 1.01 |
| TPCH(30) | TPCH-Q10 | parquet / none / none | 5.77 | 5.62 | +2.70%
| 9.84% | 2.79% | 7 | -0.80% | -0.96 | 0.68 |
| TPCH(30) | TPCH-Q21 | parquet / none / none | 28.82 | 28.62 | +0.70%
| 0.59% | 0.37% | 7 | +0.68% | 1.92 | 2.67 |
| TPCH(30) | TPCH-Q6 | parquet / none / none | 1.32 | 1.31 | +0.82%
| 0.36% | 1.53% | 7 | +0.23% | 1.28 | 1.38 |
| TPCH(30) | TPCH-Q11 | parquet / none / none | 1.30 | 1.29 | +0.60%
| 5.11% | 2.94% | 7 | +0.14% | 0.16 | 0.27 |
| TPCH(30) | TPCH-Q19 | parquet / none / none | 4.52 | 4.49 | +0.57%
| 2.98% | 1.58% | 7 | +0.12% | 0.32 | 0.44 |
| TPCH(30) | TPCH-Q16 | parquet / none / none | 2.78 | 2.77 | +0.35%
| 3.21% | 2.02% | 7 | +0.17% | 0.64 | 0.25 |
| TPCH(30) | TPCH-Q3 | parquet / none / none | 5.65 | 5.65 | -0.04%
| 1.62% | 1.91% | 7 | -0.06% | -0.32 | -0.04 |
| TPCH(30) | TPCH-Q1 | parquet / none / none | 10.88 | 10.89 | -0.11%
| 0.76% | 0.90% | 7 | +0.01% | 0.00 | -0.26 |
| TPCH(30) | TPCH-Q4 | parquet / none / none | 2.18 | 2.18 | -0.38%
| 1.33% | 2.48% | 7 | -0.04% | -0.16 | -0.36 |
| TPCH(30) | TPCH-Q5 | parquet / none / none | 3.94 | 3.95 | -0.27%
| 2.37% | 1.33% | 7 | -0.26% | -1.12 | -0.26 |
| TPCH(30) | TPCH-Q7 | parquet / none / none | 22.37 | 22.41 | -0.19%
| 2.94% | 1.92% | 7 | -0.66% | -0.80 | -0.15 |
| TPCH(30) | TPCH-Q22 | parquet / none / none | 3.68 | 3.72 | -1.14%
| 4.95% | 3.99% | 7 | +0.13% | 0.00 | -0.48 |
| TPCH(30) | TPCH-Q20 | parquet / none / none | 2.83 | 2.86 | -1.21%
| 2.69% | 2.20% | 7 | -1.60% | -0.64 | -0.93 |
| TPCH(30) | TPCH-Q17 | parquet / none / none | 4.92 | 5.04 | -2.46%
| 3.11% | 2.63% | 7 | -2.00% | -1.44 | -1.62 |
| TPCH(30) | TPCH-Q9 | parquet / none / none | 16.32 | 17.19 | -5.03%
| 2.76% | 4.56% | 7 | -4.00% | -2.24 | -2.53 |
| TPCH(30) | TPCH-Q18 | parquet / none / none | 20.45 | 23.45 |
-12.82% | 5.49% | * 23.11% * | 7 | -4.89% | -0.96 |
-1.44 |
+----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+
Change-Id: If66add99e4370f8faf22761b336205cc9f9f1867
Reviewed-on: http://gerrit.cloudera.org:8080/14717
Reviewed-by: Tim Armstrong <tarmstr...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
---
M be/src/runtime/krpc-data-stream-sender.cc
1 file changed, 62 insertions(+), 30 deletions(-)
Approvals:
Tim Armstrong: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/14717
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If66add99e4370f8faf22761b336205cc9f9f1867
Gerrit-Change-Number: 14717
Gerrit-PatchSet: 6
Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Michael Ho <k...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tmarsh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
