Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14717 )
Change subject: IMPALA-9134: pipeline data stream sender flush ...................................................................... IMPALA-9134: pipeline data stream sender flush This changes the logic so that the last TransmitData and EndDataStream RPCs can be sent to receivers in parallel, instead of them being serialized in FlushAndSendEos(). This was a problem because each call to FlushAndSendEos() would wait for one or two network round-trips, so previously the flush could easily take sum(2 * RTT). After this change the sending and waiting for RPCs is pipeline so the time taken should be closer to 2 * RTT. Testing: This code is exercised by existing tests. Ran exhaustive tests. Perf: No measurable change on local TPC-H (where network latency isn't a significant factor). +----------+-----------------------+---------+------------+------------+----------------+ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +----------+-----------------------+---------+------------+------------+----------------+ | TPCH(30) | parquet / none / none | 7.64 | -1.37% | 5.00 | +0.29% | +----------+-----------------------+---------+------------+------------+----------------+ +----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+ | Workload | Query | File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval | +----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+ | TPCH(30) | TPCH-Q14 | parquet / none / none | 3.75 | 3.39 | +10.48% | * 19.43% * | 8.88% | 7 | +7.26% | 0.48 | 1.19 | | TPCH(30) | TPCH-Q13 | parquet / none / none | 13.55 | 12.78 | +6.05% | 5.43% | 4.75% | 7 | +5.09% | 1.76 | 2.14 | | TPCH(30) | TPCH-Q2 | parquet / none / none | 1.30 | 1.26 | +3.63% | 5.53% | 3.27% | 7 | +3.93% | 1.28 | 1.46 | | TPCH(30) | TPCH-Q12 | parquet / none / none | 2.65 | 2.56 | +3.44% | 4.23% | 2.92% | 7 | +3.68% | 1.28 | 1.73 | | TPCH(30) | TPCH-Q15 | parquet / none / none | 3.80 | 3.76 | +0.91% | 0.81% | 1.80% | 7 | +1.42% | 1.12 | 1.22 | | TPCH(30) | TPCH-Q8 | parquet / none / none | 5.23 | 5.13 | +2.11% | 5.41% | 0.48% | 7 | +0.04% | 0.16 | 1.01 | | TPCH(30) | TPCH-Q10 | parquet / none / none | 5.77 | 5.62 | +2.70% | 9.84% | 2.79% | 7 | -0.80% | -0.96 | 0.68 | | TPCH(30) | TPCH-Q21 | parquet / none / none | 28.82 | 28.62 | +0.70% | 0.59% | 0.37% | 7 | +0.68% | 1.92 | 2.67 | | TPCH(30) | TPCH-Q6 | parquet / none / none | 1.32 | 1.31 | +0.82% | 0.36% | 1.53% | 7 | +0.23% | 1.28 | 1.38 | | TPCH(30) | TPCH-Q11 | parquet / none / none | 1.30 | 1.29 | +0.60% | 5.11% | 2.94% | 7 | +0.14% | 0.16 | 0.27 | | TPCH(30) | TPCH-Q19 | parquet / none / none | 4.52 | 4.49 | +0.57% | 2.98% | 1.58% | 7 | +0.12% | 0.32 | 0.44 | | TPCH(30) | TPCH-Q16 | parquet / none / none | 2.78 | 2.77 | +0.35% | 3.21% | 2.02% | 7 | +0.17% | 0.64 | 0.25 | | TPCH(30) | TPCH-Q3 | parquet / none / none | 5.65 | 5.65 | -0.04% | 1.62% | 1.91% | 7 | -0.06% | -0.32 | -0.04 | | TPCH(30) | TPCH-Q1 | parquet / none / none | 10.88 | 10.89 | -0.11% | 0.76% | 0.90% | 7 | +0.01% | 0.00 | -0.26 | | TPCH(30) | TPCH-Q4 | parquet / none / none | 2.18 | 2.18 | -0.38% | 1.33% | 2.48% | 7 | -0.04% | -0.16 | -0.36 | | TPCH(30) | TPCH-Q5 | parquet / none / none | 3.94 | 3.95 | -0.27% | 2.37% | 1.33% | 7 | -0.26% | -1.12 | -0.26 | | TPCH(30) | TPCH-Q7 | parquet / none / none | 22.37 | 22.41 | -0.19% | 2.94% | 1.92% | 7 | -0.66% | -0.80 | -0.15 | | TPCH(30) | TPCH-Q22 | parquet / none / none | 3.68 | 3.72 | -1.14% | 4.95% | 3.99% | 7 | +0.13% | 0.00 | -0.48 | | TPCH(30) | TPCH-Q20 | parquet / none / none | 2.83 | 2.86 | -1.21% | 2.69% | 2.20% | 7 | -1.60% | -0.64 | -0.93 | | TPCH(30) | TPCH-Q17 | parquet / none / none | 4.92 | 5.04 | -2.46% | 3.11% | 2.63% | 7 | -2.00% | -1.44 | -1.62 | | TPCH(30) | TPCH-Q9 | parquet / none / none | 16.32 | 17.19 | -5.03% | 2.76% | 4.56% | 7 | -4.00% | -2.24 | -2.53 | | TPCH(30) | TPCH-Q18 | parquet / none / none | 20.45 | 23.45 | -12.82% | 5.49% | * 23.11% * | 7 | -4.89% | -0.96 | -1.44 | +----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+ Change-Id: If66add99e4370f8faf22761b336205cc9f9f1867 Reviewed-on: http://gerrit.cloudera.org:8080/14717 Reviewed-by: Tim Armstrong <tarmstr...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/runtime/krpc-data-stream-sender.cc 1 file changed, 62 insertions(+), 30 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14717 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If66add99e4370f8faf22761b336205cc9f9f1867 Gerrit-Change-Number: 14717 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Michael Ho <k...@cloudera.com> Gerrit-Reviewer: Thomas Tauber-Marshall <tmarsh...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>