Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/15863 )
Change subject: IMPALA-9712: fix mem consumption of operators above selective
scan
......................................................................
IMPALA-9712: fix mem consumption of operators above selective scan
This change is motivated by excessive memory consumption of
TPC-H Q19 which has a hash join and non-grouping aggregate
above a selective scan.
This change fixes RowBatch::AtCapacity() to account for the actual
memory consumed by the RowBatch. It used total_allocated_bytes(),
which does *not* account for unused space in the MemPool chunks.
Instead it now uses total_reserved_bytes(), which includes the
whole chunks. This reduced memory consumption of the agg from
60+MB to ~16MB.
Testing:
Ran TPC-H Q19 on parquet with mt_dop=8. Aggregation mem usage was
reduced from 60+MB to ~16MB.
Added a targeted regression test that ran out of memory before this
fix.
Ran exhaustive tests.
Performance:
No significant change on TPC-H single node run with scale factor 30.
I also ran TPC-H nested scale factor 1 and there was no measureable
change, but generation of the report failed for some reason.
+----------+-----------------------+---------+------------+------------+----------------+
| Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) |
Delta(GeoMean) |
+----------+-----------------------+---------+------------+------------+----------------+
| TPCH(30) | parquet / none / none | 6.18 | -1.31% | 4.54 | -1.03%
|
+----------+-----------------------+---------+------------+------------+----------------+
+----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+
| Workload | Query | File Format | Avg(s) | Base Avg(s) |
Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval |
Tval |
+----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+
| TPCH(30) | TPCH-Q10 | parquet / none / none | 8.08 | 7.98 | +1.19%
| * 14.00% * | * 10.55% * | 20 | +0.63% | 0.60 | 0.30 |
| TPCH(30) | TPCH-Q13 | parquet / none / none | 10.03 | 9.98 | +0.44%
| 1.22% | 0.92% | 20 | +0.48% | 1.07 | 1.28 |
| TPCH(30) | TPCH-Q12 | parquet / none / none | 3.20 | 3.19 | +0.34%
| 2.02% | 2.68% | 20 | +0.08% | 0.48 | 0.45 |
| TPCH(30) | TPCH-Q21 | parquet / none / none | 22.60 | 22.54 | +0.24%
| 2.85% | 2.80% | 20 | +0.17% | 0.22 | 0.27 |
| TPCH(30) | TPCH-Q9 | parquet / none / none | 13.80 | 13.77 | +0.17%
| 1.99% | 1.70% | 20 | +0.06% | 0.10 | 0.30 |
| TPCH(30) | TPCH-Q7 | parquet / none / none | 4.52 | 4.52 | -0.01%
| 1.68% | 1.71% | 20 | +0.03% | 0.07 | -0.01 |
| TPCH(30) | TPCH-Q8 | parquet / none / none | 5.40 | 5.43 | -0.52%
| 1.60% | 1.92% | 20 | -0.23% | -0.98 | -0.94 |
| TPCH(30) | TPCH-Q16 | parquet / none / none | 1.82 | 1.83 | -0.63%
| 2.98% | 2.62% | 20 | -0.17% | -1.07 | -0.71 |
| TPCH(30) | TPCH-Q15 | parquet / none / none | 3.80 | 3.82 | -0.46%
| 1.31% | 1.21% | 20 | -0.41% | -1.30 | -1.17 |
| TPCH(30) | TPCH-Q4 | parquet / none / none | 2.72 | 2.74 | -0.68%
| 3.06% | 2.36% | 20 | -0.23% | -0.92 | -0.79 |
| TPCH(30) | TPCH-Q6 | parquet / none / none | 1.63 | 1.64 | -0.98%
| 3.28% | 2.66% | 20 | -0.23% | -1.47 | -1.04 |
| TPCH(30) | TPCH-Q22 | parquet / none / none | 1.99 | 2.01 | -0.98%
| 2.59% | 2.99% | 20 | -0.68% | -1.12 | -1.12 |
| TPCH(30) | TPCH-Q1 | parquet / none / none | 5.22 | 5.27 | -0.96%
| 1.93% | 2.25% | 20 | -0.93% | -1.39 | -1.45 |
| TPCH(30) | TPCH-Q14 | parquet / none / none | 3.61 | 3.64 | -0.73%
| 2.51% | 2.40% | 20 | -1.26% | -1.18 | -0.95 |
| TPCH(30) | TPCH-Q5 | parquet / none / none | 4.52 | 4.58 | -1.23%
| 1.41% | 1.39% | 20 | -1.15% | -2.99 | -2.79 |
| TPCH(30) | TPCH-Q19 | parquet / none / none | 4.05 | 4.10 | -1.19%
| 2.58% | 2.48% | 20 | -1.27% | -1.77 | -1.49 |
| TPCH(30) | TPCH-Q3 | parquet / none / none | 3.96 | 4.01 | -1.32%
| 1.82% | 1.82% | 20 | -1.31% | -2.09 | -2.30 |
| TPCH(30) | TPCH-Q20 | parquet / none / none | 2.93 | 2.98 | -1.47%
| 2.16% | 2.60% | 20 | -1.63% | -1.88 | -1.95 |
| TPCH(30) | TPCH-Q2 | parquet / none / none | 2.78 | 2.83 | -1.70%
| 2.29% | 2.19% | 20 | -1.77% | -1.97 | -2.41 |
| TPCH(30) | TPCH-Q11 | parquet / none / none | 1.09 | 1.12 | -2.64%
| 2.99% | 3.81% | 20 | -3.71% | -1.77 | -2.46 |
| TPCH(30) | TPCH-Q17 | parquet / none / none | 15.07 | 15.73 | -4.20%
| 3.24% | 2.26% | 20 | -5.00% | -3.84 | -4.89 |
| TPCH(30) | TPCH-Q18 | parquet / none / none | 13.23 | 14.13 | -6.38%
| * 11.87% * | * 26.04% * | 20 | -3.19% | -2.41 | -1.01 |
+----------+----------+-----------------------+--------+-------------+------------+------------+----------------+-------+----------------+---------+-------+
Change-Id: I6debae562826621411bbcbb757978e227b395441
Reviewed-on: http://gerrit.cloudera.org:8080/15863
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
---
M be/src/runtime/row-batch.h
M tests/query_test/test_mem_usage_scaling.py
2 files changed, 28 insertions(+), 1 deletion(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/15863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6debae562826621411bbcbb757978e227b395441
Gerrit-Change-Number: 15863
Gerrit-PatchSet: 8
Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tmarsh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
