lipeng...@sensorsdata.cn has uploaded this change for review. (
http://gerrit.cloudera.org:8080/19114
Change subject: IMPALA-11607: Support for TPC-H and TPC-DS test datasets stored
as Iceberg tables
......................................................................
IMPALA-11607: Support for TPC-H and TPC-DS test datasets stored as Iceberg
tables
Impala 4.1.0 has officially supported Iceberg tables, but we don't have
tools to generate TPC-H and TPC-DS test datasets for Iceberg tables. We
should use load-data.py to generate test datasets, e.g:
"${IMPALA_HOME}/bin/load-data.py -w tpcds -f -s 10
--table_formats=text/none,iceberg/none".
Testing:
- load-data.py -w tpch -s 1 --table_formats=text/none,parquet/none,
iceberg/none -e exhaustive -f
- load-data.py -w tpcds -s 1 --table_formats=text/none,parquet/zstd,
iceberg/zstd -e exhaustive -f
- single_node_perf_run.py --workloads=tpch --scale=30 --iterations=3 --
table_formats=parquet/lz4,iceberg/lz4 --num_impalads=1 --load 3ee3dc847
Workload: TPCH, Scale Factor: 30
Table Format: parquet/lz4/block
+----------+----------------+-----------+
| Query | Time Taken (s) | Client ID |
+----------+----------------+-----------+
| TPCH-Q1 | 7.73 | 1 |
| TPCH-Q1 | 7.50 | 1 |
| TPCH-Q1 | 7.55 | 1 |
| TPCH-Q10 | 3.65 | 1 |
| TPCH-Q10 | 3.70 | 1 |
| TPCH-Q10 | 3.69 | 1 |
| TPCH-Q11 | 0.68 | 1 |
| TPCH-Q11 | 0.68 | 1 |
| TPCH-Q11 | 0.67 | 1 |
| TPCH-Q12 | 1.33 | 1 |
| TPCH-Q12 | 1.08 | 1 |
| TPCH-Q12 | 1.08 | 1 |
| TPCH-Q13 | 10.63 | 1 |
| TPCH-Q13 | 10.36 | 1 |
| TPCH-Q13 | 10.52 | 1 |
| TPCH-Q14 | 1.63 | 1 |
| TPCH-Q14 | 1.79 | 1 |
| TPCH-Q14 | 1.63 | 1 |
| TPCH-Q15 | 1.44 | 1 |
| TPCH-Q15 | 1.54 | 1 |
| TPCH-Q15 | 1.45 | 1 |
| TPCH-Q16 | 2.35 | 1 |
| TPCH-Q16 | 2.35 | 1 |
| TPCH-Q16 | 2.40 | 1 |
| TPCH-Q17 | 1.20 | 1 |
| TPCH-Q17 | 1.13 | 1 |
| TPCH-Q17 | 1.10 | 1 |
| TPCH-Q18 | 17.50 | 1 |
| TPCH-Q18 | 17.04 | 1 |
| TPCH-Q18 | 14.43 | 1 |
| TPCH-Q19 | 1.61 | 1 |
| TPCH-Q19 | 1.56 | 1 |
| TPCH-Q19 | 1.56 | 1 |
| TPCH-Q2 | 0.68 | 1 |
| TPCH-Q2 | 0.68 | 1 |
| TPCH-Q2 | 0.68 | 1 |
| TPCH-Q20 | 1.09 | 1 |
| TPCH-Q20 | 1.09 | 1 |
| TPCH-Q20 | 1.15 | 1 |
| TPCH-Q21 | 24.94 | 1 |
| TPCH-Q21 | 26.42 | 1 |
| TPCH-Q21 | 25.62 | 1 |
| TPCH-Q22 | 3.69 | 1 |
| TPCH-Q22 | 3.58 | 1 |
| TPCH-Q22 | 3.24 | 1 |
| TPCH-Q3 | 3.58 | 1 |
| TPCH-Q3 | 3.68 | 1 |
| TPCH-Q3 | 3.89 | 1 |
| TPCH-Q4 | 1.37 | 1 |
| TPCH-Q4 | 1.37 | 1 |
| TPCH-Q4 | 1.32 | 1 |
| TPCH-Q5 | 1.68 | 1 |
| TPCH-Q5 | 1.69 | 1 |
| TPCH-Q5 | 1.65 | 1 |
| TPCH-Q6 | 0.46 | 1 |
| TPCH-Q6 | 0.47 | 1 |
| TPCH-Q6 | 0.47 | 1 |
| TPCH-Q7 | 2.54 | 1 |
| TPCH-Q7 | 2.49 | 1 |
| TPCH-Q7 | 2.54 | 1 |
| TPCH-Q8 | 1.90 | 1 |
| TPCH-Q8 | 1.90 | 1 |
| TPCH-Q8 | 1.90 | 1 |
| TPCH-Q9 | 12.72 | 1 |
| TPCH-Q9 | 12.02 | 1 |
| TPCH-Q9 | 12.67 | 1 |
+----------+----------------+-----------+
Table Format: iceberg/lz4/block
+----------+----------------+-----------+
| Query | Time Taken (s) | Client ID |
+----------+----------------+-----------+
| TPCH-Q1 | 7.51 | 1 |
| TPCH-Q1 | 7.61 | 1 |
| TPCH-Q1 | 7.60 | 1 |
| TPCH-Q10 | 3.79 | 1 |
| TPCH-Q10 | 3.73 | 1 |
| TPCH-Q10 | 3.67 | 1 |
| TPCH-Q11 | 0.75 | 1 |
| TPCH-Q11 | 0.88 | 1 |
| TPCH-Q11 | 0.69 | 1 |
| TPCH-Q12 | 1.18 | 1 |
| TPCH-Q12 | 1.13 | 1 |
| TPCH-Q12 | 1.04 | 1 |
| TPCH-Q13 | 10.11 | 1 |
| TPCH-Q13 | 9.76 | 1 |
| TPCH-Q13 | 10.31 | 1 |
| TPCH-Q14 | 1.64 | 1 |
| TPCH-Q14 | 1.64 | 1 |
| TPCH-Q14 | 1.53 | 1 |
| TPCH-Q15 | 1.44 | 1 |
| TPCH-Q15 | 1.39 | 1 |
| TPCH-Q15 | 1.35 | 1 |
| TPCH-Q16 | 2.24 | 1 |
| TPCH-Q16 | 2.15 | 1 |
| TPCH-Q16 | 2.15 | 1 |
| TPCH-Q17 | 1.00 | 1 |
| TPCH-Q17 | 1.00 | 1 |
| TPCH-Q17 | 0.99 | 1 |
| TPCH-Q18 | 15.14 | 1 |
| TPCH-Q18 | 14.94 | 1 |
| TPCH-Q18 | 15.04 | 1 |
| TPCH-Q19 | 1.63 | 1 |
| TPCH-Q19 | 1.66 | 1 |
| TPCH-Q19 | 1.60 | 1 |
| TPCH-Q2 | 0.69 | 1 |
| TPCH-Q2 | 0.75 | 1 |
| TPCH-Q2 | 0.69 | 1 |
| TPCH-Q20 | 1.10 | 1 |
| TPCH-Q20 | 1.09 | 1 |
| TPCH-Q20 | 1.21 | 1 |
| TPCH-Q21 | 25.38 | 1 |
| TPCH-Q21 | 25.48 | 1 |
| TPCH-Q21 | 25.18 | 1 |
| TPCH-Q22 | 3.54 | 1 |
| TPCH-Q22 | 3.59 | 1 |
| TPCH-Q22 | 3.49 | 1 |
| TPCH-Q3 | 3.95 | 1 |
| TPCH-Q3 | 3.96 | 1 |
| TPCH-Q3 | 3.79 | 1 |
| TPCH-Q4 | 1.43 | 1 |
| TPCH-Q4 | 1.43 | 1 |
| TPCH-Q4 | 1.43 | 1 |
| TPCH-Q5 | 1.84 | 1 |
| TPCH-Q5 | 1.80 | 1 |
| TPCH-Q5 | 1.71 | 1 |
| TPCH-Q6 | 0.56 | 1 |
| TPCH-Q6 | 0.47 | 1 |
| TPCH-Q6 | 0.53 | 1 |
| TPCH-Q7 | 2.75 | 1 |
| TPCH-Q7 | 2.59 | 1 |
| TPCH-Q7 | 2.49 | 1 |
| TPCH-Q8 | 1.90 | 1 |
| TPCH-Q8 | 2.00 | 1 |
| TPCH-Q8 | 1.91 | 1 |
| TPCH-Q9 | 13.34 | 1 |
| TPCH-Q9 | 12.18 | 1 |
| TPCH-Q9 | 12.71 | 1 |
+----------+----------------+-----------+
Change-Id: Ic8bbc5209e7e649f67a48144a2844b35d9f9c7f1
---
M testdata/bin/generate-schema-statements.py
M tests/common/test_dimensions.py
2 files changed, 20 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/19114/1
--
To view, visit http://gerrit.cloudera.org:8080/19114
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic8bbc5209e7e649f67a48144a2844b35d9f9c7f1
Gerrit-Change-Number: 19114
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <lipeng...@sensorsdata.cn>
