[Impala-ASF-CR] IMPALA-10090: Create aarch64 development environment on ubuntu 18.04
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16065 ) Change subject: IMPALA-10090: Create aarch64 development environment on ubuntu 18.04 .. IMPALA-10090: Create aarch64 development environment on ubuntu 18.04 Including following changes: 1 build native-toolchain local by script on aarch64 platform 2 change some native-toolchain's lib version number 3 split SKIP_TOOLCHAIN_BOOTSTRAP and DOWNLOAD_CDH_COMPONETS to two things, because on aarch64, just need to download cdp components , but not need to download toolchain. 4 download hadoop aarch64 nativelibs , impala building needs these libs. With this commit, on ubuntu 18.04 aarch64 version, just need to run bin/bootstrap_development.sh, just like x86. Change-Id: I769668c834ab0dd504a822ed9153186778275d59 Reviewed-on: http://gerrit.cloudera.org:8080/16065 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M buildall.sh 4 files changed, 63 insertions(+), 11 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I769668c834ab0dd504a822ed9153186778275d59 Gerrit-Change-Number: 16065 Gerrit-PatchSet: 17 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10090: Create aarch64 development environment on ubuntu 18.04
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16065 ) Change subject: IMPALA-10090: Create aarch64 development environment on ubuntu 18.04 .. Patch Set 16: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769668c834ab0dd504a822ed9153186778275d59 Gerrit-Change-Number: 16065 Gerrit-PatchSet: 16 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Sep 2020 06:47:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10106: Upgrade DataSketches to version 2.1.0
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16360 ) Change subject: IMPALA-10106: Upgrade DataSketches to version 2.1.0 .. Patch Set 5: Code-Review+2 Let's not wait further and merge this patch. -- To view, visit http://gerrit.cloudera.org:8080/16360 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4faa31c0b628a62c7e56a6c4b9549d0aaa8a02ff Gerrit-Change-Number: 16360 Gerrit-PatchSet: 5 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 02 Sep 2020 05:47:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10106: Upgrade DataSketches to version 2.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16360 ) Change subject: IMPALA-10106: Upgrade DataSketches to version 2.1.0 .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6384/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16360 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4faa31c0b628a62c7e56a6c4b9549d0aaa8a02ff Gerrit-Change-Number: 16360 Gerrit-PatchSet: 5 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 02 Sep 2020 05:45:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10108: Implement ds kll stringify function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16370 ) Change subject: IMPALA-10108: Implement ds_kll_stringify function .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97f654a4838bf91e3e0bed6a00d78b2c7aa96f75 Gerrit-Change-Number: 16370 Gerrit-PatchSet: 8 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 02 Sep 2020 05:38:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10108: Implement ds kll stringify function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16370 ) Change subject: IMPALA-10108: Implement ds_kll_stringify function .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6383/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97f654a4838bf91e3e0bed6a00d78b2c7aa96f75 Gerrit-Change-Number: 16370 Gerrit-PatchSet: 8 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 02 Sep 2020 05:38:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10094: Skip test refresh updated partitions on S3
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16399 ) Change subject: IMPALA-10094: Skip test_refresh_updated_partitions on S3 .. IMPALA-10094: Skip test_refresh_updated_partitions on S3 The test test_refresh_updated_partitions runs some commands using Hive which causes it fail on S3 specific jobs since we don't run HiveServer2 in those environments. This patch skips the test on non-hdfs environments. Change-Id: I0d27dd76e772e396a07419a58821ba899ac74188 Reviewed-on: http://gerrit.cloudera.org:8080/16399 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/metadata/test_reset_metadata.py 1 file changed, 8 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0d27dd76e772e396a07419a58821ba899ac74188 Gerrit-Change-Number: 16399 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-10094: Skip test refresh updated partitions on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16399 ) Change subject: IMPALA-10094: Skip test_refresh_updated_partitions on S3 .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d27dd76e772e396a07419a58821ba899ac74188 Gerrit-Change-Number: 16399 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 02 Sep 2020 04:31:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 28: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7071/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 28 Gerrit-Owner: wangsheng Gerrit-Reviewer: Anonymous Coward (606) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 02 Sep 2020 03:41:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 28: > (1 comment) > > No worries, thanks for your persistence. Seems like something is > wrong with test_iceberg.py in the dockerised test. I'm also trying > to figure out what went wrong. > > https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3075/testReport/query_test.test_iceberg/TestCreatingIcebergTable/test_create_iceberg_tables_protocol__beeswax___exec_optionbatch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0table_format__parquet_none_/ I add file_format constrint in test_iceberg.py. But when I test single case by: impala-py.test tests/query_test/test_iceberg.py. Whenever I add file_format constrint, all passed, so I'm not sure if this modify worked. -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 28 Gerrit-Owner: wangsheng Gerrit-Reviewer: Anonymous Coward (606) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 02 Sep 2020 03:26:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
wangsheng has uploaded a new patch set (#28). (
http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
..
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/1-1-5dbd44ad-18bc-40f2-9dd6-aeb2cc23457c-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/3-3-27db2521-1e8b-40c1-b846-552cd620abce-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/4-4-f1b55628-0544-4833-8b11-1b4add53dfd6-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/6-6-f75530ef-93b6-4994-b3c8-db957d44848c-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/7-7-8d9b22da-5f10-4cbf-8e4d-160f829b5e48-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/9-9-f029a1f7-9024-4bc3-a030-e20861586146-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-11-f07814ae-56cd-486b-af81-18541437da7d-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-12-967c70a4-bf4d-4a82-8c97-c90e2b4d9dcf-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-14-d0cdca7f-c050-407e-b70c-2bd076f83e4e-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-15-0e931a1f-309e-43b3-a5cf-3ef82fa4a87c-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-17-43138078-244c-4b38-8127-04a5bfbc4695-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-19-52569895-df25-4ad8-b64d-49c4540d36c9-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-20-f160c1ea-a2f5-4109-81ec-3ff9c155430f-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-22-c1f61b8c-9d9a
[Impala-ASF-CR] IMPALA-8304: Generate JUnitXML if a command run by CMake fails
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12668 ) Change subject: IMPALA-8304: Generate JUnitXML if a command run by CMake fails .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6382/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If71f2faf3ab5052b56b38f1b291fee53c390ce23 Gerrit-Change-Number: 12668 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 02 Sep 2020 03:05:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4065 Inline comparator calls into TopN::InsertBatch()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16373 ) Change subject: IMPALA-4065 Inline comparator calls into TopN::InsertBatch() .. Patch Set 9: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/7070/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/16373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I676b4c05cf10a6946c05e317b0002c1e29e78aa8 Gerrit-Change-Number: 16373 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Sep 2020 02:43:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022 part 1/2: Outer join simplification .. Patch Set 18: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6381/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 18 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 02 Sep 2020 02:39:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4065 Inline comparator calls into TopN::InsertBatch()
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16373 ) Change subject: IMPALA-4065 Inline comparator calls into TopN::InsertBatch() .. Patch Set 9: > In the work in Version 9, the STL version of the max heap is > replaced with our version, and the topN operator works when codeGen is > disabled. > > The remaining work is to code-gen the new heap code and simplify > TupleRowComparator::Compare() by dropping the IF test on > codegend_compare_fn. -- To view, visit http://gerrit.cloudera.org:8080/16373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I676b4c05cf10a6946c05e317b0002c1e29e78aa8 Gerrit-Change-Number: 16373 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Sep 2020 02:36:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4065 Inline comparator calls into TopN::InsertBatch()
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16373 ) Change subject: IMPALA-4065 Inline comparator calls into TopN::InsertBatch() .. Patch Set 9: In the work in Version 9, the STL version of the max heap is replaced with our version, and the code works when codeGen is disabled. The remaining work is to code-gen the new heap code and simplify TupleRowComparator::Compare() by dropping the IF test on codegend_compare_fn. -- To view, visit http://gerrit.cloudera.org:8080/16373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I676b4c05cf10a6946c05e317b0002c1e29e78aa8 Gerrit-Change-Number: 16373 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Sep 2020 02:35:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4065 Inline comparator calls into TopN::InsertBatch()
Qifan Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16373 Change subject: IMPALA-4065 Inline comparator calls into TopN::InsertBatch() .. IMPALA-4065 Inline comparator calls into TopN::InsertBatch() This work addresses the current limitation in TopN node by replaing the std::priority_queue with an in-house implementation. In this way, it is feasible to call more efficient version of comparator when comparing tuples during the selection of top n values. The in-house version of the priority queue is LLVM code-gened. Testing: 1. Added a new test TestBasic in a new test harness priority-queue-test.cc to verify that the priority queue works properly. 2. Ran Core tests successfully. Change-Id: I676b4c05cf10a6946c05e317b0002c1e29e78aa8 --- M be/src/exec/topn-node-ir.cc M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/util/CMakeLists.txt A be/src/util/comparator-wrapper.h A be/src/util/priority-queue-test.cc A be/src/util/priority-queue.h M be/src/util/tuple-row-compare.cc M be/src/util/tuple-row-compare.h 9 files changed, 387 insertions(+), 65 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/16373/9 -- To view, visit http://gerrit.cloudera.org:8080/16373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I676b4c05cf10a6946c05e317b0002c1e29e78aa8 Gerrit-Change-Number: 16373 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
wangsheng has posted comments on this change. (
http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
..
Patch Set 27:
> (1 comment)
>
> No worries, thanks for your persistence. Seems like something is
> wrong with test_iceberg.py in the dockerised test. I'm also trying
> to figure out what went wrong.
>
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3075/testReport/query_test.test_iceberg/TestCreatingIcebergTable/test_create_iceberg_tables_protocol__beeswax___exec_optionbatch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0table_format__parquet_none_/
Hi Zoltan, here is the test I submitted on Jenkins according to patch 27:
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11905/.
Also failed, but seems unrelated to icberg: "23:12:14 15:12:14 Error executing
hive SQL:
/home/ubuntu/Impala/logs/data_loading/sql/tpcds/load-tpcds-core-hive-generated-orc-def-block.sql
See:
/home/ubuntu/Impala/logs/data_loading/sql/tpcds/load-tpcds-core-hive-generated-orc-def-block.sql.log".
I've download the log, found the reason: "Error while compiling statement:
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get
checksum, since file
/test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451163/base_002/_orc_acid_version
is under construction" I'm not sure is this error caused by other patch?
Besides, you mentioned "Similarly to test_scanners.py, we could constrain the
file_format dimensions to Parquet.", do you mean like this?
@classmethod
def add_test_dimensions(cls):
super(TestHudiParquet, cls).add_test_dimensions()
cls.ImpalaTestMatrix.add_dimension(
create_exec_option_dimension(debug_action_options=DEBUG_ACTION_DIMS))
cls.ImpalaTestMatrix.add_constraint(
lambda v: v.get_value('table_format').file_format == 'parquet')
To be honest, I'm not very faimilar about this...
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 27
Gerrit-Owner: wangsheng
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Wed, 02 Sep 2020 02:23:11 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8304: Generate JUnitXML if a command run by CMake fails
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12668 ) Change subject: IMPALA-8304: Generate JUnitXML if a command run by CMake fails .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7069/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If71f2faf3ab5052b56b38f1b291fee53c390ce23 Gerrit-Change-Number: 12668 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 02 Sep 2020 01:59:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10090: Create aarch64 development environment on ubuntu 18.04
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16065 ) Change subject: IMPALA-10090: Create aarch64 development environment on ubuntu 18.04 .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7068/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769668c834ab0dd504a822ed9153186778275d59 Gerrit-Change-Number: 16065 Gerrit-PatchSet: 15 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Sep 2020 01:43:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8304: Generate JUnitXML if a command run by CMake fails
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12668 to look at the new patch set (#6). Change subject: IMPALA-8304: Generate JUnitXML if a command run by CMake fails .. IMPALA-8304: Generate JUnitXML if a command run by CMake fails This wraps each command executed by CMake with a wrapper that generates a JUnitXML file if the command fails. If the command succeeds, the wrapper does nothing. The wrapper applies to C++ compilation, linking, and custom shell commands (such as building the frontend via maven). It does not apply to failures coming from CMake itself. It can be disabled by setting DISABLE_CMAKE_JUNITXML. Testing: - Verified it does not impact a successful build (including with ccache and/or distcc). - Verified it generates JUnitXML for C++ and Java compilation failures. - Verified it doesn't use the wrapper when DISABLE_CMAKE_JUNITXML is set. Change-Id: If71f2faf3ab5052b56b38f1b291fee53c390ce23 --- M CMakeLists.txt M be/CMakeLists.txt A bin/junitxml_command_wrapper.sh M lib/python/impala_py_lib/jenkins/generate_junitxml.py 4 files changed, 82 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/12668/6 -- To view, visit http://gerrit.cloudera.org:8080/12668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If71f2faf3ab5052b56b38f1b291fee53c390ce23 Gerrit-Change-Number: 12668 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-10090: Create aarch64 development environment on ubuntu 18.04
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16065 ) Change subject: IMPALA-10090: Create aarch64 development environment on ubuntu 18.04 .. Patch Set 16: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6380/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769668c834ab0dd504a822ed9153186778275d59 Gerrit-Change-Number: 16065 Gerrit-PatchSet: 16 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Sep 2020 01:36:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10090: Create aarch64 development environment on ubuntu 18.04
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16065 ) Change subject: IMPALA-10090: Create aarch64 development environment on ubuntu 18.04 .. Patch Set 16: Code-Review+2 Fixed a minor thing in the commit message -- To view, visit http://gerrit.cloudera.org:8080/16065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769668c834ab0dd504a822ed9153186778275d59 Gerrit-Change-Number: 16065 Gerrit-PatchSet: 16 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Sep 2020 01:36:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10090: Create aarch64 development environment on ubuntu 18.04
Tim Armstrong has uploaded a new patch set (#16) to the change originally created by zhaoren...@hotmail.com. ( http://gerrit.cloudera.org:8080/16065 ) Change subject: IMPALA-10090: Create aarch64 development environment on ubuntu 18.04 .. IMPALA-10090: Create aarch64 development environment on ubuntu 18.04 Including following changes: 1 build native-toolchain local by script on aarch64 platform 2 change some native-toolchain's lib version number 3 split SKIP_TOOLCHAIN_BOOTSTRAP and DOWNLOAD_CDH_COMPONETS to two things, because on aarch64, just need to download cdp components , but not need to download toolchain. 4 download hadoop aarch64 nativelibs , impala building needs these libs. With this commit, on ubuntu 18.04 aarch64 version, just need to run bin/bootstrap_development.sh, just like x86. Change-Id: I769668c834ab0dd504a822ed9153186778275d59 --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M buildall.sh 4 files changed, 63 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/16065/16 -- To view, visit http://gerrit.cloudera.org:8080/16065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I769668c834ab0dd504a822ed9153186778275d59 Gerrit-Change-Number: 16065 Gerrit-PatchSet: 16 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] :IMPALA-10090 Create aarch64 development environment on ubuntu 18.04
zhaoren...@hotmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/16065 ) Change subject: :IMPALA-10090 Create aarch64 development environment on ubuntu 18.04 .. Patch Set 15: (5 comments) http://gerrit.cloudera.org:8080/#/c/16065/14//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16065/14//COMMIT_MSG@7 PS14, Line 7: environme > environment Done http://gerrit.cloudera.org:8080/#/c/16065/14//COMMIT_MSG@7 PS14, Line 7: n ubunt > ubuntu Done http://gerrit.cloudera.org:8080/#/c/16065/13/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/16065/13/bin/bootstrap_system.sh@220 PS13, Line 220: if [[ $ARCH_NAME == 'aarch64' ]]; then > Can you comment why these packages are needed on aarch64 but not x86? Is th Yes, Tim, this is for building native-toolchain http://gerrit.cloudera.org:8080/#/c/16065/13/bin/bootstrap_system.sh@480 PS13, Line 480: echo -e "\nexport SKIP_TOOLCHAIN_BOOTSTRAP=true" >> \ > We'll need to update this to point to the official native-toolchain repo Yes, Tim, the latest change is already official native-toolchain http://gerrit.cloudera.org:8080/#/c/16065/14/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/16065/14/buildall.sh@404 PS14, Line 404: locally build > "locally built" Done -- To view, visit http://gerrit.cloudera.org:8080/16065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769668c834ab0dd504a822ed9153186778275d59 Gerrit-Change-Number: 16065 Gerrit-PatchSet: 15 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Sep 2020 01:22:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] :IMPALA-10090 Create aarch64 development environment on ubuntu 18.04
zhaoren...@hotmail.com has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/16065 ) Change subject: :IMPALA-10090 Create aarch64 development environment on ubuntu 18.04 .. :IMPALA-10090 Create aarch64 development environment on ubuntu 18.04 Including following changes: 1 build native-toolchain local by script on aarch64 platform 2 change some native-toolchain's lib version number 3 split SKIP_TOOLCHAIN_BOOTSTRAP and DOWNLOAD_CDH_COMPONETS to two things, because on aarch64, just need to download cdp components , but not need to download toolchain. 4 download hadoop aarch64 nativelibs , impala building needs these libs. With this commit, on ubuntu 18.04 aarch64 version, just need to run bin/bootstrap_development.sh, just like x86. Change-Id: I769668c834ab0dd504a822ed9153186778275d59 --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M buildall.sh 4 files changed, 63 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/16065/15 -- To view, visit http://gerrit.cloudera.org:8080/16065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I769668c834ab0dd504a822ed9153186778275d59 Gerrit-Change-Number: 16065 Gerrit-PatchSet: 15 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10076: Reduce partition level update logs
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16375 )
Change subject: IMPALA-10076: Reduce partition level update logs
..
Patch Set 2:
(4 comments)
Thank you for working on the observability. I had a few minor comments but
otherwise LGTM.
Vihang, do you plan to review or should I try to bring it to +2 myself?
http://gerrit.cloudera.org:8080/#/c/16375/2/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/16375/2/be/src/catalog/catalog-server.cc@729
PS2, Line 729: return -1;
There's an overflow here, since the return value is an unsigned int - it'll
wrap around to the maximum value. This would probably cause a UBSAN error. I'd
suggest making the return value an int32 or just an int.
http://gerrit.cloudera.org:8080/#/c/16375/2/be/src/catalog/catalog-server.cc@737
PS2, Line 737: // Skip logging partition items since FE will log their
summary.
Maybe leave the JIRA here for context?
http://gerrit.cloudera.org:8080/#/c/16375/2/be/src/service/fe-support.cc
File be/src/service/fe-support.cc:
http://gerrit.cloudera.org:8080/#/c/16375/2/be/src/service/fe-support.cc@483
PS2, Line 483: uint32_t
int or int32_t to avoid the overflow
http://gerrit.cloudera.org:8080/#/c/16375/2/fe/src/main/java/org/apache/impala/catalog/PartitionMetaSummary.java
File fe/src/main/java/org/apache/impala/catalog/PartitionMetaSummary.java:
http://gerrit.cloudera.org:8080/#/c/16375/2/fe/src/main/java/org/apache/impala/catalog/PartitionMetaSummary.java@125
PS2, Line 125: } else if (partName.compareTo(partNames[0]) < 0) {
This all feels a little complex, but I think it makes sense and I don't think
it's worth overengineering since it's so encapsulated.
--
To view, visit http://gerrit.cloudera.org:8080/16375
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368
Gerrit-Change-Number: 16375
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Wed, 02 Sep 2020 00:57:33 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10121: Generate JUnitXML for TSAN messages
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16397 ) Change subject: IMPALA-10121: Generate JUnitXML for TSAN messages .. IMPALA-10121: Generate JUnitXML for TSAN messages This adds logic in bin/jenkins/finalize.sh to check the ERROR log for TSAN messages (i.e. WARNING: ThreadSanitizer: ...) and generate a JUnitXML with the message. This happens when TSAN aborts Impala. Testing: - Ran TSAN build (which is currently failing) Change-Id: I44ea33a78482499decae0ec4c7c44513094b2f44 Reviewed-on: http://gerrit.cloudera.org:8080/16397 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M bin/jenkins/finalize.sh 1 file changed, 23 insertions(+), 2 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I44ea33a78482499decae0ec4c7c44513094b2f44 Gerrit-Change-Number: 16397 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10121: Generate JUnitXML for TSAN messages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16397 ) Change subject: IMPALA-10121: Generate JUnitXML for TSAN messages .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44ea33a78482499decae0ec4c7c44513094b2f44 Gerrit-Change-Number: 16397 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 23:28:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10094: Skip test refresh updated partitions on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16399 ) Change subject: IMPALA-10094: Skip test_refresh_updated_partitions on S3 .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6379/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d27dd76e772e396a07419a58821ba899ac74188 Gerrit-Change-Number: 16399 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 01 Sep 2020 23:24:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10094: Skip test refresh updated partitions on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16399 ) Change subject: IMPALA-10094: Skip test_refresh_updated_partitions on S3 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d27dd76e772e396a07419a58821ba899ac74188 Gerrit-Change-Number: 16399 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 01 Sep 2020 23:23:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9482 Support for BINARY columns
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16066 ) Change subject: IMPALA-9482 Support for BINARY columns .. Patch Set 7: Let me know when I should take another look -- To view, visit http://gerrit.cloudera.org:8080/16066 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36861a9ca6c2047b0d76862507c86f7f153bc582 Gerrit-Change-Number: 16066 Gerrit-PatchSet: 7 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 01 Sep 2020 23:20:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10090 Create aarch64 development evironment on unbuntu 18.04
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16065 ) Change subject: IMPALA-10090 Create aarch64 development evironment on unbuntu 18.04 .. Patch Set 14: (6 comments) Thank you for the contribution, this should help a lot with building ubuntu on aarch64. I just had a few small changes I'm requesting. http://gerrit.cloudera.org:8080/#/c/16065/14//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16065/14//COMMIT_MSG@7 PS14, Line 7: evironment environment http://gerrit.cloudera.org:8080/#/c/16065/14//COMMIT_MSG@7 PS14, Line 7: unbuntu ubuntu http://gerrit.cloudera.org:8080/#/c/16065/13/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/16065/13/bin/bootstrap_system.sh@220 PS13, Line 220: if [[ $ARCH_NAME == 'aarch64' ]]; then Can you comment why these packages are needed on aarch64 but not x86? Is this for building native-toolchain? http://gerrit.cloudera.org:8080/#/c/16065/13/bin/bootstrap_system.sh@480 PS13, Line 480: echo -e "\nexport SKIP_TOOLCHAIN_BOOTSTRAP=true" >> \ We'll need to update this to point to the official native-toolchain repo http://gerrit.cloudera.org:8080/#/c/16065/14/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/16065/14/buildall.sh@404 PS14, Line 404: local builded "locally built" http://gerrit.cloudera.org:8080/#/c/16065/14/buildall.sh@486 PS14, Line 486: # architecture. so here we will let user decide this value. Thanks for the cleanup -- To view, visit http://gerrit.cloudera.org:8080/16065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769668c834ab0dd504a822ed9153186778275d59 Gerrit-Change-Number: 16065 Gerrit-PatchSet: 14 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 22:51:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10064: Support constant propagation for eligible range predicates
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/16346 ) Change subject: IMPALA-10064: Support constant propagation for eligible range predicates .. Patch Set 10: > Patch Set 10: > > Change looks good to me. I should wait for the e2e test you mentioned right? Thanks for the review. Yes, I plan to add at least 1 e2e test with the modified dataset. I hope to do it later today after some other ongoing work. -- To view, visit http://gerrit.cloudera.org:8080/16346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I811a1f8d605c27c7704d7fc759a91510c6db3c2b Gerrit-Change-Number: 16346 Gerrit-PatchSet: 10 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 22:48:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16383 ) Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files Currently Impala checks file metadata 'hive.acid.version' to decide the full ACID schema. There are cases when Hive forgets to set this value for full ACID files, e.g. query-based compactions. So it's more robust to check the schema elements instead of the metadata field. Also, sometimes Hive write the schema with different character cases, e.g. originalTransaction vs originaltransaction, so we should rather compare the column names in a case insensitive way. Testing: * added test for full ACID compaction * added test_full_acid_schema_without_file_metadata_tag to test full ACID file without metadata 'hive.acid.version' Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Reviewed-on: http://gerrit.cloudera.org:8080/16383 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/orc-metadata-utils.cc M be/src/exec/orc-metadata-utils.h M testdata/data/README A testdata/data/full_acid_schema_but_no_acid_version.orc M testdata/workloads/functional-query/queries/QueryTest/acid-compaction.test M tests/query_test/test_acid.py 7 files changed, 88 insertions(+), 27 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16383 ) Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 01 Sep 2020 22:27:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10064: Support constant propagation for eligible range predicates
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16346 ) Change subject: IMPALA-10064: Support constant propagation for eligible range predicates .. Patch Set 10: Change looks good to me. I should wait for the e2e test you mentioned right? -- To view, visit http://gerrit.cloudera.org:8080/16346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I811a1f8d605c27c7704d7fc759a91510c6db3c2b Gerrit-Change-Number: 16346 Gerrit-PatchSet: 10 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 22:26:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10094: Skip test refresh updated partitions on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16399 ) Change subject: IMPALA-10094: Skip test_refresh_updated_partitions on S3 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7067/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d27dd76e772e396a07419a58821ba899ac74188 Gerrit-Change-Number: 16399 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 01 Sep 2020 22:23:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10094: Skip test refresh updated partitions on S3
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/16399 ) Change subject: IMPALA-10094: Skip test_refresh_updated_partitions on S3 .. Patch Set 2: Code-Review+2 Thanks for putting this together -- To view, visit http://gerrit.cloudera.org:8080/16399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d27dd76e772e396a07419a58821ba899ac74188 Gerrit-Change-Number: 16399 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 01 Sep 2020 22:04:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10094: Skip test refresh updated partitions on S3
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16399 Change subject: IMPALA-10094: Skip test_refresh_updated_partitions on S3 .. IMPALA-10094: Skip test_refresh_updated_partitions on S3 The test test_refresh_updated_partitions runs some commands using Hive which causes it fail on S3 specific jobs since we don't run HiveServer2 in those environments. This patch skips the test on non-hdfs environments. Change-Id: I0d27dd76e772e396a07419a58821ba899ac74188 --- M tests/metadata/test_reset_metadata.py 1 file changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/16399/2 -- To view, visit http://gerrit.cloudera.org:8080/16399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0d27dd76e772e396a07419a58821ba899ac74188 Gerrit-Change-Number: 16399 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-10071: Impala shouldn't create filename starting with underscore during ACID TRUNCATE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16396 ) Change subject: IMPALA-10071: Impala shouldn't create filename starting with underscore during ACID TRUNCATE .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6376/ -- To view, visit http://gerrit.cloudera.org:8080/16396 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0557b9944624bc123c540752bbe3877312a7ac9 Gerrit-Change-Number: 16396 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 01 Sep 2020 21:26:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6375/ -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 01 Sep 2020 18:46:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10121: Generate JUnitXML for TSAN messages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16397 ) Change subject: IMPALA-10121: Generate JUnitXML for TSAN messages .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6378/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44ea33a78482499decae0ec4c7c44513094b2f44 Gerrit-Change-Number: 16397 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 18:19:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10121: Generate JUnitXML for TSAN messages
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16397 ) Change subject: IMPALA-10121: Generate JUnitXML for TSAN messages .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44ea33a78482499decae0ec4c7c44513094b2f44 Gerrit-Change-Number: 16397 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 18:13:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10121: Generate JUnitXML for TSAN messages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16397 ) Change subject: IMPALA-10121: Generate JUnitXML for TSAN messages .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7066/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44ea33a78482499decae0ec4c7c44513094b2f44 Gerrit-Change-Number: 16397 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 01 Sep 2020 17:54:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10121: Generate JUnitXML for TSAN messages
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16397 Change subject: IMPALA-10121: Generate JUnitXML for TSAN messages .. IMPALA-10121: Generate JUnitXML for TSAN messages This adds logic in bin/jenkins/finalize.sh to check the ERROR log for TSAN messages (i.e. WARNING: ThreadSanitizer: ...) and generate a JUnitXML with the message. This happens when TSAN aborts Impala. Testing: - Ran TSAN build (which is currently failing) Change-Id: I44ea33a78482499decae0ec4c7c44513094b2f44 --- M bin/jenkins/finalize.sh 1 file changed, 23 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/16397/1 -- To view, visit http://gerrit.cloudera.org:8080/16397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I44ea33a78482499decae0ec4c7c44513094b2f44 Gerrit-Change-Number: 16397 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16383 ) Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7065/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 01 Sep 2020 17:25:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16383 ) Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 01 Sep 2020 17:05:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16383 ) Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. Patch Set 4: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 01 Sep 2020 17:04:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16383 ) Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6377/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 01 Sep 2020 17:05:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16383 to look at the new patch set (#4). Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files Currently Impala checks file metadata 'hive.acid.version' to decide the full ACID schema. There are cases when Hive forgets to set this value for full ACID files, e.g. query-based compactions. So it's more robust to check the schema elements instead of the metadata field. Also, sometimes Hive write the schema with different character cases, e.g. originalTransaction vs originaltransaction, so we should rather compare the column names in a case insensitive way. Testing: * added test for full ACID compaction * added test_full_acid_schema_without_file_metadata_tag to test full ACID file without metadata 'hive.acid.version' Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/orc-metadata-utils.cc M be/src/exec/orc-metadata-utils.h M testdata/data/README A testdata/data/full_acid_schema_but_no_acid_version.orc M testdata/workloads/functional-query/queries/QueryTest/acid-compaction.test M tests/query_test/test_acid.py 7 files changed, 88 insertions(+), 27 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/16383/4 -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10119: Fix impala-shell history duplication test
Tim Armstrong has removed a vote on this change. Change subject: IMPALA-10119: Fix impala-shell history duplication test .. Removed Code-Review+2 by Tim Armstrong -- To view, visit http://gerrit.cloudera.org:8080/16391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I242eb47cc8093c4566de206f46b75b3feab1183c Gerrit-Change-Number: 16391 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10119: Fix impala-shell history duplication test
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16391 ) Change subject: IMPALA-10119: Fix impala-shell history duplication test .. Patch Set 3: The idea of the fix makes sense but looks like it caused the test to fail in precommit? -- To view, visit http://gerrit.cloudera.org:8080/16391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I242eb47cc8093c4566de206f46b75b3feab1183c Gerrit-Change-Number: 16391 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 16:41:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10119: Fix impala-shell history duplication test
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16391 ) Change subject: IMPALA-10119: Fix impala-shell history duplication test .. Patch Set 3: The \C might not be flushed to the console cause there's no newline or something.. just a thought. -- To view, visit http://gerrit.cloudera.org:8080/16391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I242eb47cc8093c4566de206f46b75b3feab1183c Gerrit-Change-Number: 16391 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 16:42:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10119: Fix impala-shell history duplication test
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16391 ) Change subject: IMPALA-10119: Fix impala-shell history duplication test .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I242eb47cc8093c4566de206f46b75b3feab1183c Gerrit-Change-Number: 16391 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 16:40:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16383 ) Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6374/ -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 01 Sep 2020 16:33:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10071: Impala shouldn't create filename starting with underscore during ACID TRUNCATE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16396 ) Change subject: IMPALA-10071: Impala shouldn't create filename starting with underscore during ACID TRUNCATE .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7064/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16396 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0557b9944624bc123c540752bbe3877312a7ac9 Gerrit-Change-Number: 16396 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 01 Sep 2020 16:25:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10071: Impala shouldn't create filename starting with underscore during ACID TRUNCATE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16396 ) Change subject: IMPALA-10071: Impala shouldn't create filename starting with underscore during ACID TRUNCATE .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6376/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16396 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0557b9944624bc123c540752bbe3877312a7ac9 Gerrit-Change-Number: 16396 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 01 Sep 2020 16:06:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10071: Impala shouldn't create filename starting with underscore during ACID TRUNCATE
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16396 Change subject: IMPALA-10071: Impala shouldn't create filename starting with underscore during ACID TRUNCATE .. IMPALA-10071: Impala shouldn't create filename starting with underscore during ACID TRUNCATE When Impala TRUNCATEs an ACID table, it creates a new base directory with the hidden file "_empty" in it. Newer Hive versions ignore files starting with underscore, therefore they ignore the whole base directory. To resolve this issue we can simply rename the empty file to "empty". Testing: * update acid-truncate.test accordingly Change-Id: Ia0557b9944624bc123c540752bbe3877312a7ac9 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M testdata/workloads/functional-query/queries/QueryTest/acid-truncate.test 2 files changed, 10 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/16396/1 -- To view, visit http://gerrit.cloudera.org:8080/16396 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia0557b9944624bc123c540752bbe3877312a7ac9 Gerrit-Change-Number: 16396 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10064: Support constant propagation for eligible range predicates
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16346 ) Change subject: IMPALA-10064: Support constant propagation for eligible range predicates .. Patch Set 10: Was able to find some time to look for local joins in TPCDS. There are not many at all. query2.sql:WHERE d_week_seq1 = d_week_seq2 - 53 query59.sql: AND d_week_seq1 = d_week_seq2 - 52 query59.sql:WHERE s_store_id1 = s_store_id2 Sounds like they can be used for the min/max filtering at least. -- To view, visit http://gerrit.cloudera.org:8080/16346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I811a1f8d605c27c7704d7fc759a91510c6db3c2b Gerrit-Change-Number: 16346 Gerrit-PatchSet: 10 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 15:08:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 27: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7063/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 27 Gerrit-Owner: wangsheng Gerrit-Reviewer: Anonymous Coward (606) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 01 Sep 2020 14:57:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022 part 1/2: Outer join simplification .. Patch Set 18: Code-Review+1 Looks good! -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 18 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 01 Sep 2020 13:40:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 27: (1 comment) No worries, thanks for your persistence. Seems like something is wrong with test_iceberg.py in the dockerised test. I'm also trying to figure out what went wrong. https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3075/testReport/query_test.test_iceberg/TestCreatingIcebergTable/test_create_iceberg_tables_protocol__beeswax___exec_optionbatch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0table_format__parquet_none_/ http://gerrit.cloudera.org:8080/#/c/16143/26/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/16143/26/tests/query_test/test_iceberg.py@27 PS26, Line 27: Similarly to test_scanners.py, we could constrain the file_format dimensions to Parquet. -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 27 Gerrit-Owner: wangsheng Gerrit-Reviewer: Anonymous Coward (606) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 01 Sep 2020 14:46:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 27: > FileMetadataLoaderTest.testIcebergLoading and test_create_iceberg_tables > still fail: > https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11899/testReport/ > https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3075/testReport/ Hi Zoltan, sorry about this. I forgot to modify data file name in FileMetadataLoaderTest.java. I've already adjust code and restart a Jenkins test. -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 27 Gerrit-Owner: wangsheng Gerrit-Reviewer: Anonymous Coward (606) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 01 Sep 2020 14:37:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
wangsheng has uploaded a new patch set (#27). (
http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
..
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/1-1-5dbd44ad-18bc-40f2-9dd6-aeb2cc23457c-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/3-3-27db2521-1e8b-40c1-b846-552cd620abce-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/4-4-f1b55628-0544-4833-8b11-1b4add53dfd6-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/6-6-f75530ef-93b6-4994-b3c8-db957d44848c-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/7-7-8d9b22da-5f10-4cbf-8e4d-160f829b5e48-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/9-9-f029a1f7-9024-4bc3-a030-e20861586146-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-11-f07814ae-56cd-486b-af81-18541437da7d-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-12-967c70a4-bf4d-4a82-8c97-c90e2b4d9dcf-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-14-d0cdca7f-c050-407e-b70c-2bd076f83e4e-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-15-0e931a1f-309e-43b3-a5cf-3ef82fa4a87c-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-17-43138078-244c-4b38-8127-04a5bfbc4695-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-19-52569895-df25-4ad8-b64d-49c4540d36c9-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-20-f160c1ea-a2f5-4109-81ec-3ff9c155430f-0.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-22-c1f61b8c-9d9a
[Impala-ASF-CR] IMPALA-10119: Fix impala-shell history duplication test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16391 ) Change subject: IMPALA-10119: Fix impala-shell history duplication test .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6372/ -- To view, visit http://gerrit.cloudera.org:8080/16391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I242eb47cc8093c4566de206f46b75b3feab1183c Gerrit-Change-Number: 16391 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 14:08:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 01 Sep 2020 14:28:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 26: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6373/ -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 26 Gerrit-Owner: wangsheng Gerrit-Reviewer: Anonymous Coward (606) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 01 Sep 2020 14:12:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022 part 1/2: Outer join simplification .. Patch Set 18: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7062/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 18 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 01 Sep 2020 13:38:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 26: FileMetadataLoaderTest.testIcebergLoading and test_create_iceberg_tables still fail: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11899/testReport/ https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3075/testReport/ -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 26 Gerrit-Owner: wangsheng Gerrit-Reviewer: Anonymous Coward (606) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 01 Sep 2020 13:34:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6375/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 01 Sep 2020 13:25:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022 part 1/2: Outer join simplification .. Patch Set 18: (2 comments) http://gerrit.cloudera.org:8080/#/c/16266/16//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16266/16//COMMIT_MSG@9 PS16, Line 9: Outer joins in SQL can return rows with certain columns filled with : NULLs when a match can not be found. However, such rows can be : rejected by null-rejecting predicates. The conditions in a null-rejecting : predicate that are always evaluated to FALSE for NULLs are referred to : as null-filtering conditions. : : In general, an outer join can be converted to an inner joi > Maybe reworded as Done http://gerrit.cloudera.org:8080/#/c/16266/16//COMMIT_MSG@50 PS16, Line 50: > I think we need to add a high-level description of what work is done in thi Done -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 18 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 01 Sep 2020 13:16:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Xianqing He has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022 part 1/2: Outer join simplification .. IMPALA-5022 part 1/2: Outer join simplification Outer joins in SQL can return rows with certain columns filled with NULLs when a match can not be found. However, such rows can be rejected by null-rejecting predicates. The conditions in a null-rejecting predicate that are always evaluated to FALSE for NULLs are referred to as null-filtering conditions. In general, an outer join can be converted to an inner join if there exist null-filtering conditions on the inner tables. In a left outer join, the right table is the inner table, while in a right outer join it is the left table. In a full outer join, both tables are inner tables. For example, 1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10 = A INNER JOIN B ON A.id = B.id WHERE B.v > 10 2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10 = A INNER JOIN B ON A.id = B.id WHERE A.v > 10 3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 = A LEFT JOIN B ON A.id = B.id WHERE A.v > 10 4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10 = A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10 5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10 = A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10 6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id = A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id 7. A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id = A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id 8. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id = A LEFT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id 9. A FULL JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id = A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id 10. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id = A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id In this commit, we have supported most of the cases that can convert an outer join to an inner join, except for converting the embedding inline view outer join by the join condition like "SELECT * FROM T1 JOIN (SELECT T3.A A FROM T2 LEFT JOIN T3 ON T3.B=T2.B) T4 ON T4.A=T1.A". We will support it in part 2. Tests: * Update the baseline plan Tests * Add new plan tests outer-to-inner-joins.test * Add new query tests to verify the correctness on transformation * Ran the full set of verifications in Impala Public Jenkins Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test M testdata/workloads/functional-planner/queries/PlannerTest/joins.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test A testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test M testdata/workloads/functional-query/queries/QueryTest/parqu
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7061/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 01 Sep 2020 12:57:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Tamas Mate has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. IMPALA-10051: impala-shell exits with ValueError with WITH clauses When a query a contains WITH clause impala-shell tries to identify whether it is a DML query or not, so that later it can provide appropriate result messages. Earlier shlex was used to create tokens and assess the query type based on that. However shlex can misinterpret some query strings where whitespace charachters are mixed with quotes, because it splits the string based on whitespace charachters. In some scenarios 'ValueError: No closing quotation' error can occur. This change moves the tokenization from shlex to sqlparse. Testing: - Added unit test to cover queries that contain mixed whitespaces and strings Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 19 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/16389/3 -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 2: (1 comment) Thanks for the review Fredy. Fixed the dry-run test, the assert was dev env specific, my test table was empty. http://gerrit.cloudera.org:8080/#/c/16389/2/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/16389/2/shell/impala_shell.py@1280 PS2, Line 1280: if self.DML_REGEX.match(query_type.lower()): > looks like there were failed tests in the dry-run Right, it is cleaner this way. Done. -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 01 Sep 2020 11:54:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10012: Test for undecodable binary display with impala-shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16393 ) Change subject: IMPALA-10012: Test for undecodable binary display with impala-shell .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7060/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16393 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic17bcc30a9d948fb1adc8ba5ea5bdeaded32d7ff Gerrit-Change-Number: 16393 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 01 Sep 2020 11:44:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16383 ) Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7059/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 01 Sep 2020 11:32:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10012: Test for undecodable binary display with impala-shell
Adam Tamas has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16393 Change subject: IMPALA-10012: Test for undecodable binary display with impala-shell .. IMPALA-10012: Test for undecodable binary display with impala-shell A test for making sure the impala-shell is working perfectly after trying to display an undecodable binary code. Only working if both the original IMPALA-10012 and the binary type is already implemeted (IMPALA-9482)! Should work for HS2 and HS2-http (undecodable binary display is NOT working with beeswax protocol) Change-Id: Ic17bcc30a9d948fb1adc8ba5ea5bdeaded32d7ff --- M tests/shell/test_shell_interactive.py 1 file changed, 27 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/16393/1 -- To view, visit http://gerrit.cloudera.org:8080/16393 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic17bcc30a9d948fb1adc8ba5ea5bdeaded32d7ff Gerrit-Change-Number: 16393 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Tamas
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16383 ) Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. Patch Set 2: Code-Review+2 (1 comment) Carry +2 http://gerrit.cloudera.org:8080/#/c/16383/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16383/1//COMMIT_MSG@20 PS1, Line 20: * added test_full_acid_schema_without_file_metadata_tag to test full : ACID file without metadata 'hi > I would prefer to have an automatic test with a specific file, as Hive may Done -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 01 Sep 2020 11:21:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16383 ) Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 01 Sep 2020 11:22:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16383 ) Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6374/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 01 Sep 2020 11:22:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16383 to look at the new patch set (#2). Change subject: IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files .. IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files Currently Impala checks file metadata 'hive.acid.version' to decide the full ACID schema. There are cases when Hive forgets to set this value for full ACID files, e.g. query-based compactions. So it's more robust to check the schema elements instead of the metadata field. Also, sometimes Hive write the schema with different character cases, e.g. originalTransaction vs originaltransaction, so we should rather compare the column names in a case insensitive way. Testing: * added test for full ACID compaction * added test_full_acid_schema_without_file_metadata_tag to test full ACID file without metadata 'hive.acid.version' Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/orc-metadata-utils.cc M be/src/exec/orc-metadata-utils.h M testdata/data/README M testdata/workloads/functional-query/queries/QueryTest/acid-compaction.test M tests/query_test/test_acid.py 6 files changed, 88 insertions(+), 27 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/16383/2 -- To view, visit http://gerrit.cloudera.org:8080/16383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14 Gerrit-Change-Number: 16383 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10087: IMPALA-6050 causes alluxio not to be supported
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16379 ) Change subject: IMPALA-10087: IMPALA-6050 causes alluxio not to be supported .. IMPALA-10087: IMPALA-6050 causes alluxio not to be supported This change adds file type support for alluxio. Alluxio URLs have a different prefix such as：alluxio://zk@zk-1:2181,zk-2:2181,zk-3:2181/path/ Testing: Add unit test for alluxio file system type checks. Change-Id: Id92ec9cb0ee241a039fe4a96e1bc2ab3eaaf8f77 Reviewed-on: http://gerrit.cloudera.org:8080/16379 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java 2 files changed, 11 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id92ec9cb0ee241a039fe4a96e1bc2ab3eaaf8f77 Gerrit-Change-Number: 16379 Gerrit-PatchSet: 4 Gerrit-Owner: abeltian Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10087: IMPALA-6050 causes alluxio not to be supported
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16379 ) Change subject: IMPALA-10087: IMPALA-6050 causes alluxio not to be supported .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id92ec9cb0ee241a039fe4a96e1bc2ab3eaaf8f77 Gerrit-Change-Number: 16379 Gerrit-PatchSet: 3 Gerrit-Owner: abeltian Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 09:43:00 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] MPALA-10075: Reuse unchanged partition instances
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16392 ) Change subject: [WIP] MPALA-10075: Reuse unchanged partition instances .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7058/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16392 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dd645c260d271291021e52fdac4b74924df1170 Gerrit-Change-Number: 16392 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 01 Sep 2020 09:29:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10119: Fix impala-shell history duplication test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16391 ) Change subject: IMPALA-10119: Fix impala-shell history duplication test .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7057/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I242eb47cc8093c4566de206f46b75b3feab1183c Gerrit-Change-Number: 16391 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 09:05:29 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] MPALA-10075: Reuse unchanged partition instances
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16392 Change subject: [WIP] MPALA-10075: Reuse unchanged partition instances .. [WIP] MPALA-10075: Reuse unchanged partition instances Currently, we always update the partition instance when we reload a partition. If a partition remains the same after reloading, we should reuse the old partition instance. So we won't send redudant updates on these partitions. This reduces the size of the catalog topic update. When a huge table is REFRESHed, catalogd only propagates the changes partitions. Tests: - Manually verified that repeate REFERSHs won't trigger catalog updates on partitions. - (TODO) add e2e tests Change-Id: I2dd645c260d271291021e52fdac4b74924df1170 --- M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java 4 files changed, 30 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/16392/1 -- To view, visit http://gerrit.cloudera.org:8080/16392 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2dd645c260d271291021e52fdac4b74924df1170 Gerrit-Change-Number: 16392 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 26: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6373/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 26 Gerrit-Owner: wangsheng Gerrit-Reviewer: Anonymous Coward (606) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 01 Sep 2020 08:58:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7310: Partial fix for NDV cardinality with NULLs.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16349 ) Change subject: IMPALA-7310: Partial fix for NDV cardinality with NULLs. .. IMPALA-7310: Partial fix for NDV cardinality with NULLs. This fix just handles the case where a column's cardinality is zero however it's nullable and we have null stats to indicate there are null values, therefore we adjust the cardinality from 0 to 1. The cardinality of zero was especially problematic when calculating cardinalities for multiple predicates with multiplication. The 0 would propagate up the plan tree and result in poor plan choices such as always using broadcast joins where shuffle would've been more optimal. Testing: * 26 Node TPC-DS 30TB run had better plans for Q4 and Q11 - Q4 172s -> 80s - Q11 103s -> 77s * CardinalityTest * TpcdsPlannerTest Change-Id: Iec967053b4991f8c67cde62adf003cbd3f429032 Reviewed-on: http://gerrit.cloudera.org:8080/16349 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java M fe/src/test/java/org/apache/impala/analysis/ExprNdvTest.java M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q04.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q11.test 6 files changed, 795 insertions(+), 784 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iec967053b4991f8c67cde62adf003cbd3f429032 Gerrit-Change-Number: 16349 Gerrit-PatchSet: 14 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7310: Partial fix for NDV cardinality with NULLs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16349 ) Change subject: IMPALA-7310: Partial fix for NDV cardinality with NULLs. .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec967053b4991f8c67cde62adf003cbd3f429032 Gerrit-Change-Number: 16349 Gerrit-PatchSet: 13 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 08:54:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10119: Fix impala-shell history duplication test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16391 ) Change subject: IMPALA-10119: Fix impala-shell history duplication test .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6372/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I242eb47cc8093c4566de206f46b75b3feab1183c Gerrit-Change-Number: 16391 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 08:51:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10119: Fix impala-shell history duplication test
Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16391 Change subject: IMPALA-10119: Fix impala-shell history duplication test .. IMPALA-10119: Fix impala-shell history duplication test The flaky test was TestImpalaShellInteractive.test_history_does_not_duplicate_on_interrupt The test failed with timeout error when the interrupt signal arrived later after the next test query was started. The impala-shell output was ^C instead of the expected query result. This change adds an additional blocking expect call to wait for the interrupt signal to arrive before sending in the next query. Change-Id: I242eb47cc8093c4566de206f46b75b3feab1183c --- M tests/shell/test_shell_interactive.py 1 file changed, 6 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/16391/3 -- To view, visit http://gerrit.cloudera.org:8080/16391 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I242eb47cc8093c4566de206f46b75b3feab1183c Gerrit-Change-Number: 16391 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16143 ) Change subject: IMPALA-9741: Support querying Iceberg table by impala .. Patch Set 26: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7056/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006 Gerrit-Change-Number: 16143 Gerrit-PatchSet: 26 Gerrit-Owner: wangsheng Gerrit-Reviewer: Anonymous Coward (606) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 01 Sep 2020 07:13:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10064: Support constant propagation for eligible range predicates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16346 ) Change subject: IMPALA-10064: Support constant propagation for eligible range predicates .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7055/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I811a1f8d605c27c7704d7fc759a91510c6db3c2b Gerrit-Change-Number: 16346 Gerrit-PatchSet: 10 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 07:08:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10064: Support constant propagation for eligible range predicates
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/16346 ) Change subject: IMPALA-10064: Support constant propagation for eligible range predicates .. Patch Set 10: (6 comments) http://gerrit.cloudera.org:8080/#/c/16346/9/fe/src/main/java/org/apache/impala/analysis/ConstantPredicateHandler.java File fe/src/main/java/org/apache/impala/analysis/ConstantPredicateHandler.java: http://gerrit.cloudera.org:8080/#/c/16346/9/fe/src/main/java/org/apache/impala/analysis/ConstantPredicateHandler.java@55 PS9, Line 55:* predicates. The candidates BitSet is used to determine which members of > Mention how 'candidates' is used? Done http://gerrit.cloudera.org:8080/#/c/16346/9/fe/src/main/java/org/apache/impala/analysis/ConstantPredicateHandler.java@61 PS9, Line 61: co > nit: these parens prob aren't needed, right? Done http://gerrit.cloudera.org:8080/#/c/16346/9/fe/src/main/java/org/apache/impala/analysis/ConstantPredicateHandler.java@66 PS9, Line 66: = BinaryPredicate.IS_RANGE_PREDICATE.apply(b > can't this be !=? Done http://gerrit.cloudera.org:8080/#/c/16346/9/fe/src/main/java/org/apache/impala/analysis/ConstantPredicateHandler.java@128 PS9, Line 128: opagation > Can't this be Map.Entry to keep it type-safe and avoid cast? Done http://gerrit.cloudera.org:8080/#/c/16346/9/fe/src/main/java/org/apache/impala/analysis/ConstantPredicateHandler.java@132 PS9, Line 132: dtExpr = d > Map.Entry>? Done http://gerrit.cloudera.org:8080/#/c/16346/9/testdata/workloads/functional-planner/queries/PlannerTest/constant-propagation.test File testdata/workloads/functional-planner/queries/PlannerTest/constant-propagation.test: http://gerrit.cloudera.org:8080/#/c/16346/9/testdata/workloads/functional-planner/queries/PlannerTest/constant-propagation.test@419 PS9, Line 419:predicates: timestamp_col <= TIMESTAMP '2010-12-01 00:00:00', timestamp_col >= TIMESTAMP '2009-12-01 00:00:00', date_col = CAST(timestamp_col AS DATE) > Good point. All the use cases I have seen so far were ones where date_col Made the code change to preserve the original conjunct date_col = cast(timestamp_col as date). Updated plans. TODO: Add e2e test against a dataset where date values != timestamp col's date component. -- To view, visit http://gerrit.cloudera.org:8080/16346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I811a1f8d605c27c7704d7fc759a91510c6db3c2b Gerrit-Change-Number: 16346 Gerrit-PatchSet: 10 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 01 Sep 2020 06:59:28 + Gerrit-HasComments: Yes
