[Impala-ASF-CR] IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu)
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18875 ) Change subject: IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu) .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18875 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b8696190e4713bda787e773d48943b5dfc6335e Gerrit-Change-Number: 18875 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sun, 21 Aug 2022 04:49:03 + Gerrit-HasComments: No
[Impala-ASF-CR] Test review
Joe McDonnell has abandoned this change. ( http://gerrit.cloudera.org:8080/18860 ) Change subject: Test review .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/18860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Id1c1742716c4922f125b20d719928e9a7b94ecc8 Gerrit-Change-Number: 18860 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18875 ) Change subject: IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu) .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18875 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b8696190e4713bda787e773d48943b5dfc6335e Gerrit-Change-Number: 18875 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Sun, 21 Aug 2022 02:50:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11418: The only select constant statement need not to spool results
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/18711 ) Change subject: IMPALA-11418: The only select constant statement need not to spool results .. Patch Set 4: (3 comments) Thank you for working on this! The change makes sense to me. I only have few nits. http://gerrit.cloudera.org:8080/#/c/18711/4/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java: http://gerrit.cloudera.org:8080/#/c/18711/4/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@474 PS4, Line 474: The statement only select constant need not to spool query results. nit: A statement that only selects constant does not need to spool query results. http://gerrit.cloudera.org:8080/#/c/18711/4/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@480 PS4, Line 480: clientRequest.query_options.setSpool_query_results(false); Can you add LOG.trace here? Like what we have in: https://github.com/apache/impala/blob/f1420ae51b2e00ad093be64a0d2135ca36aa6514/fe/src/main/java/org/apache/impala/planner/PlanRootSink.java#L101-L105 http://gerrit.cloudera.org:8080/#/c/18711/4/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/18711/4/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@1705 PS4, Line 1705:* Check if the select statement only select constant Can you add two example query in this method documentation for both case? - Select constant - Select constant from InlineView -- To view, visit http://gerrit.cloudera.org:8080/18711 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icd4d73c21106048df68a270cf03d4abd56bd3aac Gerrit-Change-Number: 18711 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Sun, 21 Aug 2022 02:42:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11471: Track disk usage for build-all-flag-combinations.sh
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18810 ) Change subject: IMPALA-11471: Track disk usage for build-all-flag-combinations.sh .. IMPALA-11471: Track disk usage for build-all-flag-combinations.sh This adds some calls to df and du to track disk space usage throughout the builds. This also cleans up the Impala dev environment before creating the m2 archive. Change-Id: I8ab31d8d7096b49d8404edf7521d46f23155526f Reviewed-on: http://gerrit.cloudera.org:8080/18810 Reviewed-by: Impala Public Jenkins Tested-by: Joe McDonnell --- M bin/jenkins/build-all-flag-combinations.sh 1 file changed, 17 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved Joe McDonnell: Verified -- To view, visit http://gerrit.cloudera.org:8080/18810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8ab31d8d7096b49d8404edf7521d46f23155526f Gerrit-Change-Number: 18810 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11471: Track disk usage for build-all-flag-combinations.sh
Joe McDonnell has removed a vote on this change. Change subject: IMPALA-11471: Track disk usage for build-all-flag-combinations.sh .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/18810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I8ab31d8d7096b49d8404edf7521d46f23155526f Gerrit-Change-Number: 18810 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11471: Track disk usage for build-all-flag-combinations.sh
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18810 ) Change subject: IMPALA-11471: Track disk usage for build-all-flag-combinations.sh .. Patch Set 3: Verified+1 This only impacts the all-build-options-ub1604 job, and it passed, so going ahead with this. -- To view, visit http://gerrit.cloudera.org:8080/18810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ab31d8d7096b49d8404edf7521d46f23155526f Gerrit-Change-Number: 18810 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 20 Aug 2022 22:21:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu)
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18875 ) Change subject: IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu) .. Patch Set 1: Not related, retrying -- To view, visit http://gerrit.cloudera.org:8080/18875 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b8696190e4713bda787e773d48943b5dfc6335e Gerrit-Change-Number: 18875 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Sat, 20 Aug 2022 21:56:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18875 ) Change subject: IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu) .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8479/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18875 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b8696190e4713bda787e773d48943b5dfc6335e Gerrit-Change-Number: 18875 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Sat, 20 Aug 2022 21:56:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11514: Workaround s3 connection timeout issues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18872 ) Change subject: IMPALA-11514: Workaround s3 connection timeout issues .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18872 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590 Gerrit-Change-Number: 18872 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 20 Aug 2022 09:42:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18875 ) Change subject: IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu) .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8478/ -- To view, visit http://gerrit.cloudera.org:8080/18875 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b8696190e4713bda787e773d48943b5dfc6335e Gerrit-Change-Number: 18875 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 20 Aug 2022 09:41:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11471: Track disk usage for build-all-flag-combinations.sh
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18810 ) Change subject: IMPALA-11471: Track disk usage for build-all-flag-combinations.sh .. Patch Set 3: The failure is due to IMPALA-11352. I think we can merge this regardless of it. -- To view, visit http://gerrit.cloudera.org:8080/18810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ab31d8d7096b49d8404edf7521d46f23155526f Gerrit-Change-Number: 18810 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 18 Aug 2022 11:45:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 11: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 11 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 18 Aug 2022 23:04:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num values is 0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18854 ) Change subject: IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num_values is 0 .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37 Gerrit-Change-Number: 18854 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 19 Aug 2022 07:31:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11429: Set table owner after creating an Iceberg table
Tamas Mate has posted comments on this change. (
http://gerrit.cloudera.org:8080/18837 )
Change subject: IMPALA-11429: Set table owner after creating an Iceberg table
..
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/18837/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/18837/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3626
PS5, Line 3626: it has
: // hardcoded that the
nit: this part was fuzzy for me, maybe something like:
"however, the table owner is hardcoded to be the user running the Iceberg
process"
http://gerrit.cloudera.org:8080/#/c/18837/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3633
PS5, Line 3633: throw new ImpalaRuntimeException("Failed to set table
owner after creating " +
: "Iceberg table but the table has been created
successfully", e);
With Ranger enabled and a failing alter operation this table would be
inaccessible for the user right?
http://gerrit.cloudera.org:8080/#/c/18837/5/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:
http://gerrit.cloudera.org:8080/#/c/18837/5/tests/query_test/test_iceberg.py@130
PS5, Line 130: pytest.skip()
Wouldn't a time.sleep(5) solve the issue after create?
--
To view, visit http://gerrit.cloudera.org:8080/18837
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5cac198a4a53be3599cb582864ee5f8c269202c0
Gerrit-Change-Number: 18837
Gerrit-PatchSet: 5
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Comment-Date: Fri, 19 Aug 2022 13:48:34 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7942 (part 1): Add query hints for table cardinalities
wangsheng has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18829 ) Change subject: IMPALA-7942 (part 1): Add query hints for table cardinalities .. IMPALA-7942 (part 1): Add query hints for table cardinalities Currently, We need execute 'COMPUTE STATS' manually to compute table stats info. Stats is very useful for query planning. Without these stats, query plan maybe worse. In order to solve this probelm, this patch adds a new query hint: 'TABLE_NUM_ROWS', We can use this new hint after a hdfs or kudu table in query like this: * select col from t /* +TABLE_NUM_ROWS(1000) */; If set, Impala will use this value as table scanned rows when table no stats or has corrput stats. This hint value will not valid if table stats is normal. Testing: - Added new fe test in 'PlannerTest' - Added new fe test in 'AnalyzeStmtsTest' for negative cases Change-Id: I9f0c773f4e67782a1428db64062f68afbd257af7 --- M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/table-cardinality-hint.test 8 files changed, 160 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/18829/4 -- To view, visit http://gerrit.cloudera.org:8080/18829 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9f0c773f4e67782a1428db64062f68afbd257af7 Gerrit-Change-Number: 18829 Gerrit-PatchSet: 4 Gerrit-Owner: wangsheng Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-9482: Support for BINARY columns
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16066 ) Change subject: IMPALA-9482: Support for BINARY columns .. IMPALA-9482: Support for BINARY columns This patch adds support for BINARY columns for all table formats with the exception of Kudu. In Hive the main difference between STRING and BINARY is that STRING is assumed to be UTF8 encoded, while BINARY can be any byte array. Some other differences in Hive: - BINARY can be only cast from/to STRING - Only a small subset of built-in STRING functions support BINARY. - In several file formats (e.g. text) BINARY is base64 encoded. - No NDV is calculated during COMPUTE STATISTICS. As Impala doesn't treat STRINGs as UTF8, BINARY and STRING become nearly identical, especially from the backend's perspective. For this reason, BINARY is implemented a bit differently compared to other types: while the frontend treats STRING and BINARY as two separate types, most of the backend uses PrimitiveType::TYPE_STRING for BINARY too, e.g. in SlotDesc. Only the following parts of backend need to differentiate between STRING and BINARY: - table scanners - table writers - HS2/Beeswax service These parts have access to column metadata, which allows to add special handling for BINARY. Only a very few builtins are allowed for BINARY at the moment: - length - min/max/count - coalesce and similar "selector" functions Other STRING functions can be only used by casting to STRING first. Adding support for more of these functions is very easy, as simply the BINARY type has to be "connected" to the already existing STRING function's signature. Functions where the result depends on utf8_mode need to ensure that with BINARY it always works as if utf8_mode=0 (for example length() is mapped to bytes() as length count utf8 chars if utf8_mode=1). All kinds of UDFs (native, Hive legacy, Hive generic) support BINARY, though in case of legacy Hive UDFs it is only supported if the argument and return types are set explicitely to ensure backward compatibility. See IMPALA-11340 for details. The original plan was to behave as close to Hive as possible, but I realized that Hive has more relaxed casting rules than Impala, which led to STRING<->BINARY casts being necessary in more cases in Impala. This was needed to disallow passing a BINARY to functions that expect a STRING argument. An example for the difference is that in INSERT ... VALUES () string literals need to be explicitly cast to BINARY, while this is not needed in Hive. Testing: - Added functional.binary_tbl for all file formats (except Kudu) to test scanning. - Removed functional.unsupported_types and related tests, as now Impala supports all (non-complex) types that Hive does. - Added FE/EE tests mainly based on the ones added to the DATE type Change-Id: I36861a9ca6c2047b0d76862507c86f7f153bc582 Reviewed-on: http://gerrit.cloudera.org:8080/16066 Reviewed-by: Quanlong Huang Tested-by: Impala Public Jenkins --- M be/src/exec/file-metadata-utils.cc M be/src/exec/hbase/hbase-scan-node.cc M be/src/exec/hbase/hbase-scan-node.h M be/src/exec/hbase/hbase-table-writer.cc M be/src/exec/hdfs-scanner-ir.cc M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-text-table-writer.cc M be/src/exec/orc/orc-metadata-utils.cc M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-metadata-utils.h M be/src/exec/rcfile/hdfs-rcfile-scanner.cc M be/src/exec/text-converter.cc M be/src/exec/text-converter.h M be/src/exec/text-converter.inline.h M be/src/exec/text/hdfs-text-scanner.cc M be/src/exprs/expr-test.cc M be/src/exprs/utility-functions-ir.cc M be/src/exprs/utility-functions.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/types.cc M be/src/runtime/types.h M be/src/service/hs2-util.cc M be/src/service/hs2-util.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/query-result-set.cc M be/src/testutil/test-udfs.cc M be/src/util/coding-util.cc M be/src/util/coding-util.h M bin/rat_exclude_files.txt M common/function-registry/impala_functions.py M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CastExpr.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/InPredicate.java M fe/src/main/java/org/apache/impala/analysis/LikePredicate.java M fe/src/main/java/org/apache/impala/analysis/LiteralExpr.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java M fe/src/main/java/org/apache/impala/catalog/Function.java M fe/src/main/java/org/apache/impala/catalog/PrimitiveType.java M
[Impala-ASF-CR] IMPALA-11429: Set table owner after creating an Iceberg table
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/18837 ) Change subject: IMPALA-11429: Set table owner after creating an Iceberg table .. Patch Set 5: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/18837/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/18837/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3626 PS5, Line 3626: has nit: "is" feels a bit more natural -- To view, visit http://gerrit.cloudera.org:8080/18837 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5cac198a4a53be3599cb582864ee5f8c269202c0 Gerrit-Change-Number: 18837 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Fri, 19 Aug 2022 13:28:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11294: Remove dictionary runtime filter test workarounds
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18839 ) Change subject: IMPALA-11294: Remove dictionary runtime filter test workarounds .. Patch Set 3: Failure caused by some infra issues: Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[parallel-all-tests/10340:parallel-all-tests #10340] but FlowNode was not found in storage for head id:FlowNodeId 1:19 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) ... -- To view, visit http://gerrit.cloudera.org:8080/18839 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9697f5a1a04160dd17553603b00e15d63b49e2bf Gerrit-Change-Number: 18839 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 19 Aug 2022 14:47:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11436: Change search bind authentication parameters
Tamas Mate has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/18819 )
Change subject: IMPALA-11436: Change search bind authentication parameters
..
IMPALA-11436: Change search bind authentication parameters
Impala's search bind authentication intends to mimic Spring's behaviour.
However, the login username and user dn paremeters were swapped for
group searches compared to Spring. This change intends to align these
parameters.
For user search, Spring uses {0} to replace the login username.
Meanwhile, during group search {0} is used to replace the login user dn
and {1} is used to replace the login username.
Testing:
- Ran LdapSearchBindImpalaShellTest frontend tests
Change-Id: I9808566a348f7c6200b0571fbc05e67f720f2075
Reviewed-on: http://gerrit.cloudera.org:8080/18819
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
M be/src/util/ldap-search-bind.cc
M docs/topics/impala_ldap.xml
M
fe/src/test/java/org/apache/impala/customcluster/LdapSearchBindImpalaShellTest.java
3 files changed, 30 insertions(+), 23 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/18819
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9808566a348f7c6200b0571fbc05e67f720f2075
Gerrit-Change-Number: 18819
Gerrit-PatchSet: 6
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-7942 (part 1): Add query hints for table cardinalities
wangsheng has posted comments on this change. (
http://gerrit.cloudera.org:8080/18829 )
Change subject: IMPALA-7942 (part 1): Add query hints for table cardinalities
..
Patch Set 4:
(13 comments)
Hi Quanlong, thanks for advice. I think you are right, use hint value to
replace original table stats may cause consistency when use explain. So I
modify the code, table hint is valid when no stats or has corrupt stats. Here
is a problem, I use 'functional.alltypes' for hdfs table with stats,
'functional_parquet.alltypes' for hdfs table without stats,
'functional_kudu.alltypes' for kudu table with stats.
But I did not figure out the way to test kudu table without stats.
http://gerrit.cloudera.org:8080/#/c/18829/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18829/3//COMMIT_MSG@10
PS3, Line 10: query planning.
> nit: generation? or "query planning", "query optimization"
Done
http://gerrit.cloudera.org:8080/#/c/18829/3//COMMIT_MSG@19
PS3, Line 19: l not
: valid if table stat
> nit: regardless the existense of the stats.
Done
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/analysis/TableRef.java
File fe/src/main/java/org/apache/impala/analysis/TableRef.java:
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/analysis/TableRef.java@173
PS3, Line 173: // Value of query hint 'TABLE_NUM_ROWS' on this table. Used in
constructing ScanNode if
: // the table does not have stats, or has correct stats. -1
indicates no hint. Currently,
: // this hint is valid for hd
> nit: might be better to reword to
Done
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/analysis/TableRef.java@510
PS3, Line 510:
> nit: isTableHintSupported
Done
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/analysis/TableRef.java@514
PS3, Line 514: estTable() != null &&
> nit: reword to
Done
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/analysis/TableRef.java@518
PS3, Line 518: for (PlanHint hint: tableHints_) {
> Does this mean we support such hints for Kudu tables now? I think the SCHED
Done
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/analysis/TableRef.java@555
PS3, Line 555: analyzer.setHasPlanHints();
> nit: can we remove this comment? It seems no need to explain the following
Done
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/analysis/TableRef.java@556
PS3, Line 556: Long.parseLo
> nit: can use Long.parseLong() directly, which is used internally in Long.va
Done
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/analysis/TableRef.java@564
PS3, Line 564: Returns whether the table supports hint. Currently,
> nit: reword to
Done
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1465
PS3, Line 1465: cardinality_ = extrapolatedNumRows_;
> Should we overwrite this as well if the hint exists?
Done
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1542
PS3, Line 1542:* partitions with corrupt stats.
> Could you please mention the hint in this comment?
Done
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1573
PS3, Line 1573: // by each of the partitions, as the row count for the
table.
> I thought we only use the hint when missing stats. This always overwrites t
Done
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/planner/ScanNode.java
File fe/src/main/java/org/apache/impala/planner/ScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18829/3/fe/src/main/java/org/apache/impala/planner/ScanNode.java@81
PS3, Line 81: // Refer to the comment of 'TableRef.tableNumRowsHint_'
: protected long tableNumRowsHint_ = -1;
:
: public ScanNode(PlanNodeId id, TupleDes
> nit: maybe we can just refer to the comment of TableRef.tableNumRowsHint_
Done
--
To view, visit http://gerrit.cloudera.org:8080/18829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f0c773f4e67782a1428db64062f68afbd257af7
Gerrit-Change-Number: 18829
Gerrit-PatchSet: 4
Gerrit-Owner: wangsheng
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Fucun Chu
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Fri, 19 Aug 2022 13:06:03 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11436: Change search bind authentication parameters
Tamas Mate has uploaded a new patch set (#5). (
http://gerrit.cloudera.org:8080/18819 )
Change subject: IMPALA-11436: Change search bind authentication parameters
..
IMPALA-11436: Change search bind authentication parameters
Impala's search bind authentication intends to mimic Spring's behaviour.
However, the login username and user dn paremeters were swapped for
group searches compared to Spring. This change intends to align these
parameters.
For user search, Spring uses {0} to replace the login username.
Meanwhile, during group search {0} is used to replace the login user dn
and {1} is used to replace the login username.
Testing:
- Ran LdapSearchBindImpalaShellTest frontend tests
Change-Id: I9808566a348f7c6200b0571fbc05e67f720f2075
---
M be/src/util/ldap-search-bind.cc
M docs/topics/impala_ldap.xml
M
fe/src/test/java/org/apache/impala/customcluster/LdapSearchBindImpalaShellTest.java
3 files changed, 30 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/18819/5
--
To view, visit http://gerrit.cloudera.org:8080/18819
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9808566a348f7c6200b0571fbc05e67f720f2075
Gerrit-Change-Number: 18819
Gerrit-PatchSet: 5
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-9482: Support for BINARY columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16066 ) Change subject: IMPALA-9482: Support for BINARY columns .. Patch Set 26: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16066 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36861a9ca6c2047b0d76862507c86f7f153bc582 Gerrit-Change-Number: 16066 Gerrit-PatchSet: 26 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 19 Aug 2022 13:55:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/18798/11/be/src/runtime/row-batch.h File be/src/runtime/row-batch.h: http://gerrit.cloudera.org:8080/#/c/18798/11/be/src/runtime/row-batch.h@122 PS11, Line 122: } add DCHECK(mem_allocator_ == locking_free_pool); -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 11 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 19 Aug 2022 15:15:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from
OUTER/INNER JOIN to SCANNODE
..
Patch Set 6:
(23 comments)
Thanks for your contribution, Baike! This is an important improvement. I still
need some time to finish my first round of review. Left some comments first.
http://gerrit.cloudera.org:8080/#/c/18731/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18731/6//COMMIT_MSG@9
PS6, Line 9: In order to reduce the amount of data read and transmitted, the
non-equivalent condition of Join can be pushed to SCAN_NODE.
nit: each line of the commit message should have 72 or fewer characters. The
commit title is ok.
http://gerrit.cloudera.org:8080/#/c/18731/6//COMMIT_MSG@16
PS6, Line 16:
Please introduce the new query option, ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN,
in the commit message.
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java:
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@142
PS6, Line 142: pushdownNonEquiConjunct(analyzer);
Should we move these before computeStats() at line 136 to have better
cardinality?
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@463
PS6, Line 463:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@507
PS6, Line 507:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@580
PS6, Line 580:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@592
PS6, Line 592: if (!(predicate.getChild(0) instanceof LiteralExpr
: && !(predicate.getChild(0) instanceof NullLiteral))
: && !(predicate.getChild(1) instanceof LiteralExpr
: && !(predicate.getChild(1) instanceof
NullLiteral))) {
This is not that readable. We can simplify it to
if (!Expr.IS_NON_NULL_LITERAL.apply(predicate.getChild(0))
&& !Expr.IS_NON_NULL_LITERAL.apply(predicate.getChild(1))) {
continue;
}
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@631
PS6, Line 631: groupOtherJoinConjunctsAccordingToSlotRef
This function is similar to the above one. Can we refactor them into one?
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@632
PS6, Line 632:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@675
PS6, Line 675:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@731
PS6, Line 731:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@746
PS6, Line 746: if (child instanceof LiteralExpr) {
: LiteralExpr currentValue = (LiteralExpr) child;
: if (minValue == null) {
: minValue = currentValue;
: } else {
: if (minValue.compareTo(currentValue) > 0) {
: minValue = currentValue;
: }
: }
: }
This is a common patten of the code. We can extract this into a method to
deduplicate some codes.
EDIT: the code structure in getMaxLiteralFromPredicates() looks better. It'd be
nice if we can refactor them into one method.
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@775
PS6, Line 775: < 0
Shouldn't this be "> 0" ?
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@784
PS6, Line 784: i = 1
Could you explain why we don't need "i = 0" ?
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@785
PS6, Line 785: !(predicate.getChild(i) instanceof LiteralExpr)
: || (predicate.getChild(i) instanceof
NullLiteral)
This can be simplified to
!Expr.IS_NON_NULL_LITERAL.apply(predicate.getChild(i))
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@796
PS6, Line 796: minValue = literalValues.get(0);
Shouldn't we update 'minValue' instead of replacing it? There could be more
than one IN-list, e.g. "x in [1, 2, 3] and x in [0, 1]". It
[Impala-ASF-CR] IMPALA-11294: Remove dictionary runtime filter test workarounds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18839 ) Change subject: IMPALA-11294: Remove dictionary runtime filter test workarounds .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8475/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18839 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9697f5a1a04160dd17553603b00e15d63b49e2bf Gerrit-Change-Number: 18839 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 19 Aug 2022 14:48:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11294: Remove dictionary runtime filter test workarounds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18839 ) Change subject: IMPALA-11294: Remove dictionary runtime filter test workarounds .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18839 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9697f5a1a04160dd17553603b00e15d63b49e2bf Gerrit-Change-Number: 18839 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 19 Aug 2022 14:48:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11469: (Addendum) Refactor BackendConfig for tests
Michael Smith has abandoned this change. ( http://gerrit.cloudera.org:8080/18828 ) Change subject: IMPALA-11469: (Addendum) Refactor BackendConfig for tests .. Abandoned No strong reason to refactor this. I prefer avoiding statics where possible, but I think that's mostly aesthetic in this case. -- To view, visit http://gerrit.cloudera.org:8080/18828 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I802fcbf70808f2127197f720a7247c3d85389d6f Gerrit-Change-Number: 18828 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang
[Impala-ASF-CR] IMPALA-7942 (part 1): Add query hints for table cardinalities
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18829 ) Change subject: IMPALA-7942 (part 1): Add query hints for table cardinalities .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11191/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18829 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f0c773f4e67782a1428db64062f68afbd257af7 Gerrit-Change-Number: 18829 Gerrit-PatchSet: 4 Gerrit-Owner: wangsheng Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 19 Aug 2022 13:23:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10213: Add test for local vs remote scheduling
Kurt Deschler has posted comments on this change. (
http://gerrit.cloudera.org:8080/18841 )
Change subject: IMPALA-10213: Add test for local vs remote scheduling
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18841/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18841/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1385
PS3, Line 1385: if (fsType != FileSystemUtil.FsType.OZONE)
++numScanRangesNoDiskIds_;
Add {}
--
To view, visit http://gerrit.cloudera.org:8080/18841
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e5606528404c3d4fd164c03dec8315345be5f6d
Gerrit-Change-Number: 18841
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Michael Smith
Gerrit-Comment-Date: Fri, 19 Aug 2022 17:59:14 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num values is 0
Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18854 ) Change subject: IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num_values is 0 .. IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num_values is 0 Fix Impalad crashed in the method ParquetBoolDecoder::SkipValues when the parameter 'num_values' is 0. The function should tolerate that the 'num_values' is 0 values. Testing: - Add e2e tests Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37 Reviewed-on: http://gerrit.cloudera.org:8080/18854 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/parquet/parquet-bool-decoder.cc A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-bugfix.test M tests/query_test/test_parquet_stats.py 3 files changed, 35 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37 Gerrit-Change-Number: 18854 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching
Kurt Deschler has posted comments on this change. (
http://gerrit.cloudera.org:8080/18853 )
Change subject: IMPALA-10214: Ozone remote file handle caching
..
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18853/4/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:
http://gerrit.cloudera.org:8080/#/c/18853/4/tests/custom_cluster/test_hdfs_fd_caching.py@127
PS4, Line 127: s.connect(('10.254.254.254', 1))
Hard-coded address?
--
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Comment-Date: Fri, 19 Aug 2022 18:00:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Omid Shahidi has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. IMPALA-6684: Fix untracked memory in KRPC During serialization of an row batch header, a tuple_data_ is created which will hold the compressed tuple data for an outbound row batch. We would like this tuple data to be trackable as it is responsible for a significant portion of untrackable memory from the krpc data stream sender. By using free pool, we are able to allocate tuple data and compression scratch and account for it in the memory tracker of the KrpcDataStreamSender. This solution creates a RAII class responsible for memory allocation and changes the existing code to use a char buffer pointed by a char* tuple_data_ instead of the previously used std::string tuple_data_. The thrift implementation is left unchanged and the protobuf implementation is seperated. Testing: - Passed core tests. - Ran a single node benchmark which shows no regression. - Updated row-batch-serialize-test and row-batch-serialize-benchmark to test the row-batch serialization used by KRPC. - Manually collected query-profile, heap growth, and memory usage log showing untracked memory decreased by 1/2. - Add end-end unit-test to verify the new counters in runtime profile New row-batch serialization benchmark: serialize: Func10% 50% 90% 10% 50% 90% ile (rel) (rel) (rel) --- ser_no_dups_baseline8.36 8.6 8.7 1X 1X 1X ser_no_dups 6.73 6.85 6.93 0.804X 0.796X 0.796X ser_no_dups_full5.28 5.38 5.55 0.631X 0.625X 0.637X ser_adjacent_dups_baseline 12.9 13.2 13.4 1X 1X 1X ser_adjacent_dups 23.2 23.7 24.1 1.8X 1.8X 1.8X ser_adjacent_dups_full 19.9 20.3 20.7 1.54X 1.54X 1.55X ser_dups_baseline 9.17 9.54 9.72 1X 1X 1X ser_dups7.45 7.69 7.86 0.812X 0.806X 0.809X ser_dups_full 14.6 15 15.3 1.6X 1.57X 1.57X deserialize: Func10% 50% 90% 10% 50% 90% ile (rel) (rel) (rel) --- deser_no_dups_baseline 32.6 33.5 34 1X 1X1X deser_no_dups 32.5 33.1 33.7 0.999X 0.99X 0.992X deser_adjacent_dups_baseline 53.1 54 54.7 1X 1X 1X deser_adjacent_dups 80.3 81.6 82.5 1.51X 1.51X 1.51X deser_dups_baseline 52.4 54 54.7 1X 1X 1X deser_dups 86.8 88.4 89.7 1.66X 1.64X 1.64X Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 --- M be/src/benchmarks/row-batch-serialize-benchmark.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h A be/src/runtime/row-batch.inline.h A testdata/workloads/tpch/queries/datastream-sender.test A tests/query_test/test_datastream_sender.py 9 files changed, 655 insertions(+), 214 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/18798/13 -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 13 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10213: Add test for local vs remote scheduling
Hello Kurt Deschler, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18841 to look at the new patch set (#4). Change subject: IMPALA-10213: Add test for local vs remote scheduling .. IMPALA-10213: Add test for local vs remote scheduling Impala already supports locality-aware scheduling with Ozone because it returns location data on partitions. That data doesn't include specific storage ids in getStorageIds, so we skip a warning that will always trigger on Ozone. Updates Ozone to add implicit rules mapping localhost -> 127.0.0.1 for local development. HDFS translates localhost to 127.0.0.1 for host names in its location data, which Impala will identify as colocated with executors in the dev environment. Ozone doesn't, and the default Impala hostname is the machine hostname - not localhost - so without this change all HDFS access in the minicluster is local but all Ozone access is remote. Adds a test to verify local vs remote assignment by using custom clusters with hostnames that either do or don't match storage hostnames. Change-Id: I4e5606528404c3d4fd164c03dec8315345be5f6d --- M be/src/runtime/io/disk-io-mgr.cc M be/src/scheduling/executor-group.cc M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M tests/common/skip.py A tests/custom_cluster/test_scheduler_locality.py M tests/metadata/test_stats_extrapolation.py 6 files changed, 74 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/18841/4 -- To view, visit http://gerrit.cloudera.org:8080/18841 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4e5606528404c3d4fd164c03dec8315345be5f6d Gerrit-Change-Number: 18841 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching
Hello Kurt Deschler, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18853 to look at the new patch set (#5). Change subject: IMPALA-10214: Ozone remote file handle caching .. IMPALA-10214: Ozone remote file handle caching Enables support for caching remote file handles for Ozone. Local file handles were already cached unintentionally, similar to HDFS. Updates file handle cache enablement to be more stringent about enabling caching, similar to what's suggested in IMPALA-10375. File handle caching is enabled if a max_cached_file_handles is non-zero and any of the following are true - HDFS file is local - HDFS file is remote and cache_remote_file_handles is enabled - Ozone file is local or remote and cache_ozone_file_handles is enabled - S3 file is remote and cache_s3_file_handles is enabled - ABFS file is remote and cache_abfs_file_handles is enabled Enables testing Ozone in test_hdfs_fd_caching, and adds a test that remote caching can be disabled using individual flags. Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9 --- M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/handle-cache.inline.h M be/src/runtime/io/scan-range.cc M tests/custom_cluster/test_hdfs_fd_caching.py 4 files changed, 60 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18853/5 -- To view, visit http://gerrit.cloudera.org:8080/18853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9 Gerrit-Change-Number: 18853 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Omid Shahidi has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 13: (2 comments) http://gerrit.cloudera.org:8080/#/c/18798/13/be/src/runtime/krpc-data-stream-sender.cc File be/src/runtime/krpc-data-stream-sender.cc: http://gerrit.cloudera.org:8080/#/c/18798/13/be/src/runtime/krpc-data-stream-sender.cc@1095 PS13, Line 1095: if (outbound_rb_mem_pool_.get() != nullptr) possibly change to if(UNLIKELY(outbound_rb_mem_pool.get() != nullptr) same comment for line 1095 http://gerrit.cloudera.org:8080/#/c/18798/13/be/src/runtime/krpc-data-stream-sender.cc@1103 PS13, Line 1103: delete outbound_rb_free_pool_; check if outbound_rb_free_pool_ != nullptr -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 13 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 19 Aug 2022 22:01:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11294: Remove dictionary runtime filter test workarounds
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18839 ) Change subject: IMPALA-11294: Remove dictionary runtime filter test workarounds .. IMPALA-11294: Remove dictionary runtime filter test workarounds This commit removes a workaround in the dictionary runtime filter tests that was necessary till IMPALA-11047 got merged. Testing - Executed e2e test locally. Change-Id: I9697f5a1a04160dd17553603b00e15d63b49e2bf Reviewed-on: http://gerrit.cloudera.org:8080/18839 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_runtime_filters.py 1 file changed, 0 insertions(+), 7 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18839 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9697f5a1a04160dd17553603b00e15d63b49e2bf Gerrit-Change-Number: 18839 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/18853 )
Change subject: IMPALA-10214: Ozone remote file handle caching
..
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18853/4/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:
http://gerrit.cloudera.org:8080/#/c/18853/4/tests/custom_cluster/test_hdfs_fd_caching.py@127
PS4, Line 127: s.connect(('10.254.254.254', 1))
> Hard-coded address?
Ah, the timeout means it doesn't matter if it resolves (we kind of don't want
it to). I should've left a comment.
--
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Michael Smith
Gerrit-Comment-Date: Fri, 19 Aug 2022 21:44:18 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11294: Remove dictionary runtime filter test workarounds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18839 ) Change subject: IMPALA-11294: Remove dictionary runtime filter test workarounds .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18839 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9697f5a1a04160dd17553603b00e15d63b49e2bf Gerrit-Change-Number: 18839 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 19 Aug 2022 19:35:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11192/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 13 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 19 Aug 2022 20:30:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8476/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 13 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 19 Aug 2022 20:49:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11514: Workaround s3 connection timeout issues
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18872 Change subject: IMPALA-11514: Workaround s3 connection timeout issues .. IMPALA-11514: Workaround s3 connection timeout issues When running on s3, dataload is failing with errors like "Timeout waiting for connection from pool". The underlying issue is a subtle issue in the async draining codepath (HADOOP-18410). As a temporary workaround, this adds fs.s3a.input.async.drain.threshold=512G to core-site.xml. This disables the async drain codepath. Testing: - An s3 job passed with this setting Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590 --- M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py 1 file changed, 3 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/18872/1 -- To view, visit http://gerrit.cloudera.org:8080/18872 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590 Gerrit-Change-Number: 18872 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu)
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18875 Change subject: IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu) .. IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu) Machines that don't have en_US.UTF-8 installed see issues when running ExprTest.Utf8MaskTest. This currently impacts the Docker-based tests. This installs the appropriate language packs to have en_US.UTF-8 installed. Testing: - Ran docker-based tests and verified that ExprTest.Utf8MaskTest passes. Change-Id: I1b8696190e4713bda787e773d48943b5dfc6335e --- M bin/bootstrap_system.sh 1 file changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/18875/1 -- To view, visit http://gerrit.cloudera.org:8080/18875 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1b8696190e4713bda787e773d48943b5dfc6335e Gerrit-Change-Number: 18875 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 ) Change subject: IMPALA-10214: Ozone remote file handle caching .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11193/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9 Gerrit-Change-Number: 18853 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 19 Aug 2022 22:06:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10213: Add test for local vs remote scheduling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18841 ) Change subject: IMPALA-10213: Add test for local vs remote scheduling .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11194/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18841 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e5606528404c3d4fd164c03dec8315345be5f6d Gerrit-Change-Number: 18841 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 19 Aug 2022 22:12:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11514: Workaround s3 connection timeout issues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18872 ) Change subject: IMPALA-11514: Workaround s3 connection timeout issues .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11195/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18872 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590 Gerrit-Change-Number: 18872 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 19 Aug 2022 23:58:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 13: (2 comments) http://gerrit.cloudera.org:8080/#/c/18798/13/be/src/runtime/krpc-data-stream-sender.cc File be/src/runtime/krpc-data-stream-sender.cc: http://gerrit.cloudera.org:8080/#/c/18798/13/be/src/runtime/krpc-data-stream-sender.cc@1095 PS13, Line 1095: if (outbound_rb_mem_pool_.get() != nullptr) > possibly change to if(UNLIKELY(outbound_rb_mem_pool.get() != nullptr) ok http://gerrit.cloudera.org:8080/#/c/18798/13/be/src/runtime/krpc-data-stream-sender.cc@1103 PS13, Line 1103: delete outbound_rb_free_pool_; > check if outbound_rb_free_pool_ != nullptr that will be safe -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 13 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sat, 20 Aug 2022 02:39:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 13: There are 4 end-end unit-test failures due to memory limit exceeded for large rows. This is expected behavior since we allocate more memory from trackable buffer pool, especially affect test with large rows. To fix these failures, increase mem_limit for these test cases. -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 13 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sat, 20 Aug 2022 02:38:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18875 ) Change subject: IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu) .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11196/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18875 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b8696190e4713bda787e773d48943b5dfc6335e Gerrit-Change-Number: 18875 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 20 Aug 2022 01:04:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 13: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8476/ -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 13 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sat, 20 Aug 2022 01:31:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11514: Workaround s3 connection timeout issues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18872 ) Change subject: IMPALA-11514: Workaround s3 connection timeout issues .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8477/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18872 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590 Gerrit-Change-Number: 18872 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 20 Aug 2022 04:55:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18875 ) Change subject: IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu) .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8478/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18875 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b8696190e4713bda787e773d48943b5dfc6335e Gerrit-Change-Number: 18875 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 20 Aug 2022 04:55:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11494: Don't always produce Ranger audit log for authorized query
Fang-Yu Rao has posted comments on this change. (
http://gerrit.cloudera.org:8080/18850 )
Change subject: IMPALA-11494: Don't always produce Ranger audit log for
authorized query
..
Patch Set 4:
(1 comment)
> Patch Set 3:
>
> (1 comment)
>
> lgtm, one comment about code cleanness
Hi all, I have slightly revised the patch set 3 according to Csaba's
suggestion. Please let me know if there is any other comment
Thank you very much for the help!
http://gerrit.cloudera.org:8080/#/c/18850/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java:
http://gerrit.cloudera.org:8080/#/c/18850/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@203
PS3, Line 203: // to determine whether the correct audit events are
collected.
: auditHandler.getAuthzEvents().clear();
: } else {
: // We send audit log entries to the Ranger server only if
authorization failed or
: // analysis succeeded.
> The two conditions are the opposites of each other - can you convert this t
Thanks Csaba!
I will change the above to an if-else block in the next patch.
--
To view, visit http://gerrit.cloudera.org:8080/18850
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I701652e457d3118f43249e83be933713b17ce48f
Gerrit-Change-Number: 18850
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Wed, 17 Aug 2022 16:35:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11494: Don't always produce Ranger audit log for authorized query
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18850 ) Change subject: IMPALA-11494: Don't always produce Ranger audit log for authorized query .. Patch Set 4: Code-Review+2 Thanks for the changes! -- To view, visit http://gerrit.cloudera.org:8080/18850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I701652e457d3118f43249e83be933713b17ce48f Gerrit-Change-Number: 18850 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 17 Aug 2022 16:45:53 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-11500] Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num values is 0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18854 ) Change subject: [IMPALA-11500] Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num_values is 0 .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37 Gerrit-Change-Number: 18854 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 16 Aug 2022 16:24:47 + Gerrit-HasComments: No
[Impala-ASF-CR] Test review
Joe McDonnell has removed a vote on this change. Change subject: Test review .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/18860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Id1c1742716c4922f125b20d719928e9a7b94ecc8 Gerrit-Change-Number: 18860 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-11494: Don't always produce Ranger audit log for authorized query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18850 ) Change subject: IMPALA-11494: Don't always produce Ranger audit log for authorized query .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11172/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I701652e457d3118f43249e83be933713b17ce48f Gerrit-Change-Number: 18850 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 17 Aug 2022 16:54:22 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-11484: Create SCAN plan for Iceberg V2 position delete tables
Hello Tamas Mate, Gabor Kaszab, lipeng...@sensorsdata.cn, Gergely Fürnstáhl, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18847 to look at the new patch set (#2). Change subject: WIP: IMPALA-11484: Create SCAN plan for Iceberg V2 position delete tables .. WIP: IMPALA-11484: Create SCAN plan for Iceberg V2 position delete tables This patch adds support for reading Iceberg V2 tables use position deletes. Equality deletes are still not supported. Position delete files store the file path and file position of the deleted rows. When an Iceberg table has position delete files we need to do an ANTI JOIN between data files and delete files. From the data files we need to query the virtual columns INPUT__FILE__NAME and FILE__POSITION, while from the delete files we need the data columns 'file_path' and 'pos'. The latter data columns are added as 'hidden columns' to Iceberg tables. 'Hidden column' is a new concept introduced by this patch. This patch introduces a new class 'IcebergScanPlanner' which has the responsibility of doing a plan for Iceberg table scans. It creates the aforementioned ANTI JOIN. Also, if there are data files without corresponding delete files, we can have a separate SCAN node and its results would be UNIONed to the rows coming from the ANTI JOIN: UNION / \ SCAN data ANTI JOIN / \ SCAN dataSCAN deletes Predicate pushdown and time travel logic is transferred from IcebergScanNode to IcebergScanPlanner. TODO: * better cardinality estimates * add tests Change-Id: I672cfee18d8e131772d90378d5b12ad4d0f7dd48 --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Column.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java A fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-1.parquet 18 files changed, 850 insertions(+), 537 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/18847/2 -- To view, visit http://gerrit.cloudera.org:8080/18847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I672cfee18d8e131772d90378d5b12ad4d0f7dd48 Gerrit-Change-Number: 18847 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[native-toolchain-CR] IMPALA-9999: Build the toolchain with GCC 10
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18806 ) Change subject: IMPALA-: Build the toolchain with GCC 10 .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2fbfcdcb497ad60772bf0a7837e4043ae75bcfaa Gerrit-Change-Number: 18806 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 17 Aug 2022 15:45:30 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-11484: Create SCAN plan for Iceberg V2 position delete tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18847 ) Change subject: WIP: IMPALA-11484: Create SCAN plan for Iceberg V2 position delete tables .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11171/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I672cfee18d8e131772d90378d5b12ad4d0f7dd48 Gerrit-Change-Number: 18847 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Wed, 17 Aug 2022 14:32:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Omid Shahidi has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. IMPALA-6684: Fix untracked memory in KRPC During serialization of an row batch header, a tuple_data_ is created which will hold the compressed tuple data for an outbound row batch. We would like this tuple data to be trackable as it is responsible for a significant portion of untrackable memory from the krpc data stream sender. By using free pool, we are able to allocate tuple data and compression scratch and account for it in the memory tracker of the KrpcDataStreamSender. This solution creates a RAII class responsible for memory allocation and changes the existing code to use a char buffer pointed by a char* tuple_data_ instead of the previously used std::string tuple_data_. The thrift implementation is left unchanged and the protobuf implementation is seperated. Testing: - Passed core tests. - Ran a single node benchmark which shows no regression. - Updated row-batch-serialize-test and row-batch-serialize-benchmark to test the row-batch serialization used by KRPC. - Manually collected query-profile, heap growth, and memory usage log showing untracked memory decreased by 1/2. - Add end-end unit-test to verify the new counters in runtime profile Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 --- M be/src/benchmarks/row-batch-serialize-benchmark.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h A be/src/runtime/row-batch.inline.h A testdata/workloads/functional-query/queries/datastream-sender.test A testdata/workloads/tpch/queries/datastream-sender.test A tests/query_test/test_datastream_sender.py 10 files changed, 657 insertions(+), 173 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/18798/8 -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 8 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11494: Don't always produce Ranger audit log for authorized query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18850 ) Change subject: IMPALA-11494: Don't always produce Ranger audit log for authorized query .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I701652e457d3118f43249e83be933713b17ce48f Gerrit-Change-Number: 18850 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 17 Aug 2022 21:51:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11494: Don't always produce Ranger audit log for authorized query
Fang-Yu Rao has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18850 ) Change subject: IMPALA-11494: Don't always produce Ranger audit log for authorized query .. IMPALA-11494: Don't always produce Ranger audit log for authorized query Before this patch, when Impala could not resolve a given table '.' during the query analysis, Impala would still attempt to register 2 privilege requests. One was for the table '' under the database '' and the other was for the table '' under the database 'default'. The first one should be registered since Impala had to determine whether such an access should be allowed (even though in fact the table did not exist), whereas the second one was incorrect in that 'default.' definitely was not '.' in general. Furthermore, Impala always sent audit log entries to the Ranger server for an authorized query against non-existing table(s). The 2 facts described above resulted in Impala producing Ranger audit log entries for the tables '.' and 'default.' when a requesting user granted sufficient privileges on the databases of '' and 'default' submitted a query against a non-existing table '.'. None of the audit log entries should be generated because i) the privilege request for '.' was allowed and '.' did not exist, and ii) 'default.' did not correspond to a table. This patch fixes the 2 issues mentioned above so that Impala will not generate any Ranger audit log entry for an authorized query against a non-existing table. Testing: - Added a frontend test case to verify no Ranger audit log entry will be produced for an authorized query against a non-existing table. Change-Id: I701652e457d3118f43249e83be933713b17ce48f --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java 7 files changed, 75 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18850/4 -- To view, visit http://gerrit.cloudera.org:8080/18850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I701652e457d3118f43249e83be933713b17ce48f Gerrit-Change-Number: 18850 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11494: Don't always produce Ranger audit log for authorized query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18850 ) Change subject: IMPALA-11494: Don't always produce Ranger audit log for authorized query .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8457/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I701652e457d3118f43249e83be933713b17ce48f Gerrit-Change-Number: 18850 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 17 Aug 2022 17:03:50 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-11500] Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num values is 0
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18854 ) Change subject: [IMPALA-11500] Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num_values is 0 .. Patch Set 1: (2 comments) Thanks for working on this! http://gerrit.cloudera.org:8080/#/c/18854/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18854/1//COMMIT_MSG@7 PS1, Line 7: [IMPALA-11500] nit: you shouldn't use brackets, first line should be: IMPALA-11500: http://gerrit.cloudera.org:8080/#/c/18854/1/be/src/exec/parquet/parquet-column-readers.cc File be/src/exec/parquet/parquet-column-readers.cc: http://gerrit.cloudera.org:8080/#/c/18854/1/be/src/exec/parquet/parquet-column-readers.cc@396 PS1, Line 396: if (num_values == 0) return true; Maybe the solution can be made a bit more future-proof if we make the bool decoder's SkipValues() function tolerate 0 values. So if we start using SkipValues() at other places we won't run into this crash again. Actually I think it's already working fine in case of 0, we just need to change the initial DCHECK_GT to DCHECK_GE(num_values, 0); -- To view, visit http://gerrit.cloudera.org:8080/18854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37 Gerrit-Change-Number: 18854 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 17 Aug 2022 11:32:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11494: Don't always produce Ranger audit log for authorized query
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18850 ) Change subject: IMPALA-11494: Don't always produce Ranger audit log for authorized query .. IMPALA-11494: Don't always produce Ranger audit log for authorized query Before this patch, when Impala could not resolve a given table '.' during the query analysis, Impala would still attempt to register 2 privilege requests. One was for the table '' under the database '' and the other was for the table '' under the database 'default'. The first one should be registered since Impala had to determine whether such an access should be allowed (even though in fact the table did not exist), whereas the second one was incorrect in that 'default.' definitely was not '.' in general. Furthermore, Impala always sent audit log entries to the Ranger server for an authorized query against non-existing table(s). The 2 facts described above resulted in Impala producing Ranger audit log entries for the tables '.' and 'default.' when a requesting user granted sufficient privileges on the databases of '' and 'default' submitted a query against a non-existing table '.'. None of the audit log entries should be generated because i) the privilege request for '.' was allowed and '.' did not exist, and ii) 'default.' did not correspond to a table. This patch fixes the 2 issues mentioned above so that Impala will not generate any Ranger audit log entry for an authorized query against a non-existing table. Testing: - Added a frontend test case to verify no Ranger audit log entry will be produced for an authorized query against a non-existing table. Change-Id: I701652e457d3118f43249e83be933713b17ce48f Reviewed-on: http://gerrit.cloudera.org:8080/18850 Reviewed-by: Csaba Ringhofer Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java 7 files changed, 75 insertions(+), 21 deletions(-) Approvals: Csaba Ringhofer: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/18850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I701652e457d3118f43249e83be933713b17ce48f Gerrit-Change-Number: 18850 Gerrit-PatchSet: 5 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR](asf-site) Update 4.1.0 release notes with more items
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18845 ) Change subject: Update 4.1.0 release notes with more items .. Patch Set 1: Code-Review+2 Thanks for adding these items! I checked the links and all looks good! -- To view, visit http://gerrit.cloudera.org:8080/18845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I44b85afa19806ef0c654eba267f40278d421868c Gerrit-Change-Number: 18845 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 17 Aug 2022 16:53:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9999: Switch to GCC 10.4
Hello Andrew Sherman, Abhishek Rawat, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18134 to look at the new patch set (#11). Change subject: IMPALA-: Switch to GCC 10.4 .. IMPALA-: Switch to GCC 10.4 This upgrades GCC and libstdc++ to version 10.4. This required patching or upgrading several dependencies so they could compile with GCC 10. The toolchain companion change has details on what items needed to be upgraded and why. The toolchain companion change switches GCC to build with toolchain binutils rather than host binutils. This means that the python virtualenv initialization needs to include binutils on the path. This disables two warnings introduced in the new GCC versions (Wclass-memaccess and Winit-list-lifetime). These two warnings occur in our code and also in dependencies like LLVM and rapidjson. These are not critical warnings, so they can be addressed independently and reenabled later. Binary sizes increase, particulary when including debug symbols: | GCC 7.5 | GCC 10.4 impalad RELEASE stripped | 83204768 | 88702824 impalad RELEASE | 707278904 | 971711456 impalad DEBUG stripped | 106677672 | 97391944 impalad DEBUG| 725864760 | 867647512 Testing: - Multiple test jobs (core, release exhaustive, ASAN) - Performance testing for TPC-H and TPC-DS shows a modest improvement (2-4%). - Code compiles without warnings on debug and release Change-Id: Ibe6857b822925226d39fd4d6413457ef6bbaabec --- M be/CMakeLists.txt M be/src/codegen/llvm-codegen-test.cc M be/src/runtime/string-value.inline.h M be/src/udf/udf.cc M be/src/util/parquet-reader.cc M bin/impala-config.sh M fe/pom.xml M infra/python/bootstrap_virtualenv.py 8 files changed, 54 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/18134/11 -- To view, visit http://gerrit.cloudera.org:8080/18134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe6857b822925226d39fd4d6413457ef6bbaabec Gerrit-Change-Number: 18134 Gerrit-PatchSet: 11 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] Test review
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18860 ) Change subject: Test review .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8458/ -- To view, visit http://gerrit.cloudera.org:8080/18860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id1c1742716c4922f125b20d719928e9a7b94ecc8 Gerrit-Change-Number: 18860 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 17 Aug 2022 23:45:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Omid Shahidi has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 7: (5 comments) > Patch Set 7: > > (4 comments) http://gerrit.cloudera.org:8080/#/c/18798/7/be/src/benchmarks/row-batch-serialize-benchmark.cc File be/src/benchmarks/row-batch-serialize-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/18798/7/be/src/benchmarks/row-batch-serialize-benchmark.cc@38 PS7, Line 38: : // Benchmark to measure how quickly we can serialize and deserialize row batches. More : // specifically, this benchmark was developed to measure the overhead of deduplication. : // The benchmarks are divided into serialization and deserialization benchmarks. : // The serialization benchmarks test different serialization methods (the new default of : // adjacent deduplication vs. the baseline of no deduplication) on row batches with : // different patterns of duplication: no_dups and adjacent_dups. : // For all benchmarks we use (int, string) tuples to exercise both variable-length and : // fixed-length slot handling. The small tuples with few slots emphasizes per-tuple : // dedup performance rather than per-slot serialization/deserialization performance. : // : // serialize:Function Rate (iters/ms) Comparison : // -- : // ser_no_dups_baseline 17.43 1X : // ser_no_dups 17.33 0.9944X : // ser_no_dups_full14.1 0.8092X : // : //ser_adjacent_dups_baseline 26.65 1X : // ser_adjacent_dups 63.98 2.4X : //ser_adjacent_dups_full 55.88 2.096X : // : // ser_dups_baseline 19.26 1X : // ser_dups 19.55 1.015X : // ser_dups_full32.4 1.682X : // : // deserialize: Function Rate (iters/ms) Comparison : // -- : //deser_no_dups_baseline 64.94 1X : // deser_no_dups 69.24 1.066X : // : // deser_adjacent_dups_baseline 112 1X : // deser_adjacent_dups 207.4 1.852X : // : // deser_dups_baseline 114.8 1X : //deser_dups 208.5 1.817X : // : // Earlier results with LossyHashTable : // serialize:Function Rate (iters/ms) Comparison : // -- : // ser_no_dups_lossy 15.93 0.9139X : // ser_adjacent_dups_lossy 58.21 2.184X : //ser_dups_lossy 50.46 2.62X : // : // Earlier results with boost::unordered_map : // serialize:Function Rate (iters/ms) Comparison : // -- : // ser_no_dups_full8.73 0.5582X : // : //ser_adjacent_dups_full38.7 1.634X : // : // ser_dups_full27.5 1.54X Should this be removed and updated with the current benchmark scores? http://gerrit.cloudera.org:8080/#/c/18798/7/be/src/runtime/row-batch.inline.h File be/src/runtime/row-batch.inline.h: http://gerrit.cloudera.org:8080/#/c/18798/7/be/src/runtime/row-batch.inline.h@22 PS7, Line 22: > extra line Done http://gerrit.cloudera.org:8080/#/c/18798/7/be/src/runtime/row-batch.inline.h@34 PS7, Line 34: for_compression > These three lines check for_compression, you can change code as: Done http://gerrit.cloudera.org:8080/#/c/18798/7/testdata/workloads/tpch/queries/datastream-sender.test File testdata/workloads/tpch/queries/datastream-sender.test:
[Impala-ASF-CR] Test review
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18860 ) Change subject: Test review .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8458/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id1c1742716c4922f125b20d719928e9a7b94ecc8 Gerrit-Change-Number: 18860 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 17 Aug 2022 23:45:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num values is 0
lipeng...@sensorsdata.cn has posted comments on this change. (
http://gerrit.cloudera.org:8080/18854 )
Change subject: IMPALA-11500: Fix Impalad crashed in
ParquetBoolDecoder::SkipValues when num_values is 0
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18854/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18854/1//COMMIT_MSG@7
PS1, Line 7: IMPALA-11500: F
> nit: you shouldn't use brackets, first line should be:
That is my mistakes! Subconsciously using the style of the company I work for.
http://gerrit.cloudera.org:8080/#/c/18854/1/be/src/exec/parquet/parquet-column-readers.cc
File be/src/exec/parquet/parquet-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/18854/1/be/src/exec/parquet/parquet-column-readers.cc@396
PS1, Line 396: if (bool_decoder_) {
> Maybe the solution can be made a bit more future-proof if we make the bool
Done
--
To view, visit http://gerrit.cloudera.org:8080/18854
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37
Gerrit-Change-Number: 18854
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 18 Aug 2022 02:58:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18817 ) Change subject: IMPALA-11207: Use hadoop-cloud-storage for Cloud dependencies .. Patch Set 2: I don't think hadoop-cos is available in the versions of hadoop-cloud-storage we pull in yet, so I plan to leave it for now. This seems ready to merge. -- To view, visit http://gerrit.cloudera.org:8080/18817 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a1631289f990513823c2b17eb9241cc1b5a7ffd Gerrit-Change-Number: 18817 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 15 Aug 2022 16:06:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num values is 0
lipeng...@sensorsdata.cn has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18854 ) Change subject: IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num_values is 0 .. IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num_values is 0 Fix Impalad crashed in the method ParquetBoolDecoder::SkipValues when the parameter 'num_values' is 0. The function should tolerate that the 'num_values' is 0 values. Testing: - Add e2e tests Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37 --- M be/src/exec/parquet/parquet-bool-decoder.cc A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-bugfix.test M tests/query_test/test_parquet_stats.py 3 files changed, 35 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/18854/3 -- To view, visit http://gerrit.cloudera.org:8080/18854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37 Gerrit-Change-Number: 18854 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num values is 0
lipeng...@sensorsdata.cn has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18854 ) Change subject: IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num_values is 0 .. IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num_values is 0 Fix Impalad crashed in the method ParquetBoolDecoder::SkipValues when the parameter 'num_values' is 0. In the method ParquetBoolDecoder::SkipValues, if the parameter 'num_values' is 0 return true without other logic. Testing: - Add e2e tests Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37 --- M be/src/exec/parquet/parquet-bool-decoder.cc A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-bugfix.test M tests/query_test/test_parquet_stats.py 3 files changed, 35 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/18854/2 -- To view, visit http://gerrit.cloudera.org:8080/18854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37 Gerrit-Change-Number: 18854 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10800: Tidy up the be/src/exec directory
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18815 ) Change subject: IMPALA-10800: Tidy up the be/src/exec directory .. Patch Set 3: Code-Review+2 Thanks for working on this change! -- To view, visit http://gerrit.cloudera.org:8080/18815 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie936c400ea8b112073bba892497ab8a1498c418d Gerrit-Change-Number: 18815 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 15 Aug 2022 10:55:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9482: Support for BINARY columns
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/16066 )
Change subject: IMPALA-9482: Support for BINARY columns
..
Patch Set 22:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/16066/14/be/src/runtime/descriptors.h
File be/src/runtime/descriptors.h:
http://gerrit.cloudera.org:8080/#/c/16066/14/be/src/runtime/descriptors.h@256
PS14, Line 256: return col_descs_[slot_desc->col_path().back()];
> Change the analyzer to disallow complex types in select list if they have b
Thanks for digging into this!
Sorry that my initial confusion is that "col_descs_" are the top-level columns
of the table, but the last item in SchemaPath is not always the top-level
column index. Usually the first item of SchemaPath is the top-level column
index, and the next items are the index inside the nested type. E.g. the 6th
column in table complextypestbl is
nested_struct struct<
a: int,
b: array,
...
>
If the query selects "nested_struct.a" in the SelectList, the corresponding
SchemaPath is [5, 0]. Here [5] is the SchemaPath of "nested_struct". But we are
using 0 (the last item) here as the index of col_descs_. So I hope we can add a
test of selecting the binary column directly inside a struct top level
column.
Maybe I've missed something. Just explaning my confusion.
http://gerrit.cloudera.org:8080/#/c/16066/14/fe/src/main/java/org/apache/impala/analysis/LikePredicate.java
File fe/src/main/java/org/apache/impala/analysis/LikePredicate.java:
http://gerrit.cloudera.org:8080/#/c/16066/14/fe/src/main/java/org/apache/impala/analysis/LikePredicate.java@116
PS14, Line 116: private static boolean isLikeableType(Type type) {
> :D poor other types - tbh I don't remember whether the naming was intention
haha
http://gerrit.cloudera.org:8080/#/c/16066/22/testdata/bin/generate-schema-statements.py
File testdata/bin/generate-schema-statements.py:
http://gerrit.cloudera.org:8080/#/c/16066/22/testdata/bin/generate-schema-statements.py@222
PS22, Line 222: 'BINARY': 'bytes'
nit: it'd be nice to add a trailing comma so future changes don't need to touch
this line.
http://gerrit.cloudera.org:8080/#/c/16066/22/testdata/datasets/functional/functional_schema_template.sql
File testdata/datasets/functional/functional_schema_template.sql:
http://gerrit.cloudera.org:8080/#/c/16066/22/testdata/datasets/functional/functional_schema_template.sql@3532
PS22, Line 3532: binary_in_complex_types
Can we add some data to this table and add some e2e tests? e.g.
select binary_member_col.b from binary_in_complex_types;
select a.item from binary_in_complex_types t, t.binary_item_col;
select m.key from binary_in_complex_types t, t.binary_key_col;
select m.value from binary_in_complex_types t, t.binary_value_col;
--
To view, visit http://gerrit.cloudera.org:8080/16066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36861a9ca6c2047b0d76862507c86f7f153bc582
Gerrit-Change-Number: 16066
Gerrit-PatchSet: 22
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Steve Carlin
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 18 Aug 2022 09:43:26 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9999: Switch to GCC 10.4
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18134 ) Change subject: IMPALA-: Switch to GCC 10.4 .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/18134/10/be/src/runtime/string-value.inline.h File be/src/runtime/string-value.inline.h: http://gerrit.cloudera.org:8080/#/c/18134/10/be/src/runtime/string-value.inline.h@41 PS10, Line 41: // GCC gives a warning about overflowing the size argument of memcmp, because > The int type is coming from StringValue/StringVal's len field. These are us I like using len <= 0 to cover it. -- To view, visit http://gerrit.cloudera.org:8080/18134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe6857b822925226d39fd4d6413457ef6bbaabec Gerrit-Change-Number: 18134 Gerrit-PatchSet: 11 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 18 Aug 2022 05:26:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9999: Switch to GCC 10.4
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18134 ) Change subject: IMPALA-: Switch to GCC 10.4 .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/18134/10/be/src/runtime/string-value.inline.h File be/src/runtime/string-value.inline.h: http://gerrit.cloudera.org:8080/#/c/18134/10/be/src/runtime/string-value.inline.h@41 PS10, Line 41: // GCC gives a warning about overflowing the size argument of memcmp, because > This isn't something we want to update to use an unsigned int? The int type is coming from StringValue/StringVal's len field. These are used all over the place, and StringVal is part of our UDF interface. My general read is that switching to unsigned would be a lot of work that doesn't get us a lot. I'm open to ideas. Another alternative is to have: const int result = (len <= 0) ? 0 : memcmp(s1, s2, len) -- To view, visit http://gerrit.cloudera.org:8080/18134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe6857b822925226d39fd4d6413457ef6bbaabec Gerrit-Change-Number: 18134 Gerrit-PatchSet: 10 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 18 Aug 2022 03:49:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Omid Shahidi has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 8: Added benchmark results to commit message for patch 9 -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 8 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 18 Aug 2022 03:49:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11472: Reduce test dimensions for TestClientSsl
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18843 ) Change subject: IMPALA-11472: Reduce test dimensions for TestClientSsl .. Patch Set 1: Code-Review+1 Jenkins error is in an unrelated test case. -- To view, visit http://gerrit.cloudera.org:8080/18843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d4a4792a37cba2231de8bfa2279ba029a05 Gerrit-Change-Number: 18843 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 15 Aug 2022 15:54:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11471: Track disk usage for build-all-flag-combinations.sh
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18810 to look at the new patch set (#2). Change subject: IMPALA-11471: Track disk usage for build-all-flag-combinations.sh .. IMPALA-11471: Track disk usage for build-all-flag-combinations.sh This adds some calls to df and du to track disk space usage throughout the builds. This also cleans up the Impala dev environment before creating the m2 archive. Change-Id: I8ab31d8d7096b49d8404edf7521d46f23155526f --- M bin/jenkins/build-all-flag-combinations.sh 1 file changed, 17 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/18810/2 -- To view, visit http://gerrit.cloudera.org:8080/18810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8ab31d8d7096b49d8404edf7521d46f23155526f Gerrit-Change-Number: 18810 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11494: Don't always produce Ranger audit log for authorized query
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/18850 ) Change subject: IMPALA-11494: Don't always produce Ranger audit log for authorized query .. Patch Set 1: Hi all, please let me know if you have any comment on the patch. Thank you very much for the help! -- To view, visit http://gerrit.cloudera.org:8080/18850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I701652e457d3118f43249e83be933713b17ce48f Gerrit-Change-Number: 18850 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 15 Aug 2022 16:39:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11472: Reduce test dimensions for TestClientSsl
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18843 ) Change subject: IMPALA-11472: Reduce test dimensions for TestClientSsl .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8447/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d4a4792a37cba2231de8bfa2279ba029a05 Gerrit-Change-Number: 18843 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 15 Aug 2022 16:46:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11178/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 9 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 18 Aug 2022 04:14:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11498: Change port range of TEZ's web UI server
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18851 Change subject: IMPALA-11498: Change port range of TEZ's web UI server .. IMPALA-11498: Change port range of TEZ's web UI server After TEZ-4347, by default TEZ would attempt to start a web UI server before opening a session. The default port range for the server specified in TezConfiguration.java (in the TEZ repository) is "5-50050", which does not seem to be a good choice in Impala's testing environment in that there are always some other client programs holding those ports when TEZ attempts to start its web UI server. As a result, TEZ could not bind a port in the port range to start its web UI server, resulting in TEZ session not being created, which in turn fail some end-to-end test relying on TEZ, e.g., test_json_file_unsupported. This patch explicitly sets the port range to "32000-32100" to avoid port collision based on the parameter of 'ip_local_port_range', which is "32768-60999" on the machine where Impala's tests are run. Testing: - Verified in 2 Jenkins runs of exhaustive RELEASE build that we do not see test_json_file_unsupported fail again after this patch. Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791 --- M fe/src/test/resources/hive-site.xml.py 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/18851/1 -- To view, visit http://gerrit.cloudera.org:8080/18851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791 Gerrit-Change-Number: 18851 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Omid Shahidi has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. IMPALA-6684: Fix untracked memory in KRPC During serialization of an row batch header, a tuple_data_ is created which will hold the compressed tuple data for an outbound row batch. We would like this tuple data to be trackable as it is responsible for a significant portion of untrackable memory from the krpc data stream sender. By using free pool, we are able to allocate tuple data and compression scratch and account for it in the memory tracker of the KrpcDataStreamSender. This solution creates a RAII class responsible for memory allocation and changes the existing code to use a char buffer pointed by a char* tuple_data_ instead of the previously used std::string tuple_data_. The thrift implementation is left unchanged and the protobuf implementation is seperated. Testing: - Passed core tests. - Ran a single node benchmark which shows no regression. - Updated row-batch-serialize-test and row-batch-serialize-benchmark to test the row-batch serialization used by KRPC. - Manually collected query-profile, heap growth, and memory usage log showing untracked memory decreased by 1/2. - Add end-end unit-test to verify the new counters in runtime profile serialize: Func10% 50% 90% 10% 50% 90% ile (rel) (rel) (rel) --- ser_no_dups_baseline8.36 8.6 8.7 1X 1X 1X ser_no_dups 6.73 6.85 6.93 0.804X 0.796X 0.796X ser_no_dups_full5.28 5.38 5.55 0.631X 0.625X 0.637X ser_adjacent_dups_baseline 12.9 13.2 13.4 1X 1X 1X ser_adjacent_dups 23.2 23.7 24.1 1.8X 1.8X 1.8X ser_adjacent_dups_full 19.9 20.3 20.7 1.54X 1.54X 1.55X ser_dups_baseline 9.17 9.54 9.72 1X 1X 1X ser_dups7.45 7.69 7.86 0.812X 0.806X 0.809X ser_dups_full 14.6 15 15.3 1.6X 1.57X 1.57X deserialize: Func10% 50% 90% 10% 50% 90% ile (rel) (rel) (rel) --- deser_no_dups_baseline 32.6 33.5 34 1X 1X1X deser_no_dups 32.5 33.1 33.7 0.999X 0.99X 0.992X deser_adjacent_dups_baseline 53.1 54 54.7 1X 1X 1X deser_adjacent_dups 80.3 81.6 82.5 1.51X 1.51X 1.51X deser_dups_baseline 52.4 54 54.7 1X 1X 1X deser_dups 86.8 88.4 89.7 1.66X 1.64X 1.64X Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 --- M be/src/benchmarks/row-batch-serialize-benchmark.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h A be/src/runtime/row-batch.inline.h A testdata/workloads/functional-query/queries/datastream-sender.test A testdata/workloads/tpch/queries/datastream-sender.test A tests/query_test/test_datastream_sender.py 10 files changed, 657 insertions(+), 173 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/18798/9 -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 9 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num values is 0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18854 ) Change subject: IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num_values is 0 .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11174/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37 Gerrit-Change-Number: 18854 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 18 Aug 2022 03:25:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11471: Track disk usage for build-all-flag-combinations.sh
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18810 ) Change subject: IMPALA-11471: Track disk usage for build-all-flag-combinations.sh .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/18810/1/bin/jenkins/build-all-flag-combinations.sh File bin/jenkins/build-all-flag-combinations.sh: http://gerrit.cloudera.org:8080/#/c/18810/1/bin/jenkins/build-all-flag-combinations.sh@166 PS1, Line 166: ./bin/clean.sh > This won't remove toolchain. Should we remove it as well? The bulk of the space usage is in the be/build directory, so I updated this to remove only that. We could remove the toolchain, but removing the be/build directory gives us plenty of headroom for the disk usage. -- To view, visit http://gerrit.cloudera.org:8080/18810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ab31d8d7096b49d8404edf7521d46f23155526f Gerrit-Change-Number: 18810 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 18 Aug 2022 03:39:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num values is 0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18854 ) Change subject: IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num_values is 0 .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8459/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37 Gerrit-Change-Number: 18854 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 18 Aug 2022 03:43:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11498: Change port range of TEZ's web UI server
Fang-Yu Rao has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18851 ) Change subject: IMPALA-11498: Change port range of TEZ's web UI server .. IMPALA-11498: Change port range of TEZ's web UI server After TEZ-4347, by default TEZ would attempt to start a web UI server before opening a session. The default port range for the server specified in TezConfiguration.java (in the TEZ repository) is "5-50050", which does not seem to be a good choice in Impala's testing environment in that there are always some other client programs holding those ports when TEZ attempts to start its web UI server. As a result, TEZ could not bind a port in the port range to start its web UI server, resulting in TEZ session not being created, which in turn failed some end-to-end test relying on TEZ, e.g., test_json_file_unsupported. This patch explicitly sets the port range to "32000-32100" to avoid port collision based on the parameter of 'ip_local_port_range', which is "32768-60999" on the machine where Impala's tests are run. Testing: - Verified in 2 Jenkins runs of exhaustive RELEASE build that we do not see test_json_file_unsupported fail again after this patch. Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791 --- M fe/src/test/resources/hive-site.xml.py 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/18851/2 -- To view, visit http://gerrit.cloudera.org:8080/18851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791 Gerrit-Change-Number: 18851 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-11469: (Addendum) Initialize BackendConfig
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18840 ) Change subject: IMPALA-11469: (Addendum) Initialize BackendConfig .. Patch Set 1: Code-Review+2 Thanks for fixing the test issue! I prefer this one to the other patch (https://gerrit.cloudera.org/c/18828/) since the FileSystemUtil changes in it is not related to the failure. For more details about how BackendConfig is initialized in FrontendTestBase/AbstractFrontendTest, see the stacktrace here: https://gerrit.cloudera.org/c/18828/3/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java#43 -- To view, visit http://gerrit.cloudera.org:8080/18840 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaefea5ac982e42a8b0f3b693d16a8db10cc7de31 Gerrit-Change-Number: 18840 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 18 Aug 2022 06:53:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num values is 0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18854 ) Change subject: IMPALA-11500: Fix Impalad crashed in ParquetBoolDecoder::SkipValues when num_values is 0 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11173/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c4c5a4dff9e9e75913c7b524b4ae70967febb37 Gerrit-Change-Number: 18854 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 18 Aug 2022 03:18:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11281: Load table metadata for ResetMetadataStmt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18561 ) Change subject: IMPALA-11281: Load table metadata for ResetMetadataStmt .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8449/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c90b413974223886661697f11844d99a68fdebf Gerrit-Change-Number: 18561 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 16 Aug 2022 12:37:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11281: Load table metadata for ResetMetadataStmt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18561 ) Change subject: IMPALA-11281: Load table metadata for ResetMetadataStmt .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c90b413974223886661697f11844d99a68fdebf Gerrit-Change-Number: 18561 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 16 Aug 2022 12:37:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11471: Track disk usage for build-all-flag-combinations.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18810 ) Change subject: IMPALA-11471: Track disk usage for build-all-flag-combinations.sh .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8460/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ab31d8d7096b49d8404edf7521d46f23155526f Gerrit-Change-Number: 18810 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 18 Aug 2022 06:03:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9442: (Addendum) Fix Ozone jar name for older Ozone
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18852 Change subject: IMPALA-9442: (Addendum) Fix Ozone jar name for older Ozone .. IMPALA-9442: (Addendum) Fix Ozone jar name for older Ozone Ozone 1.2 changed the name of its hadoop adapter jar. Logic to link against the jar in pom.xml was correctly updated, but the jar name used in HADOOP_CLASSPATH assumed Ozone 1.2. Updates the jar in HADOOP_CLASSPATH to use the correct jar for the detected Ozone version. Change-Id: I7e401075268ebd81af8b8cac72504f0d1a4e59f1 --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/18852/1 -- To view, visit http://gerrit.cloudera.org:8080/18852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7e401075268ebd81af8b8cac72504f0d1a4e59f1 Gerrit-Change-Number: 18852 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith
[Impala-ASF-CR] IMPALA-9999: Switch to GCC 10.4
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18134 ) Change subject: IMPALA-: Switch to GCC 10.4 .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11177/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe6857b822925226d39fd4d6413457ef6bbaabec Gerrit-Change-Number: 18134 Gerrit-PatchSet: 11 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 18 Aug 2022 04:02:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11471: Track disk usage for build-all-flag-combinations.sh
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18810 ) Change subject: IMPALA-11471: Track disk usage for build-all-flag-combinations.sh .. Patch Set 2: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/18810/1/bin/jenkins/build-all-flag-combinations.sh File bin/jenkins/build-all-flag-combinations.sh: http://gerrit.cloudera.org:8080/#/c/18810/1/bin/jenkins/build-all-flag-combinations.sh@166 PS1, Line 166: if [[ "$GENERATE_M2_ARCHIVE" == true ]]; then > The bulk of the space usage is in the be/build directory, so I updated this Ack -- To view, visit http://gerrit.cloudera.org:8080/18810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ab31d8d7096b49d8404edf7521d46f23155526f Gerrit-Change-Number: 18810 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 18 Aug 2022 06:03:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11471: Track disk usage for build-all-flag-combinations.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18810 ) Change subject: IMPALA-11471: Track disk usage for build-all-flag-combinations.sh .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11176/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ab31d8d7096b49d8404edf7521d46f23155526f Gerrit-Change-Number: 18810 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 18 Aug 2022 03:59:29 + Gerrit-HasComments: No
