[Impala-ASF-CR] [tools] fix buildall.sh -testdata with prior data
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18522 ) Change subject: [tools] fix buildall.sh -testdata with prior data .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8104/ -- To view, visit http://gerrit.cloudera.org:8080/18522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc4ec4bb746abf53f6787fce4db493919806aaa9 Gerrit-Change-Number: 18522 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 14 May 2022 05:14:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11274: CNF Rewrite causes a regress in join node performance
Aman Sinha has posted comments on this change. (
http://gerrit.cloudera.org:8080/18458 )
Change subject: IMPALA-11274: CNF Rewrite causes a regress in join node
performance
..
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18458/7/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java:
http://gerrit.cloudera.org:8080/#/c/18458/7/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@605
PS7, Line 605: public void TestFeasibleToConvertToCNF() {
> TestFeasibleToConvertToCNF passes. I'd appreciate thoughts on examples for
I added 2 tests to convert-to-cnf.test. The first one which uses UPPER function
is expected to not be converted to CNF and it does not .. so this is the
expected behavior. However, the second query uses CAST function which I would
expect to be converted to CNF. But it does not.
# IMPALA-11274: Test with string functions in the disjunctive predicate.
# In this case the predicate is not converted to CNF
select count(*) from lineitem, orders
where l_orderkey = o_orderkey and
((upper(l_returnflag) = 'Y' and upper(o_orderpriority) = 'HIGH')
or (upper(l_returnflag) = 'N' and upper(o_orderpriority) = 'LOW'))
and l_partkey > 0;
# IMPALA-11274: Functions like CAST should still be eligible for CNF
select count(*) from lineitem, orders
where l_orderkey = o_orderkey and
((cast(l_returnflag as varchar(2)) = 'Y' and cast(o_orderpriority as
varchar(5)) = 'HIGH')
or (cast(l_returnflag as varchar(2)) = 'N' and cast(o_orderpriority as
varchar(5)) = 'LOW'))
and l_partkey > 0;
--
To view, visit http://gerrit.cloudera.org:8080/18458
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I326406c6b004fe31ec0e2a2f390a3845b8925aa9
Gerrit-Change-Number: 18458
Gerrit-PatchSet: 9
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Sat, 14 May 2022 02:13:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [tools] fix buildall.sh -testdata with prior data
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18522 ) Change subject: [tools] fix buildall.sh -testdata with prior data .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc4ec4bb746abf53f6787fce4db493919806aaa9 Gerrit-Change-Number: 18522 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 14 May 2022 00:47:48 + Gerrit-HasComments: No
[Impala-ASF-CR] [tools] fix buildall.sh -testdata with prior data
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18522 ) Change subject: [tools] fix buildall.sh -testdata with prior data .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8104/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc4ec4bb746abf53f6787fce4db493919806aaa9 Gerrit-Change-Number: 18522 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 14 May 2022 00:48:21 + Gerrit-HasComments: No
[Impala-ASF-CR] [tools] fix buildall.sh -testdata with prior data
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18522 ) Change subject: [tools] fix buildall.sh -testdata with prior data .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc4ec4bb746abf53f6787fce4db493919806aaa9 Gerrit-Change-Number: 18522 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 14 May 2022 00:48:20 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-11264: (Addendum) Bump ORC to 1.7.0-p14 to fix ORC-1159
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18525 ) Change subject: IMPALA-11264: (Addendum) Bump ORC to 1.7.0-p14 to fix ORC-1159 .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I726e84b85f3713fc54165007d63a21dab2af84f2 Gerrit-Change-Number: 18525 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Sat, 14 May 2022 00:40:22 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-11264: (Addendum) Bump ORC to 1.7.0-p14 to fix ORC-1159
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18525 ) Change subject: IMPALA-11264: (Addendum) Bump ORC to 1.7.0-p14 to fix ORC-1159 .. IMPALA-11264: (Addendum) Bump ORC to 1.7.0-p14 to fix ORC-1159 This patch adds ORC-1160 and ORC-1159 which fixes the bug introduced by ORC-969. Change-Id: I726e84b85f3713fc54165007d63a21dab2af84f2 Reviewed-on: http://gerrit.cloudera.org:8080/18525 Reviewed-by: Joe McDonnell Tested-by: Quanlong Huang --- M buildall.sh A source/orc/orc-1.7.0-patches/0013-ORC-1160-C-Fix-seekToRow-can-t-seek-within-selected-.patch A source/orc/orc-1.7.0-patches/0014-ORC-1159-C-Fix-crash-when-the-last-stripe-is-skipped.patch 3 files changed, 518 insertions(+), 1 deletion(-) Approvals: Joe McDonnell: Looks good to me, approved Quanlong Huang: Verified -- To view, visit http://gerrit.cloudera.org:8080/18525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I726e84b85f3713fc54165007d63a21dab2af84f2 Gerrit-Change-Number: 18525 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-11283: Push-down IS NULL and NOT NULL predicates to iceberg
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18504 ) Change subject: IMPALA-11283: Push-down IS_NULL and NOT_NULL predicates to iceberg .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c3608af67b552bebc55dcc5526f61f5439967bf Gerrit-Change-Number: 18504 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 13 May 2022 23:10:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11283: Push-down IS NULL and NOT NULL predicates to iceberg
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18504 ) Change subject: IMPALA-11283: Push-down IS_NULL and NOT_NULL predicates to iceberg .. IMPALA-11283: Push-down IS_NULL and NOT_NULL predicates to iceberg This patch implements pushing the IS_NULL and NOT_NULL predicates down to Iceberg. Testing: - Added end-to-end test Change-Id: I9c3608af67b552bebc55dcc5526f61f5439967bf Reviewed-on: http://gerrit.cloudera.org:8080/18504 Reviewed-by: Zoltan Borok-Nagy Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-is-null-predicate-push-down.test M tests/query_test/test_iceberg.py 3 files changed, 427 insertions(+), 0 deletions(-) Approvals: Zoltan Borok-Nagy: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/18504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9c3608af67b552bebc55dcc5526f61f5439967bf Gerrit-Change-Number: 18504 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 8: We can use UDS address as file path or address in the UNIX domain "abstract namespace" (a leading '@' indicates "abstract namespace"). I found that UDS addresses are not released immediately by system when impala daemons are killed if we use UDS address in the form of file path. This cause socket binding failures when restarting impala daemons. Don't see this issue with UDS address in the form of address in abstract namespace. So we use UDS address as address in abstract namespace. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 13 May 2022 21:41:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11275: log thread info during minidump
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18508 ) Change subject: IMPALA-11275: log thread info during minidump .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea2bdf10db29a0f8ccbe5e767b708781d42a9b8a Gerrit-Change-Number: 18508 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 13 May 2022 21:09:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11275: log thread info during minidump
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18508 ) Change subject: IMPALA-11275: log thread info during minidump .. IMPALA-11275: log thread info during minidump Writes ThreadDebugInfo to stdout/stderr when a minidump is generated to capture thread and query details related to the dump. Example message: > Minidump in thread [1790536]async-exec-thread running query 1a47cc1e2df94cb4:88dfa082, fragment instance : Refactors DumpCallback so that repeated writes to STDOUT/STDERR are less redundant. Adds unit tests to run with ThreadDebugInfo. Removes the 'static' prefix from DumpCallback so it can be invoked from unit tests, but doesn't add it to the header as it's intended to be for internal use. Testing: - Added crash to Coordinator::Exec and manually tested dump handling. - Added a new unit test for DumpCallback. - Ran tests/custom_cluster/test_breakpad.py to verify nothing broke in refactor. Those tests don't have ThreadDebugInfo available. Change-Id: Iea2bdf10db29a0f8ccbe5e767b708781d42a9b8a Reviewed-on: http://gerrit.cloudera.org:8080/18508 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/util/CMakeLists.txt M be/src/util/debug-util-test.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h A be/src/util/minidump-test.cc M be/src/util/minidump.cc M tests/custom_cluster/test_breakpad.py 7 files changed, 190 insertions(+), 17 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iea2bdf10db29a0f8ccbe5e767b708781d42a9b8a Gerrit-Change-Number: 18508 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11274: CNF Rewrite causes a regress in join node performance
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18458 ) Change subject: IMPALA-11274: CNF Rewrite causes a regress in join node performance .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10575/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326406c6b004fe31ec0e2a2f390a3845b8925aa9 Gerrit-Change-Number: 18458 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 13 May 2022 19:19:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11274: CNF Rewrite causes a regress in join node performance
Michael Smith has uploaded a new patch set (#9) to the change originally
created by Qifan Chen. ( http://gerrit.cloudera.org:8080/18458 )
Change subject: IMPALA-11274: CNF Rewrite causes a regress in join node
performance
..
IMPALA-11274: CNF Rewrite causes a regress in join node performance
This patch defines a subset of all predicates that are common and
relatively inexpensive to compute. Such predicates must involve
columns, constants, simple math or cast functions only.
Examples of the subset of the predicates allowed:
1. (a = 1 AND cast(b as int) = 2) OR (c = d AND e = f)
2. a in ('1', '2', '3') OR ((b = 'abc') AND (c = d))
3. (a between 1 and 100) OR ((b is null) AND (c = d))
Examples of the predicates not allowed:
1. (upper(a) != 'Y') AND b = 2) OR (c = d AND e = f)
2. (coalesce(CAST(a AS string), '') = '') AND b = 2) OR
(c = d AND e = f)
This patch further restricts the predicates to be converted to
conjunctive normal form (CNF) to be such a subset, with the aim to
reduce the run-time evaluation overhead of CNFs in which some
of the predicates can be duplicated.
Testing:
1. Added a new test TestFeasibleToConvertToCNF() in ExprRewriterTest.java
2. Ran fe tests
3. perf-AB-test [TBD]
Change-Id: I326406c6b004fe31ec0e2a2f390a3845b8925aa9
---
M fe/src/main/java/org/apache/impala/analysis/ExistsPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/LikePredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java
M fe/src/main/java/org/apache/impala/rewrite/ConvertToCNFRule.java
M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java
M
testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
11 files changed, 124 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/18458/9
--
To view, visit http://gerrit.cloudera.org:8080/18458
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I326406c6b004fe31ec0e2a2f390a3845b8925aa9
Gerrit-Change-Number: 18458
Gerrit-PatchSet: 9
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10574/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 13 May 2022 18:55:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11283: Push-down IS NULL and NOT NULL predicates to iceberg
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18504 ) Change subject: IMPALA-11283: Push-down IS_NULL and NOT_NULL predicates to iceberg .. Patch Set 6: Code-Review+2 Great work again! -- To view, visit http://gerrit.cloudera.org:8080/18504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c3608af67b552bebc55dcc5526f61f5439967bf Gerrit-Change-Number: 18504 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 13 May 2022 18:44:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11283: Push-down IS NULL and NOT NULL predicates to iceberg
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18504 ) Change subject: IMPALA-11283: Push-down IS_NULL and NOT_NULL predicates to iceberg .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8103/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c3608af67b552bebc55dcc5526f61f5439967bf Gerrit-Change-Number: 18504 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 13 May 2022 18:44:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS address. It could be 'ip_address', 'backend_id', or 'none'. - Add field uds_address in NetworkAddressPB. Replace TNetworkAddress with NetworkAddressPB for RPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization with unique_id specified by starting flag FLAGS_uds_address_unique_id. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. - rpcz Web page show TCP related stats as 'N/A' when using UDS. Show remote UDS address for KRPC inbound connections on rpcz Web page as '*' when using UDS since the remote UDS addresses are not available. - Add new unit-tests for UDS. TODO: - Advertise BackendId of admissiond in global admission control mode. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto A tests/custom_cluster/test_krpc_socket.py M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py M www/rpcz.tmpl 38 files changed, 593 insertions(+), 138 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/8 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10745 (part 2): Support Kerberos over HTTP for impala-shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18523 ) Change subject: IMPALA-10745 (part 2): Support Kerberos over HTTP for impala-shell .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10573/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id34074cbe725ba2cf1407fcf59e00475cd417a6d Gerrit-Change-Number: 18523 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 13 May 2022 17:56:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11274: CNF Rewrite causes a regress in join node performance
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/18458 )
Change subject: IMPALA-11274: CNF Rewrite causes a regress in join node
performance
..
Patch Set 8:
(2 comments)
I'm working on updating this patch.
http://gerrit.cloudera.org:8080/#/c/18458/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18458/3//COMMIT_MSG@32
PS3, Line 32: 2. Core tests;
> Definitely need to check that the transform is still applied in improved ca
https://github.com/apache/impala/blob/master/testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q13.test
was originally added for that ticket and is unchanged by this patch.
http://gerrit.cloudera.org:8080/#/c/18458/7/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java:
http://gerrit.cloudera.org:8080/#/c/18458/7/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@605
PS7, Line 605: public void TestFeasibleToConvertToCNF() {
> I added two more tests on AND and OR but was not able to test it out as my
TestFeasibleToConvertToCNF passes. I'd appreciate thoughts on examples for
convert-to-cnf.test.
--
To view, visit http://gerrit.cloudera.org:8080/18458
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I326406c6b004fe31ec0e2a2f390a3845b8925aa9
Gerrit-Change-Number: 18458
Gerrit-PatchSet: 8
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Fri, 13 May 2022 17:46:55 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10745 (part 2): Support Kerberos over HTTP for impala-shell
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18523 ) Change subject: IMPALA-10745 (part 2): Support Kerberos over HTTP for impala-shell .. IMPALA-10745 (part 2): Support Kerberos over HTTP for impala-shell This patch adds kerberos-1.3.1 Python module to shell/ext-py so that the egg file of Kerberos module is built and added into impala-shell tarball when running script shell/make_shell_tarball.sh. Kerberos Python module is distributed under Apache License Version 2. Its source distribution is available at: https://pypi.org/project/kerberos/ Testing: - Passed core run. - Installed impala-shell from impala-shell tarball on dev box as standalone package. Verified that impala-shell could be ran without additional configurations. - Installed impala-shell from impala-shell tarball on a real cluster with a full Kerberos setup. Verified that impala-shell could connect to impala server with options "-k --protocol=hs2-http". Change-Id: Id34074cbe725ba2cf1407fcf59e00475cd417a6d --- M bin/rat_exclude_files.txt A shell/ext-py/kerberos-1.3.1/MANIFEST.in A shell/ext-py/kerberos-1.3.1/PKG-INFO A shell/ext-py/kerberos-1.3.1/README.md A shell/ext-py/kerberos-1.3.1/kerberos.egg-info/PKG-INFO A shell/ext-py/kerberos-1.3.1/kerberos.egg-info/SOURCES.txt A shell/ext-py/kerberos-1.3.1/kerberos.egg-info/dependency_links.txt A shell/ext-py/kerberos-1.3.1/kerberos.egg-info/top_level.txt A shell/ext-py/kerberos-1.3.1/pysrc/kerberos.py A shell/ext-py/kerberos-1.3.1/setup.cfg A shell/ext-py/kerberos-1.3.1/setup.py A shell/ext-py/kerberos-1.3.1/src/base64.c A shell/ext-py/kerberos-1.3.1/src/base64.h A shell/ext-py/kerberos-1.3.1/src/kerberos.c A shell/ext-py/kerberos-1.3.1/src/kerberosbasic.c A shell/ext-py/kerberos-1.3.1/src/kerberosbasic.h A shell/ext-py/kerberos-1.3.1/src/kerberosgss.c A shell/ext-py/kerberos-1.3.1/src/kerberosgss.h A shell/ext-py/kerberos-1.3.1/src/kerberospw.c A shell/ext-py/kerberos-1.3.1/src/kerberospw.h 20 files changed, 3,603 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18523/2 -- To view, visit http://gerrit.cloudera.org:8080/18523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id34074cbe725ba2cf1407fcf59e00475cd417a6d Gerrit-Change-Number: 18523 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11275: log thread info during minidump
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18508 ) Change subject: IMPALA-11275: log thread info during minidump .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8102/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea2bdf10db29a0f8ccbe5e767b708781d42a9b8a Gerrit-Change-Number: 18508 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 13 May 2022 16:41:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11275: log thread info during minidump
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18508 ) Change subject: IMPALA-11275: log thread info during minidump .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea2bdf10db29a0f8ccbe5e767b708781d42a9b8a Gerrit-Change-Number: 18508 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 13 May 2022 16:41:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11275: log thread info during minidump
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18508 ) Change subject: IMPALA-11275: log thread info during minidump .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea2bdf10db29a0f8ccbe5e767b708781d42a9b8a Gerrit-Change-Number: 18508 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 13 May 2022 16:40:24 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-11264: (Addendum) Bump ORC to 1.7.0-p14 to fix ORC-1159
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18525 ) Change subject: IMPALA-11264: (Addendum) Bump ORC to 1.7.0-p14 to fix ORC-1159 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I726e84b85f3713fc54165007d63a21dab2af84f2 Gerrit-Change-Number: 18525 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Fri, 13 May 2022 15:53:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11205: Implement Statistical functions : CORR(), COVAR SAMP() and COVAR POP()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18413 ) Change subject: IMPALA-11205: Implement Statistical functions : CORR(), COVAR_SAMP() and COVAR_POP() .. Patch Set 14: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32ad627c953ba24d9cde2d5549bdd0d27a9c0d06 Gerrit-Change-Number: 18413 Gerrit-PatchSet: 14 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 13 May 2022 15:47:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11283: Push-down IS NULL and NOT NULL predicates to iceberg
lipeng...@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/18504 ) Change subject: IMPALA-11283: Push-down IS_NULL and NOT_NULL predicates to iceberg .. Patch Set 6: (2 comments) Thanks for your cr. http://gerrit.cloudera.org:8080/#/c/18504/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18504/4//COMMIT_MSG@7 PS4, Line 7: IMPALA-11283 > This should be IMPALA-11283 I think Done http://gerrit.cloudera.org:8080/#/c/18504/5/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/18504/5/tests/query_test/test_iceberg.py@482 PS5, Line 482: assert datafiles[0]['value_counts'] == \ > belongs to the other CR. Done -- To view, visit http://gerrit.cloudera.org:8080/18504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c3608af67b552bebc55dcc5526f61f5439967bf Gerrit-Change-Number: 18504 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 13 May 2022 15:03:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11205: Implement Statistical functions : CORR(), COVAR SAMP() and COVAR POP()
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18413 ) Change subject: IMPALA-11205: Implement Statistical functions : CORR(), COVAR_SAMP() and COVAR_POP() .. Patch Set 14: Code-Review+1 LGTM -- To view, visit http://gerrit.cloudera.org:8080/18413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32ad627c953ba24d9cde2d5549bdd0d27a9c0d06 Gerrit-Change-Number: 18413 Gerrit-PatchSet: 14 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 13 May 2022 12:45:23 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-9057: Disable test transactional insert events"
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18524 ) Change subject: Revert "IMPALA-9057: Disable test_transactional_insert_events" .. Patch Set 1: Verified in two jobs: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/16440 https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/5662/ -- To view, visit http://gerrit.cloudera.org:8080/18524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I369353c8bb9b5437c2f5c4d411fb05e2fd751b9b Gerrit-Change-Number: 18524 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Fri, 13 May 2022 12:43:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11205: Implement Statistical functions : CORR(), COVAR SAMP() and COVAR POP()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18413 ) Change subject: IMPALA-11205: Implement Statistical functions : CORR(), COVAR_SAMP() and COVAR_POP() .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10572/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32ad627c953ba24d9cde2d5549bdd0d27a9c0d06 Gerrit-Change-Number: 18413 Gerrit-PatchSet: 14 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 13 May 2022 11:43:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11205: Implement Statistical functions : CORR(), COVAR SAMP() and COVAR POP()
pranav.lo...@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/18413 ) Change subject: IMPALA-11205: Implement Statistical functions : CORR(), COVAR_SAMP() and COVAR_POP() .. Patch Set 14: (3 comments) > Patch Set 13: > > (3 comments) http://gerrit.cloudera.org:8080/#/c/18413/12/testdata/workloads/functional-query/queries/QueryTest/aggregation.test File testdata/workloads/functional-query/queries/QueryTest/aggregation.test: http://gerrit.cloudera.org:8080/#/c/18413/12/testdata/workloads/functional-query/queries/QueryTest/aggregation.test@1526 PS12, Line 1526: select s_store_sk, corr(s_number_employees, s_floor_space) over (partition by s_city) from tpcds.store; > sorry that I mean changing the query to No problem at all, I've updated the query. http://gerrit.cloudera.org:8080/#/c/18413/12/testdata/workloads/functional-query/queries/QueryTest/aggregation.test@1683 PS12, Line 1683: > sorry, I mean changing the query to No problem at all, I've updated the query. http://gerrit.cloudera.org:8080/#/c/18413/12/testdata/workloads/functional-query/queries/QueryTest/aggregation.test@1700 PS12, Line 1700: TYPES > sorry, I mean changing it to No problem at all, I've updated the query. -- To view, visit http://gerrit.cloudera.org:8080/18413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32ad627c953ba24d9cde2d5549bdd0d27a9c0d06 Gerrit-Change-Number: 18413 Gerrit-PatchSet: 14 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 13 May 2022 11:31:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11205: Implement Statistical functions : CORR(), COVAR SAMP() and COVAR POP()
pranav.lo...@cloudera.com has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/18413 ) Change subject: IMPALA-11205: Implement Statistical functions : CORR(), COVAR_SAMP() and COVAR_POP() .. IMPALA-11205: Implement Statistical functions : CORR(), COVAR_SAMP() and COVAR_POP() CORR() function takes two numeric type columns as arguments and returns the Pearson's correlation coefficient between them. COVAR_SAMP() function takes two numeric type columns and returns sample covariance between them. COVAR_POP() function takes two numeric type columns and returns population covariance between them. These UDAFs are tested with a few query tests written in aggregation.test. Change-Id: I32ad627c953ba24d9cde2d5549bdd0d27a9c0d06 --- M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/aggregate-functions.h M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M testdata/workloads/functional-query/queries/QueryTest/aggregation.test 4 files changed, 759 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/18413/14 -- To view, visit http://gerrit.cloudera.org:8080/18413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32ad627c953ba24d9cde2d5549bdd0d27a9c0d06 Gerrit-Change-Number: 18413 Gerrit-PatchSet: 14 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11205: Implement Statistical functions : CORR(), COVAR SAMP() and COVAR POP()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18413 ) Change subject: IMPALA-11205: Implement Statistical functions : CORR(), COVAR_SAMP() and COVAR_POP() .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8101/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32ad627c953ba24d9cde2d5549bdd0d27a9c0d06 Gerrit-Change-Number: 18413 Gerrit-PatchSet: 14 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 13 May 2022 11:24:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11205: Implement Statistical functions : CORR(), COVAR SAMP() and COVAR POP()
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18413 ) Change subject: IMPALA-11205: Implement Statistical functions : CORR(), COVAR_SAMP() and COVAR_POP() .. Patch Set 13: (3 comments) http://gerrit.cloudera.org:8080/#/c/18413/12/testdata/workloads/functional-query/queries/QueryTest/aggregation.test File testdata/workloads/functional-query/queries/QueryTest/aggregation.test: http://gerrit.cloudera.org:8080/#/c/18413/12/testdata/workloads/functional-query/queries/QueryTest/aggregation.test@1526 PS12, Line 1526: select corr(s_store_sk, s_floor_space) over (partition by s_city) from tpcds.store; > Done sorry that I mean changing the query to select s_store_sk, corr(s_number_employees,s_floor_space) over (partition by s_city) from tpcds.store Some result rows are identical. Adding the s_store_sk column helps to distinguish them. http://gerrit.cloudera.org:8080/#/c/18413/12/testdata/workloads/functional-query/queries/QueryTest/aggregation.test@1683 PS12, Line 1683: > Done sorry, I mean changing the query to select s_store_sk, covar_samp(s_number_employees, s_floor_space) over (partition by s_city) from tpcds.store http://gerrit.cloudera.org:8080/#/c/18413/12/testdata/workloads/functional-query/queries/QueryTest/aggregation.test@1700 PS12, Line 1700: TYPES > Done sorry, I mean changing it to select s_store_sk, covar_pop(s_number_employees, s_floor_space) over (partition by s_city) from tpcds.store -- To view, visit http://gerrit.cloudera.org:8080/18413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32ad627c953ba24d9cde2d5549bdd0d27a9c0d06 Gerrit-Change-Number: 18413 Gerrit-PatchSet: 13 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 13 May 2022 10:46:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11283: Push-down IS NULL and NOT NULL predicates to iceberg
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18504 ) Change subject: IMPALA-11283: Push-down IS_NULL and NOT_NULL predicates to iceberg .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10571/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c3608af67b552bebc55dcc5526f61f5439967bf Gerrit-Change-Number: 18504 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 13 May 2022 09:58:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11283: Push-down IS NULL and NOT NULL predicates to iceberg
lipeng...@sensorsdata.cn has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18504 ) Change subject: IMPALA-11283: Push-down IS_NULL and NOT_NULL predicates to iceberg .. IMPALA-11283: Push-down IS_NULL and NOT_NULL predicates to iceberg This patch implements pushing the IS_NULL and NOT_NULL predicates down to Iceberg. Testing: - Added end-to-end test Change-Id: I9c3608af67b552bebc55dcc5526f61f5439967bf --- M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-is-null-predicate-push-down.test M tests/query_test/test_iceberg.py 3 files changed, 427 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/18504/6 -- To view, visit http://gerrit.cloudera.org:8080/18504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9c3608af67b552bebc55dcc5526f61f5439967bf Gerrit-Change-Number: 18504 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7942: Add query hints for cardinalities and selectivities
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18023 ) Change subject: IMPALA-7942: Add query hints for cardinalities and selectivities .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10570/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18023 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b Gerrit-Change-Number: 18023 Gerrit-PatchSet: 9 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 13 May 2022 09:07:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7942: Add query hints for cardinalities and selectivities
wangsheng has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/18023 ) Change subject: IMPALA-7942: Add query hints for cardinalities and selectivities .. IMPALA-7942: Add query hints for cardinalities and selectivities Currently, Impala only uses simple estimation to compute selectivity for some predicates, and this may lead to worse query plan due to CBO. Hence, we add new hints to reduce such errors. Maybe in the future, we can use histograms to get more precise query plan. This patch adds two query hints: 'HDFS_NUM_ROWS' and 'SELECTIVITY'. We can add 'HDFS_NUM_ROWS' after a hdfs table in query like this: * select col from t /* +TABLE_NUM_ROWS(1000) */; If set, Impala will use this value as table scanned rows, even if table has stats. For 'SELECTIVITY' hint, we can use in these 'Predicate': * BinaryPredicate * InPredicate * IsNullPredicate * LikePredicate, including 'not like' syntax * BetweenPredicate, including 'not between and' syntax Format like this: select col from t where a=1 /* +SELECTIVITY(0.5) */; This value will replace original selectivity computing. These formats are not allowed: * select col from t where (a=1) /* +SELECTIVITY(0.5) */; * select col from t where (a=1 and b<2) /* +SELECTIVITY(0.5) */; * select col from t1 where exists (...) /* +SELECTIVITY(0.5) */; Pay attention, if you set selectivity hint like this: * select col from t where (a=1 /* +SELECTIVITY(0.5) */ and b>2); Impala will set 0.5 for first binary predicate, second is -1, so Impala can not compute this predicate.The whole compound predicate selectivity is still unavailable. Hence, for compound predicate, we need to ensure that each child selectivity has been set by hint or computable. Otherwise, this hint might not take the expected effect. Another thing, for 'BetweenPredicate', Impala will transfom this predicate to a 'CompoundPredicate' with two 'BinaryPredicate', if set hint for 'BetweenPredicate' in query, we will split this hint value for two 'BinaryPredicate' children. Testing: - Added new fe tests in 'PlannerTest' - Added new fe tests in 'AnalyzeStmtsTest' for negative cases Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/InPredicate.java M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/Predicate.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/hdfs-cardinality-hint.test A testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test 16 files changed, 1,588 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/9 -- To view, visit http://gerrit.cloudera.org:8080/18023 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b Gerrit-Change-Number: 18023 Gerrit-PatchSet: 9 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-7942: Add query hints for cardinalities and selectivities
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/18023 ) Change subject: IMPALA-7942: Add query hints for cardinalities and selectivities .. Patch Set 9: Hi Quanlong, Qifan and Zoltan, I've already modify patch to support selectivity hint for CompoundPredicate 1. We need add bracket for each predicate to use selectivity hint, such as: (a = 1) /* +SELECTIVITY(0.2)*/, (a = 1 AND b = 2) /* +SELECTIVITY(0.2)*/, bracket is necessary due to CompoundPredicate supporting; 2. This syntax is allowed: ((a = 1) /* +SELECTIVITY(0.2)*/) /* +SELECTIVITY(0.3)*/, and the outer selectivity will override inner selectivity; 3. FE planner will keep 'AND' CompoundPredicate as a conjunct when setting selectivity hint, instead of split to two BinaryPredicate. I will modify test cases and commit msg after we discuss this adjust. Hope for your advice. -- To view, visit http://gerrit.cloudera.org:8080/18023 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b Gerrit-Change-Number: 18023 Gerrit-PatchSet: 9 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 13 May 2022 08:48:27 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-9057: Disable test transactional insert events"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18524 ) Change subject: Revert "IMPALA-9057: Disable test_transactional_insert_events" .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10569/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I369353c8bb9b5437c2f5c4d411fb05e2fd751b9b Gerrit-Change-Number: 18524 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 13 May 2022 07:26:23 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-11264: (Addendum) Bump ORC to 1.7.0-p14 to fix ORC-1159
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18525 Change subject: IMPALA-11264: (Addendum) Bump ORC to 1.7.0-p14 to fix ORC-1159 .. IMPALA-11264: (Addendum) Bump ORC to 1.7.0-p14 to fix ORC-1159 This patch adds ORC-1160 and ORC-1159 which fixes the bug introduced by ORC-969. Change-Id: I726e84b85f3713fc54165007d63a21dab2af84f2 --- M buildall.sh A source/orc/orc-1.7.0-patches/0013-ORC-1160-C-Fix-seekToRow-can-t-seek-within-selected-.patch A source/orc/orc-1.7.0-patches/0014-ORC-1159-C-Fix-crash-when-the-last-stripe-is-skipped.patch 3 files changed, 518 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/25/18525/1 -- To view, visit http://gerrit.cloudera.org:8080/18525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I726e84b85f3713fc54165007d63a21dab2af84f2 Gerrit-Change-Number: 18525 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] Revert "IMPALA-9057: Disable test transactional insert events"
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18524 Change subject: Revert "IMPALA-9057: Disable test_transactional_insert_events" .. Revert "IMPALA-9057: Disable test_transactional_insert_events" This reverts commit a873289508dc4e80256718c9921be1e337d263d6. After HIVE-25137 and IMPALA-10923, we are able to add back the test. Change-Id: I369353c8bb9b5437c2f5c4d411fb05e2fd751b9b --- M tests/metadata/test_event_processing.py 1 file changed, 0 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/18524/1 -- To view, visit http://gerrit.cloudera.org:8080/18524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I369353c8bb9b5437c2f5c4d411fb05e2fd751b9b Gerrit-Change-Number: 18524 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
