[Impala-ASF-CR] IMPALA-2019(part-4): Add UTF-8 support for case conversion functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17785 ) Change subject: IMPALA-2019(part-4): Add UTF-8 support for case conversion functions .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7822/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17785 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I443e89d46f4638ce85664b021666bc4f03ee8abd Gerrit-Change-Number: 17785 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 09 Feb 2022 07:22:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18207 ) Change subject: IMPALA-11109: Catch class loading error for UDFs. .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 7 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 09 Feb 2022 06:40:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18207 ) Change subject: IMPALA-11109: Catch class loading error for UDFs. .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 09 Feb 2022 04:08:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18207 ) Change subject: IMPALA-11109: Catch class loading error for UDFs. .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10122/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 8 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 09 Feb 2022 03:44:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Hello Daniel Becker, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18207 to look at the new patch set (#8). Change subject: IMPALA-11109: Catch class loading error for UDFs. .. IMPALA-11109: Catch class loading error for UDFs. Fixes regression caused by IMPALA-10997. If a UDF fails to load, an exception needs to be caught and logged or else catalogd will not start up. The method HiveLegacyJavaFunction.extract() will catch any hidden exception thrown, and rethrow it as a CatalogException. The caller already handles the CatalogException and logs the message appropriately. The specific "hidden" exception thrown in our regression tests was a "ClassNotFoundException", but in case there are other exceptions that are thrown, we catch the generic "Throwable" exception because the "extract" method should not prevent the server from coming up. Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf --- M fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java M fe/src/test/java/org/apache/impala/hive/executor/HiveLegacyJavaFunctionTest.java 2 files changed, 23 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/18207/8 -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 8 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18207 ) Change subject: IMPALA-11109: Catch class loading error for UDFs. .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7821/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 8 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 09 Feb 2022 03:23:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11097: Call wait to finish before fetch results for HS2.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18133 ) Change subject: IMPALA-11097: Call wait_to_finish before fetch_results for HS2. .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10121/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18133 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ab4b90ba2e1a439119d37fe9fb9c55eeeb53ba0 Gerrit-Change-Number: 18133 Gerrit-PatchSet: 13 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 09 Feb 2022 00:31:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18207 ) Change subject: IMPALA-11109: Catch class loading error for UDFs. .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10120/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 7 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 09 Feb 2022 00:22:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11097: Call wait to finish before fetch results for HS2.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/18133 ) Change subject: IMPALA-11097: Call wait_to_finish before fetch_results for HS2. .. Patch Set 12: (2 comments) http://gerrit.cloudera.org:8080/#/c/18133/12//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18133/12//COMMIT_MSG@8 PS12, Line 8: > Can you add some more context about the issue? Done http://gerrit.cloudera.org:8080/#/c/18133/12/tests/common/impala_connection.py File tests/common/impala_connection.py: http://gerrit.cloudera.org:8080/#/c/18133/12/tests/common/impala_connection.py@342 PS12, Line 342: # For Hive statements that have no result set (eg USE), they may still be : # running, and we need to wait for them to finish before we can proceed. > Is the comment still valid? now we are always calling wait_to_finish(), eve Done -- To view, visit http://gerrit.cloudera.org:8080/18133 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ab4b90ba2e1a439119d37fe9fb9c55eeeb53ba0 Gerrit-Change-Number: 18133 Gerrit-PatchSet: 12 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 09 Feb 2022 00:07:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11097: Call wait to finish before fetch results for HS2.
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18133 to look at the new patch set (#13). Change subject: IMPALA-11097: Call wait_to_finish before fetch_results for HS2. .. IMPALA-11097: Call wait_to_finish before fetch_results for HS2. When executing a query in Hive, the query goes into RUNNING state and then FINISHED state. The FINISHED state is the signal for the client that it is ok to start fetching results. In the test framework, we issue the queries async to Hive, so it is necessary to wait_to_finish to be called. The tests were working prior to the fix, but only because the query hit the FINISH state quickly enough so that it was ok to retrieve the result set. Change-Id: I5ab4b90ba2e1a439119d37fe9fb9c55eeeb53ba0 --- M tests/common/impala_connection.py 1 file changed, 9 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/18133/13 -- To view, visit http://gerrit.cloudera.org:8080/18133 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5ab4b90ba2e1a439119d37fe9fb9c55eeeb53ba0 Gerrit-Change-Number: 18133 Gerrit-PatchSet: 13 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Steve Carlin has posted comments on this change. (
http://gerrit.cloudera.org:8080/18207 )
Change subject: IMPALA-11109: Catch class loading error for UDFs.
..
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18207/5/fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java
File
fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java:
http://gerrit.cloudera.org:8080/#/c/18207/5/fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java@167
PS5, Line 167: if (result.isEmpty()) {
: throw new CatalogException("No compatible function
signatures found.");
: }
> This looks a bit weird - the outer catch will catch the inner Catalog excep
Done
http://gerrit.cloudera.org:8080/#/c/18207/5/fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java@170
PS5, Line 170: Throwable
> Can you add a comment about why we check Throwable?
Done
--
To view, visit http://gerrit.cloudera.org:8080/18207
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf
Gerrit-Change-Number: 18207
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Steve Carlin
Gerrit-Comment-Date: Wed, 09 Feb 2022 00:01:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Hello Daniel Becker, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18207 to look at the new patch set (#7). Change subject: IMPALA-11109: Catch class loading error for UDFs. .. IMPALA-11109: Catch class loading error for UDFs. Fixes regression caused by IMPALA-10997. If a UDF fails to load, an exception needs to be caught and logged or else catalogd will not start up. The method HiveLegacyJavaFunction.extract() will catch any hidden exception thrown, and rethrow it as a CatalogException. The caller already handles the CatalogException and logs the message appropriately. The specific "hidden" exception thrown in our regression tests was a "ClassNotFoundException", but in case there are other exceptions that are thrown, we catch the generic "Throwable" exception because the "extract" method should not prevent the server from coming up. Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf --- M fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java M fe/src/test/java/org/apache/impala/hive/executor/HiveLegacyJavaFunctionTest.java 2 files changed, 23 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/18207/7 -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 7 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18207 ) Change subject: IMPALA-11109: Catch class loading error for UDFs. .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7820/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 7 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 09 Feb 2022 00:01:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18207 ) Change subject: IMPALA-11109: Catch class loading error for UDFs. .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/18207/7/fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java File fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java: http://gerrit.cloudera.org:8080/#/c/18207/7/fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java@169 PS7, Line 169: // is ClassNotFoundException thrown by UDF_.getClass(). We want to catch all possible line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 7 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 09 Feb 2022 00:01:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11097: Call wait to finish before fetch results for HS2.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18133 ) Change subject: IMPALA-11097: Call wait_to_finish before fetch_results for HS2. .. Patch Set 12: (2 comments) http://gerrit.cloudera.org:8080/#/c/18133/12//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18133/12//COMMIT_MSG@8 PS12, Line 8: Can you add some more context about the issue? http://gerrit.cloudera.org:8080/#/c/18133/12/tests/common/impala_connection.py File tests/common/impala_connection.py: http://gerrit.cloudera.org:8080/#/c/18133/12/tests/common/impala_connection.py@342 PS12, Line 342: # For Hive statements that have no result set (eg USE), they may still be : # running, and we need to wait for them to finish before we can proceed. Is the comment still valid? now we are always calling wait_to_finish(), even if the query has a result set. -- To view, visit http://gerrit.cloudera.org:8080/18133 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ab4b90ba2e1a439119d37fe9fb9c55eeeb53ba0 Gerrit-Change-Number: 18133 Gerrit-PatchSet: 12 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 08 Feb 2022 23:33:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Joe McDonnell has removed a vote on this change. Change subject: IMPALA-11109: Catch class loading error for UDFs. .. Removed Code-Review+2 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/18207 )
Change subject: IMPALA-11109: Catch class loading error for UDFs.
..
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18207/5/fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java
File
fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java:
http://gerrit.cloudera.org:8080/#/c/18207/5/fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java@167
PS5, Line 167: if (result.isEmpty()) {
: throw new CatalogException("No compatible function
signatures found.");
: }
This looks a bit weird - the outer catch will catch the inner Catalog
exception, and rethrow it with a bit different text.
This check could could be moved just before "return result;"
http://gerrit.cloudera.org:8080/#/c/18207/5/fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java@170
PS5, Line 170: Throwable
Can you add a comment about why we check Throwable?
--
To view, visit http://gerrit.cloudera.org:8080/18207
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf
Gerrit-Change-Number: 18207
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Steve Carlin
Gerrit-Comment-Date: Tue, 08 Feb 2022 22:38:04 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18207 ) Change subject: IMPALA-11109: Catch class loading error for UDFs. .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7819/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Tue, 08 Feb 2022 21:32:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18207 ) Change subject: IMPALA-11109: Catch class loading error for UDFs. .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Tue, 08 Feb 2022 21:32:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11109: Catch class loading error for UDFs.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18207 ) Change subject: IMPALA-11109: Catch class loading error for UDFs. .. Patch Set 4: Code-Review+2 I ran custom cluster tests in exhaustive mode with this change and they all passed. This fix makes sense to me, I think we should go ahead. -- To view, visit http://gerrit.cloudera.org:8080/18207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16813209cd4c2367c45e569c2aac13eff7ce2dbf Gerrit-Change-Number: 18207 Gerrit-PatchSet: 4 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Tue, 08 Feb 2022 21:31:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10961: Implementing adaptive 3-way quicksort in sorter
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18184 ) Change subject: IMPALA-10961: Implementing adaptive 3-way quicksort in sorter .. Patch Set 7: (6 comments) I'm really excited about this change as it will greatly improve partitioned INSERTs. The change looks really good, I've only found a few naming issues. http://gerrit.cloudera.org:8080/#/c/18184/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18184/7//COMMIT_MSG@9 PS7, Line 9: 3way nit: 3-way http://gerrit.cloudera.org:8080/#/c/18184/7//COMMIT_MSG@23 PS7, Line 23: select * order by l_linestatus, NDV=2: : original 2-way vs adaptive quicksort - 1 : 0.67 : select l_shipmode order by l_shipmode, NDV=7: : original 2-way vs adaptive quicksort - 1 : 0.42 : select * order by l_shipmode, NDV=7: : original 2-way vs adaptive quicksort - 1 : 0.57 : large NDV, unique data: no significant difference Using https://ozh.github.io/ascii-tables/ (for these benchmark tables it doesn't matter if the line widths are longer) +--+++ | Test | Original 2-way | Adaptive Quicksort | +--+++ | select * order by l_linestatus, NDV=2: | 1 | 0.67 | | select l_shipmode order by l_shipmode, NDV=7 | 1 | 0.42 | | select * order by l_shipmode, NDV=7 | 1 | 0.57 | | large NDV, unique data | 1 | 1 | (no difference) +--+++ http://gerrit.cloudera.org:8080/#/c/18184/7/be/src/runtime/sorter-internal.h File be/src/runtime/sorter-internal.h: http://gerrit.cloudera.org:8080/#/c/18184/7/be/src/runtime/sorter-internal.h@533 PS7, Line 533: hasEquals nit: we name variables with underscores instead of camel case in the backend. I.e. it should be 'has_equals' http://gerrit.cloudera.org:8080/#/c/18184/7/be/src/runtime/sorter-ir.cc File be/src/runtime/sorter-ir.cc: http://gerrit.cloudera.org:8080/#/c/18184/7/be/src/runtime/sorter-ir.cc@157 PS7, Line 157: reinterpret_cast(&temp_tuple) nit: Since it's being used at multiple places, maybe create a variable 'temp_tuple_row' for it. And how about calling 'temp_tuple' as 'pivot_tuple' and 'temp_tuple_row' as 'pivot_tuple_row'? http://gerrit.cloudera.org:8080/#/c/18184/7/be/src/runtime/sorter-ir.cc@260 PS7, Line 260: hasEquals nit: should be 'has_equals' http://gerrit.cloudera.org:8080/#/c/18184/7/be/src/runtime/sorter-ir.cc@320 PS7, Line 320: lt Maybe rename it to 't1_cmp_t2'? -- To view, visit http://gerrit.cloudera.org:8080/18184 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81e7b36a04a43de3b83e6aeee49ca0943f0bf202 Gerrit-Change-Number: 18184 Gerrit-PatchSet: 7 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 08 Feb 2022 19:34:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17774 ) Change subject: IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2 .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17774 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f08db5f6da735ac431819063060941f0941f606 Gerrit-Change-Number: 17774 Gerrit-PatchSet: 12 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 08 Feb 2022 14:27:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6636: Use async IO in ORC scanner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15370 ) Change subject: IMPALA-6636: Use async IO in ORC scanner .. Patch Set 30: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7817/ -- To view, visit http://gerrit.cloudera.org:8080/15370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I348ad9e55f0cae7dff0d74d941b026dcbf5e4074 Gerrit-Change-Number: 15370 Gerrit-PatchSet: 30 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 08 Feb 2022 12:17:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10961: Implementing adaptive 3-way quicksort in sorter
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18184 ) Change subject: IMPALA-10961: Implementing adaptive 3-way quicksort in sorter .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10119/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18184 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81e7b36a04a43de3b83e6aeee49ca0943f0bf202 Gerrit-Change-Number: 18184 Gerrit-PatchSet: 7 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 08 Feb 2022 11:27:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10961: Implementing adaptive 3-way quicksort in sorter
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18184 ) Change subject: IMPALA-10961: Implementing adaptive 3-way quicksort in sorter .. Patch Set 7: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18184 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81e7b36a04a43de3b83e6aeee49ca0943f0bf202 Gerrit-Change-Number: 18184 Gerrit-PatchSet: 7 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 08 Feb 2022 11:11:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10961: Implementing adaptive 3-way quicksort in sorter
Noemi Pap-Takacs has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18184 Change subject: IMPALA-10961: Implementing adaptive 3-way quicksort in sorter .. IMPALA-10961: Implementing adaptive 3-way quicksort in sorter Based on a 3way partitioning implementation by Kurt Deschler. 3-way quicksort performs much better on data with large number of duplicates, but has a small regression in case of large NDV. This adaptive implementation keeps the advantages of both 2-way and 3-way quicksort. If duplicates are found during pivot selection (among the 3 randomly selected candidates),the 3-way partitioning function is called in SortHelper, otherwise partitioning goes 2-way. Some benchmark results: On a view created from 4 tpch_parquet lineitem tables Full sort, 1 node, 1 run - no spills (only in-memory sort is changed) Time of sorting adaptively during query execution compared to the original implementation (sort node profile): select * order by l_linestatus, NDV=2: original 2-way vs adaptive quicksort - 1 : 0.67 select l_shipmode order by l_shipmode, NDV=7: original 2-way vs adaptive quicksort - 1 : 0.42 select * order by l_shipmode, NDV=7: original 2-way vs adaptive quicksort - 1 : 0.57 large NDV, unique data: no significant difference Change-Id: I81e7b36a04a43de3b83e6aeee49ca0943f0bf202 --- M be/src/runtime/sorter-internal.h M be/src/runtime/sorter-ir.cc M be/src/runtime/sorter.cc 3 files changed, 174 insertions(+), 43 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/18184/7 -- To view, visit http://gerrit.cloudera.org:8080/18184 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I81e7b36a04a43de3b83e6aeee49ca0943f0bf202 Gerrit-Change-Number: 18184 Gerrit-PatchSet: 7 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Noemi Pap-Takacs
[Impala-ASF-CR] IMPALA-10838: Error when struct returned from WITH()
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/17847 )
Change subject: IMPALA-10838: Error when struct returned from WITH()
..
Patch Set 15:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/17847/15/fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java
File fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java:
http://gerrit.cloudera.org:8080/#/c/17847/15/fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java@294
PS15, Line 294: don;t
typo
http://gerrit.cloudera.org:8080/#/c/17847/15/fe/src/main/java/org/apache/impala/analysis/Path.java
File fe/src/main/java/org/apache/impala/analysis/Path.java:
http://gerrit.cloudera.org:8080/#/c/17847/15/fe/src/main/java/org/apache/impala/analysis/Path.java@475
PS15, Line 475: public List getRawPathWithoutPrefix(Path prefix) {
Are we still using this somewhere?
http://gerrit.cloudera.org:8080/#/c/17847/15/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
File fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java:
http://gerrit.cloudera.org:8080/#/c/17847/15/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java@194
PS15, Line 194: parentStructSlotDesc.getParent() : null;
nit: +2 indentation
http://gerrit.cloudera.org:8080/#/c/17847/15/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java@218
PS15, Line 218:* TODO: {type}.getSlotSize() is used in many other places,
for example in
:* HdfsScanNode.java:getStatsNumRows and ColumnStats.java. Is
it a problem? In those
:* places SlotDescriptors are not available, either only
types or 'Column'.
Good question.
I don't know how stats handling deals with structs - do we completely ignore
it, or calculate stats for members individually? I think that the size of a
type should be irrelevant in both cases. If we would calculate stats for the
structs themselves, then I think that only the number of nulls would be
interesting.
In HdfsScanNode.java:getStatsNumRow we only check the size for scalar types:
https://github.com/apache/impala/blob/d59ec73990d89bfa4d4fa3d8fe598d53eb2918b7/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java#L1522
This code only runs in case there is no row count stat for a table or
partition, so we have to estimate the number of rows based on the size of the
files + ALL columns within. So if we would want to include structs in this
estimate, we should simply include all members as if it was materialized. This
is a very rough estimate anyway, so there is no clear good answer.
I think that stat handling of structs could totally deserve a separate jira :)
http://gerrit.cloudera.org:8080/#/c/17847/15/testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test
File
testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test:
http://gerrit.cloudera.org:8080/#/c/17847/15/testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test@121
PS15, Line 121: QUERY
Can you add tests where different kind of view-s are used, e.g. a WITH clause
which is used within an inline view?
I have added tests like this in
https://gerrit.cloudera.org/#/c/17811/31/testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
e.g.
# Unnesting array returned by view wrapped in inline view + WITH clause.
with v2 as (select id, int_array from complextypes_arrays_only_view)
select v.id, a.item from (select id, int_array from v2) v, v.int_array a;
--
To view, visit http://gerrit.cloudera.org:8080/17847
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57
Gerrit-Change-Number: 17847
Gerrit-PatchSet: 15
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Comment-Date: Tue, 08 Feb 2022 10:17:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10989: fix race for result set metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18212 ) Change subject: IMPALA-10989: fix race for result set metadata .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18212 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0833ed20d62474c434fa94bbbf8cd8ea99a7cf4 Gerrit-Change-Number: 18212 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 08 Feb 2022 09:11:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17774 ) Change subject: IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2 .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10118/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17774 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f08db5f6da735ac431819063060941f0941f606 Gerrit-Change-Number: 17774 Gerrit-PatchSet: 12 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 08 Feb 2022 08:36:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2019(part-4): Add UTF-8 support for case conversion functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17785 ) Change subject: IMPALA-2019(part-4): Add UTF-8 support for case conversion functions .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10117/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17785 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I443e89d46f4638ce85664b021666bc4f03ee8abd Gerrit-Change-Number: 17785 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 08 Feb 2022 08:29:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2
Fucun Chu has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/17774 ) Change subject: IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2 .. IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2 Like IMPALA-8369, this patch adds a compatibility shim in fe so that Impala can interoperate with Hive 3.1.2. we need adds a new Metastoreshim class under compat-apache-hive-3 directory. These shim classes implement method which are different in hive-3 vs apache-hive-3 and are used by front end code. At the build time, based on the environment variable IMPALA_HIVE_DIST_TYPE one of the two shims is added to as source using the fe/pom.xml build plugin. Some codes that directly use Hive 4 APIs need to be ignored in compilation, eg. fe/src/main/java/org/apache/impala/catalog/metastore/. Use Maven profile to ignore some codes, profile will automatically activated based on the IMPALA_HIVE_DIST_TYPE. Testing: 1. Code compiles and runs against both HMS-3 and ASF-HMS-3 2. Ran full-suite of tests against HMS-3 3. Running full-tests against ASF-HMS-3 will need more work supporting Tez in the mini-cluster (for dataloading) and HMS transaction support. This will be on-going effort and test failures on ASF-Hive-3 will be fixed in additional sub-tasks. Notes: 1. Patch uses a custom build of Apache Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21569, HIVE-20038. This hack will be added to the build script in additional sub-tasks. Change-Id: I9f08db5f6da735ac431819063060941f0941f606 --- M bin/rat_exclude_files.txt M buildall.sh M fe/pom.xml A fe/src/compat-apache-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java A fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java A fe/src/main/java/org/apache/impala/catalog/metastore/CatalogHmsUtils.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServer.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M java/pom.xml A testdata/bin/patch_hive.sh A testdata/cluster/hive/README A testdata/cluster/hive/patch0-HIVE-21586.diff 25 files changed, 2,998 insertions(+), 261 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/17774/12 -- To view, visit http://gerrit.cloudera.org:8080/17774 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9f08db5f6da735ac431819063060941f0941f606 Gerrit-Change-Number: 17774 Gerrit-PatchSet: 12 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-2019(part-4): Add UTF-8 support for case conversion functions
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17785 ) Change subject: IMPALA-2019(part-4): Add UTF-8 support for case conversion functions .. Patch Set 6: (6 comments) Thanks for the feedback, Qifan! http://gerrit.cloudera.org:8080/#/c/17785/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17785/6//COMMIT_MSG@9 PS6, Line 9: string functions doing case conversion > nit. case conversion string functions Done http://gerrit.cloudera.org:8080/#/c/17785/6//COMMIT_MSG@11 PS6, Line 11: unicode > nit Unicode Done http://gerrit.cloudera.org:8080/#/c/17785/6/be/src/exprs/string-functions-ir.cc File be/src/exprs/string-functions-ir.cc: http://gerrit.cloudera.org:8080/#/c/17785/6/be/src/exprs/string-functions-ir.cc@366 PS6, Line 366: if (wc_bytes == 0) break; > Per https://en.cppreference.com/w/cpp/string/multibyte/mbtowc, on return v Thanks for catching this! http://gerrit.cloudera.org:8080/#/c/17785/6/be/src/exprs/string-functions-ir.cc@370 PS6, Line 370: ; > nit. 0xFFFD. Done http://gerrit.cloudera.org:8080/#/c/17785/6/be/src/exprs/string-functions-ir.cc@371 PS6, Line 371: wc_bytes = 3; > nit. should it be 4? Thanks for catching this! I think we should jump to the next legal UTF-8 start byte. http://gerrit.cloudera.org:8080/#/c/17785/6/be/src/exprs/string-functions-ir.cc@408 PS6, Line 408: iswspace > nit. UNLIKELY() Done -- To view, visit http://gerrit.cloudera.org:8080/17785 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I443e89d46f4638ce85664b021666bc4f03ee8abd Gerrit-Change-Number: 17785 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 08 Feb 2022 08:06:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-2019(part-4): Add UTF-8 support for case conversion functions
Hello Qifan Chen, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17785 to look at the new patch set (#7). Change subject: IMPALA-2019(part-4): Add UTF-8 support for case conversion functions .. IMPALA-2019(part-4): Add UTF-8 support for case conversion functions There are 3 builtin case conversion string functions: upper(), lower(), and initcap(). Previously they only convert English alphabetic characters. This patch adds support to deal with Unicode characters. There are many corner cases in case conversion depending on the locale and context. E.g. 1) Case conversion is locale-sensitive. Turkish has 4 letter "I"s. English has only two, a lowercase dotted i and an uppercase dotless I. Turkish has lowercase and uppercase forms of both dotted and dotless I. So simply converting "i" to "I" for upper case is wrong in Turkish: +---++-+ | | Dotted | Dotless | +---++-+ | Upper | İ | I | +---++-+ | Lower | i | ı | +---++-+ 2) Case conversion may change a string's length. The German word "grüßen" should be converted to "GRÜSSEN" in upper case: the letter "ß" should be converted to "SS". 3) Case conversion is context-sensitive. The Greek word "ὈΔΥΣΣΕΎΣ" should be converted to "ὀδυσσεύς", where the Greek letter "Σ" is converted to "σ" or to "ς", depending on its position in the word. This patch uses the simple wchar_t conversions of std::towupper and stdd::towlower so the above cases are not handled. We will try supporting them in follow-up JIRAs. Test: - Add BE unit tests and e2e tests. Change-Id: I443e89d46f4638ce85664b021666bc4f03ee8abd --- M be/src/exprs/expr-test.cc M be/src/exprs/string-functions-ir.cc M be/src/exprs/string-functions.h M common/function-registry/impala_functions.py M testdata/workloads/functional-query/queries/QueryTest/utf8-string-functions.test 5 files changed, 212 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/17785/7 -- To view, visit http://gerrit.cloudera.org:8080/17785 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I443e89d46f4638ce85664b021666bc4f03ee8abd Gerrit-Change-Number: 17785 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang
