[Impala-ASF-CR] IMPALA-12699: Set timeout for catalog RPCs
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21146 ) Change subject: IMPALA-12699: Set timeout for catalog RPCs .. Patch Set 1: > Patch Set 1: > > Build Failed > > https://jenkins.impala.io/job/gerrit-code-review-checks/15508/ : Initial code > review checks failed. See linked job for details on the failure. The failure is an existing issue. Filed IMPALA-12915 -- To view, visit http://gerrit.cloudera.org:8080/21146 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad39a79d0c89f2b04380f610a7e60558429e9c6e Gerrit-Change-Number: 21146 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 18 Mar 2024 08:29:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
pranav.lo...@cloudera.com has posted comments on this change. (
http://gerrit.cloudera.org:8080/21131 )
Change subject: IMPALA-11499: Refactor UrlEncode function to handle special
characters
..
Patch Set 3:
(1 comment)
> Patch Set 3:
>
> (1 comment)
http://gerrit.cloudera.org:8080/#/c/21131/3/be/src/util/coding-util.cc
File be/src/util/coding-util.cc:
http://gerrit.cloudera.org:8080/#/c/21131/3/be/src/util/coding-util.cc@83
PS3, Line 83: if (!hive_compat) {
> Are these additional encodings the same that were done before this change?
I think '$' and other special characters are not supported or is it another bug:
ERROR: TableLoadingException: Failed to load metadata for table:
default.my_part_tbl
CAUSED BY: NullPointerException: Invalid partition name: p=$@%23%^&
--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Zihao Ye
Gerrit-Comment-Date: Mon, 18 Mar 2024 09:24:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12903: Querying virtual column FILE POSITION for TEXT and JSON tables crashes Impala
Hello Daniel Becker, Riza Suminto, Gabor Kaszab, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21148 to look at the new patch set (#2). Change subject: IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala .. IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala Impala generates segmentation fault when it queries the virtual column FILE__POSITION for TEXT or JSON tables. When the scanners that do not support the FILE__POSITION virtual column detect its presence they try to report an error and close themselves. The segfault is in the scanners' Close() method when they try to dereference a NULL stream object. This patch simply adds NULL-checks in Close(). Alternatively we could detect the presence of FILE__POSITION during planning in the HdfsScanNode, but doing it in the scanners lets us handle more queries, e.g. queries that dynamically prune partitions and the surviving partitions all have file formats that support FILE__POSITION. Testing: * added negative tests to properly report the errors * added tests for mixed file format tables Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 --- M be/src/exec/json/hdfs-json-scanner.cc M be/src/exec/text/hdfs-text-scanner.cc M testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-generic.test A testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-negative.test M tests/query_test/test_scanners.py 5 files changed, 92 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/21148/2 -- To view, visit http://gerrit.cloudera.org:8080/21148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 Gerrit-Change-Number: 21148 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12903: Querying virtual column FILE POSITION for TEXT and JSON tables crashes Impala
Hello Daniel Becker, Riza Suminto, Gabor Kaszab, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21148 to look at the new patch set (#3). Change subject: IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala .. IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala Impala generates segmentation fault when it queries the virtual column FILE__POSITION for TEXT or JSON tables. When the scanners that do not support the FILE__POSITION virtual column detect its presence they try to report an error and close themselves. The segfault is in the scanners' Close() method when they try to dereference a NULL stream object. This patch simply adds NULL-checks in Close(). Alternatively we could detect the presence of FILE__POSITION during planning in the HdfsScanNode, but doing it in the scanners lets us handle more queries, e.g. queries that dynamically prune partitions and the surviving partitions all have file formats that support FILE__POSITION. Testing: * added negative tests to properly report the errors * added tests for mixed file format tables Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 --- M be/src/exec/json/hdfs-json-scanner.cc M be/src/exec/text/hdfs-text-scanner.cc M testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-generic.test A testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-negative.test M tests/query_test/test_scanners.py 5 files changed, 92 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/21148/3 -- To view, visit http://gerrit.cloudera.org:8080/21148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 Gerrit-Change-Number: 21148 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12903: Querying virtual column FILE POSITION for TEXT and JSON tables crashes Impala
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21148 ) Change subject: IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala .. Patch Set 3: (6 comments) Thanks for the comments! http://gerrit.cloudera.org:8080/#/c/21148/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21148/1//COMMIT_MSG@19 PS1, Line 19: let > Nit: lets. Done http://gerrit.cloudera.org:8080/#/c/21148/1/testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-generic.test File testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-generic.test: http://gerrit.cloudera.org:8080/#/c/21148/1/testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-generic.test@158 PS1, Line 158: QUERY > Are these the queries where some files in the table do not support FILE_POS Done http://gerrit.cloudera.org:8080/#/c/21148/1/testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-generic.test@159 PS1, Line 159: # Regression test for IMPALA-12903. The following query uses static pruning. The surviving > nit: could you add a comment that in this test we prune partitions that doe Done http://gerrit.cloudera.org:8080/#/c/21148/1/testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-negative.test File testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-negative.test: http://gerrit.cloudera.org:8080/#/c/21148/1/testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-negative.test@1 PS1, Line 1: > Is FILE_POSITION the only virtual column that could cause this bug before t Only FILE__POSITION cause this problem. INPUT__FILE__NAME is supported for all file formats. http://gerrit.cloudera.org:8080/#/c/21148/1/testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-negative.test@40 PS1, Line 40: Virtual column FILE__POSITION is not supported > Could you replace some of the FILE__POSITIONS to some other virtual columns INPUT__FILE__NAME is supported for all file formats. http://gerrit.cloudera.org:8080/#/c/21148/1/tests/query_test/test_scanners.py File tests/query_test/test_scanners.py: http://gerrit.cloudera.org:8080/#/c/21148/1/tests/query_test/test_scanners.py@183 PS1, Line 183: ))) > Or just fix the table_format dimension to text/none and remove this constra Thanks for the suggestions. I went with the uncompressed text dimension option. -- To view, visit http://gerrit.cloudera.org:8080/21148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 Gerrit-Change-Number: 21148 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 18 Mar 2024 10:18:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10380/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Mon, 18 Mar 2024 10:24:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12903: Querying virtual column FILE POSITION for TEXT and JSON tables crashes Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21148 ) Change subject: IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15550/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 Gerrit-Change-Number: 21148 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 18 Mar 2024 10:40:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12903: Querying virtual column FILE POSITION for TEXT and JSON tables crashes Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21148 ) Change subject: IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15551/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 Gerrit-Change-Number: 21148 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 18 Mar 2024 10:40:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12487: Skip reloading file metadata for ALTER TABLE events with trivial changes in StorageDescriptor
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/21019 )
Change subject: IMPALA-12487: Skip reloading file metadata for ALTER_TABLE
events with trivial changes in StorageDescriptor
..
Patch Set 7: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/21019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1922
PS7, Line 1922: infoLog("Change in table schema detected for table
{}.{} from {} to {} ",
: tblName_, dbName_, beforeCols.get(i).getName() +
" (" +
: beforeCols.get(i).getType() +")",
afterCols.get(i).getName() + " (" +
: afterCols.get(i).getType() + "). So file
metadata reload can be " +
: "skipped.");
nit: might be a bit clearer to refactor this
infoLog("Change in table schema detected for table {}.{} from {} ({})
" +
"to {} ({}). So file metadata reload can be skipped.",
tblName_, dbName_, beforeCols.get(i).getName(),
beforeCols.get(i).getType(),
afterCols.get(i).getName(), afterCols.get(i).getType());
--
To view, visit http://gerrit.cloudera.org:8080/21019
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6fd9a9504bf93d2529dc7accbf436ad83e51d8ac
Gerrit-Change-Number: 21019
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Comment-Date: Mon, 18 Mar 2024 12:30:48 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21151 ) Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/21151/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21151/1//COMMIT_MSG@21 PS1, Line 21: test_type_conversions_hive2 Maybe one day we want to add this back for IMPALA-12349. It's ok to remove it here. -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 18 Mar 2024 13:05:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10380/ -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Mon, 18 Mar 2024 14:59:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21151 ) Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/21151/2/tests/query_test/test_scanners.py File tests/query_test/test_scanners.py: http://gerrit.cloudera.org:8080/#/c/21151/2/tests/query_test/test_scanners.py@1717 PS2, Line 1717: flake8: W291 trailing whitespace http://gerrit.cloudera.org:8080/#/c/21151/2/tests/query_test/test_scanners.py@1717 PS2, Line 1717: # TODO(IMPALA-12349): Rename the columns to use the correct names (see line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 18 Mar 2024 16:14:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21151 to look at the new patch set (#2). Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions test_type_conversions_hive3 silently passes because we are not creating the test dimenstion for query option orc_shema_resolution correctly. If we set orc_shema_resolution correctly, i.e. to also exercise the name-based schema resolution, the test fails. The cause of the failure is that the ill-typed tables have dummy column names like 'c1', 'c2', etc. These are completely fine for position-based schema resolution, but it is not OK for name-based schema resolution. The test just wants to check error messages related to type errors, the column names are irrelevant, so we can just use the correct names. The test was copied from the old test_type_conversions_hive2 which is not relevant anymore, so this CR also removes it. Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 --- M testdata/workloads/functional-query/queries/DataErrorsTest/orc-type-checks.test M tests/query_test/test_scanners.py 2 files changed, 44 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/21151/2 -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21151 to look at the new patch set (#3). Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions test_type_conversions_hive3 silently passes because we are not creating the test dimenstion for query option orc_shema_resolution correctly. If we set orc_shema_resolution correctly, i.e. to also exercise the name-based schema resolution, the test fails. The cause of the failure is that the ill-typed tables have dummy column names like 'c1', 'c2', etc. These are completely fine for position-based schema resolution, but it is not OK for name-based schema resolution. The test just wants to check error messages related to type errors, the column names are irrelevant, so we can just use the correct names. The test was copied from the old test_type_conversions_hive2 which is not relevant anymore, so this CR also removes it. Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 --- M testdata/workloads/functional-query/queries/DataErrorsTest/orc-type-checks.test M tests/query_test/test_scanners.py 2 files changed, 44 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/21151/3 -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21151 to look at the new patch set (#4). Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions test_type_conversions_hive3 silently passes because we are not creating the test dimenstion for query option orc_shema_resolution correctly. If we set orc_shema_resolution correctly, i.e. to also exercise the name-based schema resolution, the test fails. The cause of the failure is that the ill-typed tables have dummy column names like 'c1', 'c2', etc. These are completely fine for position-based schema resolution, but it is not OK for name-based schema resolution. The test just wants to check error messages related to type errors, the column names are irrelevant, so we can just use the correct names. Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 --- M testdata/workloads/functional-query/queries/DataErrorsTest/orc-type-checks.test M tests/query_test/test_scanners.py 2 files changed, 44 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/21151/4 -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21151 ) Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. Patch Set 4: (3 comments) Thanks for the comments! http://gerrit.cloudera.org:8080/#/c/21151/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21151/1//COMMIT_MSG@21 PS1, Line 21: f3f3b1427b20a1d2d28 > Maybe one day we want to add this back for IMPALA-12349. It's ok to remove Thanks for pointing me to IMPALA-12349. If we have such plans then I think we shouldn't remove test_type_conversions_hive2 because we might forget to re-add it later. I didn't fix the column names in test_type_conversions_hive2 because I cannot test it, but at least I've left a hint for the future contributor of IMPALA-12349. http://gerrit.cloudera.org:8080/#/c/21151/2/tests/query_test/test_scanners.py File tests/query_test/test_scanners.py: http://gerrit.cloudera.org:8080/#/c/21151/2/tests/query_test/test_scanners.py@1717 PS2, Line 1717: # TODO(IMPALA-12349): Rename the columns to use the correct names (see > line has trailing whitespace Done http://gerrit.cloudera.org:8080/#/c/21151/2/tests/query_test/test_scanners.py@1717 PS2, Line 1717: > flake8: W291 trailing whitespace Done -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 18 Mar 2024 16:18:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11938: added definition of num nodes query option for test in QueryOptions.SetIntOptions
j_ansh...@yahoo.in has abandoned this change. ( http://gerrit.cloudera.org:8080/21150 ) Change subject: IMPALA-11938: added definition of num_nodes query option for test in QueryOptions.SetIntOptions .. Abandoned by mistake commit JIRA IMPALA-11938 without amend -- To view, visit http://gerrit.cloudera.org:8080/21150 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Iccfb7617cc9959f6bc6f1df7c36e47b65972d586 Gerrit-Change-Number: 21150 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21151 ) Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15552/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 18 Mar 2024 16:37:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21151 ) Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15553/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 18 Mar 2024 16:42:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12872: Use Calcite for ...
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for ... .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/21109/5/fe/pom.xml File fe/pom.xml: http://gerrit.cloudera.org:8080/#/c/21109/5/fe/pom.xml@616 PS5, Line 616: : org.apache.calcite : calcite-core : 1.36.0 : : : org.apache.calcite.avatica : avatica-core : 1.23.0 : > I think fe doesn't actually need these dependencies, and they could be decl Yeah. This was an interesting dilemma for me I put this in the fe/pom.xml because it made my life slightly easier. Eventually we should put this Calcite code under fe and this would be the place it would go. But by having it under the "java" subdirectory, it means that the jar files from Calcite won't be in the main target directory. So I would either have to come up with a build mechanism under the java file with its own "target" dir that contains all the dependent jar files or shade them into the Impala Calcite jar file. And this would only be presumably temporary. But it does make sense to do it this way because otherwise we are putting Calcite in the distribution when we are not really using it. Given all this, how do you think I should handle it? -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Mon, 18 Mar 2024 17:12:30 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12900: Build binutils with -O3
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/21145 ) Change subject: IMPALA-12900: Build binutils with -O3 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21145 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e75db0759b4d3d4e6cc2ce929b1741808f1b771 Gerrit-Change-Number: 21145 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 18 Mar 2024 18:43:45 + Gerrit-HasComments: No
[native-toolchain-CR] Omnibus: Add new versions of several libraries
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/21144 ) Change subject: Omnibus: Add new versions of several libraries .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21144 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a900bc9b0c939391a3c6f970b1b5d99fbb42fcf Gerrit-Change-Number: 21144 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 18 Mar 2024 18:43:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Query History Table
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20770 ) Change subject: IMPALA-12426: Query History Table .. Patch Set 52: Code-Review-1 (1 comment) http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@422 PS52, Line 422: private static final List impalaSysTables = Arrays.asList( : BackendConfig.INSTANCE.queryLogTableName()); Several FE test broken in exhaustive test. Running CatalogHmsSyncToLatestEventIdTest in my local machine leads to NPE on this field. Caused by: java.lang.NullPointerException at org.apache.impala.catalog.CatalogServiceCatalog.(CatalogServiceCatalog.java:423) ... 19 more Switching this to non-static may help. -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 52 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 18 Mar 2024 22:02:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21151 ) Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 18 Mar 2024 22:28:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluser Tests
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/21153 )
Change subject: IMPALA-12913: Refactor Workload Management Custom Cluser Tests
..
Patch Set 1:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/21153/1/testdata/datasets/functional/functional_schema_template.sql
File testdata/datasets/functional/functional_schema_template.sql:
http://gerrit.cloudera.org:8080/#/c/21153/1/testdata/datasets/functional/functional_schema_template.sql@1804
PS1, Line 1804: CREATE EXTERNAL TABLE IF NOT EXISTS
{db_name}{db_suffix}.{table_name} (
> There seem to be support for all this above with COLUMNS, ROW_FORMAT, DEPEN
Agree, using COLUMNS section might be better.
http://gerrit.cloudera.org:8080/#/c/21153/1/tests/custom_cluster/test_query_log.py
File tests/custom_cluster/test_query_log.py:
http://gerrit.cloudera.org:8080/#/c/21153/1/tests/custom_cluster/test_query_log.py@44
PS1, Line 44: super(TestQueryLogTableBase, cls).add_test_dimensions()
Add the following after L44
cls.ImpalaTestMatrix.add_dimension(create_single_exec_option_dimension())
This will only test with disable_codegen_options=False.
http://gerrit.cloudera.org:8080/#/c/21153/1/tests/custom_cluster/test_query_log.py@438
PS1, Line 438: client =
self.create_impala_client(protocol=vector.get_value('protocol'))
> Why do you create a client?
ImpalaTestSuite and its subclasses initialize 3 clients for each test: client
(beeswax), hs2_client (HS2), and hs2_http_client (HS2 over HTTP)
https://github.com/apache/impala/blob/4477398ae46415d3fb32db2a8fd5e6d2060cbd3f/tests/common/impala_test_suite.py#L322-L353
run_test_case() use them, and they are closed automatically after test run
complete. So no need to declare try-finally. But the benefit stops there.
Reusing them is probably good if test only need one client of that kind. But
once the test gets complex (using 2 client like this method, or using
create_client_for_nth_impalad), I think it is fine to doo it this way too.
http://gerrit.cloudera.org:8080/#/c/21153/1/tests/custom_cluster/test_query_log.py@614
PS1, Line 614: @pytest.mark.parametrize("buffer_pool_limit", [None, "14.97MB"])
: def test_query_log_table_query_select(self, vector,
buffer_pool_limit):
Contain this test in its own test class, and define buffer_pool_limit test
dimension just for this. For example:
class TestQueryLogTableBufferPool(CustomClusterTestSuite):
"""Tests to assert the query log table is correctly populated."""
@classmethod
def add_test_dimensions(cls):
super(TestQueryLogTableBufferPool, cls).add_test_dimensions()
add_exec_option_dimension(cls, 'buffer_pool_limit', [None, '16.05MB'])
add_mandatory_exec_option(cls, 'max_mem_estimate_for_admission', '10MB')
@CustomClusterTestSuite.with_args(impalad_args="--enable_workload_mgmt "
"--query_log_write_interval_s=1 "
"--cluster_id=test_query_hist_1 "
"--shutdown_grace_period_s=10 "
"--shutdown_deadline_s=60 "
"--scratch_dirs={0}:5G"
.format(SCRATCH_DIR),
catalogd_args="--enable_workload_mgmt",
impalad_graceful_shutdown=True)
def test_query_log_table_query_select(self, vector, unique_database):
"""Asserts the values written to the query log table match the values from
the
query profile. If the buffer_pool_limit parameter is not None, then this
test
requires that the query spills to disk to assert that the spill metrics
are correct
in the completed queries table."""
tbl_name = unique_database + ".test_query_log"
client_protocol = vector.get_value('protocol')
buffer_pool_limit = vector.get_value('exec_option')['buffer_pool_limit']
client = self.create_impala_client(protocol=client_protocol)
http://gerrit.cloudera.org:8080/#/c/21153/1/tests/custom_cluster/test_query_log.py@635
PS1, Line 635: client.set_configuration_option("SPOOL_QUERY_RESULTS", "TRUE")
Can probably remove this, since it is True by default.
--
To view, visit http://gerrit.cloudera.org:8080/21153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b
Gerrit-Change-Number: 21153
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Mon, 18 Mar 2024 22:28:06 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12872: Use Calcite for ...
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for ... .. Patch Set 5: (9 comments) http://gerrit.cloudera.org:8080/#/c/21109/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21109/5//COMMIT_MSG@7 PS5, Line 7: IMPALA-12872: Use Calcite for ... nit: combine the two lines. it seems short enough. How about 'Use Calcite for optimization - part 1: simple queries' http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/pom.xml File java/experimental-planner/pom.xml: http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/pom.xml@29 PS5, Line 29: experimental-planner you could call it calcite-planner since there's a clear association. I would think that if there was a configuration option introduced (at a later time) to enable this planner, it would likely be something like 'use_new_planner' or 'use_calcite_planner' rather than use_experimental_planner. http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java File java/experimental-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java: http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java@33 PS5, Line 33: analyzer nit:use underscore to be consistent http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java@36 PS5, Line 36: ctx nit: use underscore to be consistent with other variable names. http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java@39 PS5, Line 39: public final Class parentClass_; We should use the actual base class name (RelNode/ImpalaPlanRel) instead of the generic Class. http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java File java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java: http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java@31 PS5, Line 31: tableMap nit: use underscore http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java File java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java: http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java@78 PS5, Line 78: private List foreignKeys; This field is not used in this implementation. Is it for future use ? http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java File java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java: http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java@87 PS5, Line 87: LOG.debug(getDebugString(resultObject)); nit: should this logging be in an else block ? http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/src/main/java/org/apache/impala/calcite/util/NotSupported.java File java/experimental-planner/src/main/java/org/apache/impala/calcite/util/NotSupported.java: http://gerrit.cloudera.org:8080/#/c/21109/5/java/experimental-planner/src/main/java/org/apache/impala/calcite/util/NotSupported.java@33 PS5, Line 33: nit: missing 'are' . Here and subsequent messages. -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Mon, 18 Mar 2024 22:58:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12426: Query History Table
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20770 ) Change subject: IMPALA-12426: Query History Table .. Patch Set 52: (1 comment) http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@444 PS52, Line 444: return !impalaSysTables.contains(table.getTbl()); Can replace with return !BackendConfig.INSTANCE.queryLogTableName().equals(table.getTbl()); -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 52 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 18 Mar 2024 23:05:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21151 ) Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 18 Mar 2024 23:43:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21151 ) Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10381/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 18 Mar 2024 23:43:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21151 ) Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. Patch Set 4: Code-Review+2 Carry Riza's +1. -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 18 Mar 2024 23:42:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21151 ) Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. Patch Set 4: Thank Riza and Zoltan for finding this issue! Merging this. -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 18 Mar 2024 23:43:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: [WIP] Support locally built Ranger
Fang-Yu Rao has uploaded this change for review. (
http://gerrit.cloudera.org:8080/21160
Change subject: IMPALA-12921: [WIP] Support locally built Ranger
..
IMPALA-12921: [WIP] Support locally built Ranger
This patch adds the support for locally built Ranger in Impala's
minicluster. Suppose that we have already built Ranger locally under the
folder $RANGER_SRC_DIR, and that the produced tar files are under
$RANGER_SRC_DIR/target. It suffices to execute the following to start
the locally built Ranger server.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
3. source $IMPALA_HOME/bin/impala-config.sh
4. tar zxv $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz
5. $IMPALA_HOME/bin/create-test-configuration.sh
6. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
7. $IMPALA_HOME/testdata/bin/run-ranger-server.sh
8. $IMPALA_HOME/testdata/bin/setup-ranger.sh
To-do:
- Currently the GRANT/REVOKE APIs do not work with Apache Impala.
We should figure out why 'userName' in
ServiceREST#ensureAdminAccess() is null.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
---
M bin/impala-config.sh
M testdata/bin/setup-ranger.sh
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
5 files changed, 25 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/1
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao
[Impala-ASF-CR] IMPALA-12921: [WIP] Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: [WIP] Support locally built Ranger .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15554/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Mar 2024 01:38:19 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12915: Build shared library of GTest
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21161 Change subject: IMPALA-12915: Build shared library of GTest .. IMPALA-12915: Build shared library of GTest When building Impala with shared libraries, we currently link against the static library of GTest in both libkudu_test_util.so and unifiedbetests. libkudu_test_util.so is also linked by unifiedbetests. When unifiedbetests exits, both binaries try to delete the global variables of GTest, which leads to a double-free memory issue. This patch builds both the shared and static library of GTest so Impala can pick the corresponding library correctly. Test: - Built GTest locally and verify the issue in Impala got resolved Change-Id: Ie34023fe4fb0691fbca1dc547d1e40ada1beca22 --- M source/gtest/build.sh 1 file changed, 13 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/61/21161/1 -- To view, visit http://gerrit.cloudera.org:8080/21161 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie34023fe4fb0691fbca1dc547d1e40ada1beca22 Gerrit-Change-Number: 21161 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-12896 (Part 2): JDBC table must be created as external table
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21159 Change subject: IMPALA-12896 (Part 2): JDBC table must be created as external table .. IMPALA-12896 (Part 2): JDBC table must be created as external table In some of the deployment environments, default table type is transactional. In these scenarios, JDBC tables which are created as non external table are not accepted by HMS due to strict managed table check failures. This patch forces JDBC tables to be created as external table, and requires at least 1 column for JDBC tables. Testing: - Updated frontend unit tests and end-to-end unit tests to create JDBC tables as external tables. - Passed core tests Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9 --- M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java M testdata/bin/create-ext-data-source-table.sql M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source-with-keystore.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 9 files changed, 37 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/21159/1 -- To view, visit http://gerrit.cloudera.org:8080/21159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9 Gerrit-Change-Number: 21159 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat
[Impala-ASF-CR] IMPALA-12896 (Part 2): JDBC table must be created as external table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21159 ) Change subject: IMPALA-12896 (Part 2): JDBC table must be created as external table .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9 Gerrit-Change-Number: 21159 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 19 Mar 2024 03:31:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21151 ) Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Mar 2024 04:32:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12904: test type conversions hive3 silently passes because of wrongly defined test dimensions
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21151 ) Change subject: IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions .. IMPALA-12904: test_type_conversions_hive3 silently passes because of wrongly defined test dimensions test_type_conversions_hive3 silently passes because we are not creating the test dimenstion for query option orc_shema_resolution correctly. If we set orc_shema_resolution correctly, i.e. to also exercise the name-based schema resolution, the test fails. The cause of the failure is that the ill-typed tables have dummy column names like 'c1', 'c2', etc. These are completely fine for position-based schema resolution, but it is not OK for name-based schema resolution. The test just wants to check error messages related to type errors, the column names are irrelevant, so we can just use the correct names. Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Reviewed-on: http://gerrit.cloudera.org:8080/21151 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/workloads/functional-query/queries/DataErrorsTest/orc-type-checks.test M tests/query_test/test_scanners.py 2 files changed, 44 insertions(+), 36 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I786a5eaae9243b4728484f3f3b1427b20a1d2d28 Gerrit-Change-Number: 21151 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12898: Tidy up test dimensions of test scanner.py
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21162 Change subject: IMPALA-12898: Tidy up test dimensions of test_scanner.py .. IMPALA-12898: Tidy up test dimensions of test_scanner.py This patch tidies up the test dimensions of test_scanner.py. 'exec_option' initialization is moved to add_test_dimensions() method as much as possible. It ensures correct permutation and execution of test cases. After this patch, the total collected tests of test_scanner.py is 993 for core/pairwise exploration and 7514 for exhaustive exploration. Before, they were 794 and 11864 accordingly. The increase in test count after refactoring with core exploration is because exec option dimensions are now permuted correctly along with other default exec option dimensions. The reduction in exhaustive exploration is due to a reduction in the overall dimension to permute and a reduction in test skipping (the test was run, but only called pytest.skip()). Testing: - Pass query_test/test_scanners.py in exhaustive exploration. Change-Id: I5efd2b483338fb55b958d8e1a0acf6b365f8093e --- M tests/query_test/test_scanners.py 1 file changed, 99 insertions(+), 89 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/21162/1 -- To view, visit http://gerrit.cloudera.org:8080/21162 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5efd2b483338fb55b958d8e1a0acf6b365f8093e Gerrit-Change-Number: 21162 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto
[Impala-ASF-CR] IMPALA-12898: Tidy up test dimensions of test scanner.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21162 ) Change subject: IMPALA-12898: Tidy up test dimensions of test_scanner.py .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15556/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21162 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5efd2b483338fb55b958d8e1a0acf6b365f8093e Gerrit-Change-Number: 21162 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Mar 2024 05:30:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12898: Tidy up test dimensions of test scanner.py
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21162 ) Change subject: IMPALA-12898: Tidy up test dimensions of test_scanner.py .. Patch Set 1: This is a precursor to IMPALA-12922. -- To view, visit http://gerrit.cloudera.org:8080/21162 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5efd2b483338fb55b958d8e1a0acf6b365f8093e Gerrit-Change-Number: 21162 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Mar 2024 05:30:36 + Gerrit-HasComments: No
