[Impala-ASF-CR](branch-4.1.1) IMPALA-11345: Parquet Bloom filtering failure if column is added to the schema
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/1 ) Change subject: IMPALA-11345: Parquet Bloom filtering failure if column is added to the schema .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8484/ -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ief3e6b6358d3dff3abe5beeda752033a7e8e16a6 Gerrit-Change-Number: 1 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Aug 2022 07:32:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10069: Support TLS 1.3 ciphersuites
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18316 ) Change subject: IMPALA-10069: Support TLS 1.3 ciphersuites .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8486/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18316 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6974dae7fb429599847165614adc4eaaf338f744 Gerrit-Change-Number: 18316 Gerrit-PatchSet: 7 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 07:33:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11394: Update jackson-databind to 2.12.6.1
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18891 ) Change subject: IMPALA-11394: Update jackson-databind to 2.12.6.1 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18891 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8db403a102097a22c48f5d9d42ced3b85930078f Gerrit-Change-Number: 18891 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 07:39:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11513: Upgrade postregresql packge to 42.4.1
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18890 ) Change subject: IMPALA-11513: Upgrade postregresql packge to 42.4.1 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic891b75a8f3383c6bb1229a41033896873441e21 Gerrit-Change-Number: 18890 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 07:40:13 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11447: Fix crash when fetching arrays/structs with result caching
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18889 ) Change subject: IMPALA-11447: Fix crash when fetching arrays/structs with result caching .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dd8f76187dc3555207e2d30d46d811e0a7a126 Gerrit-Change-Number: 18889 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 07:49:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10069: Support TLS 1.3 ciphersuites
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18316 ) Change subject: IMPALA-10069: Support TLS 1.3 ciphersuites .. Patch Set 7: Code-Review+2 Carry +1 from Michael -- To view, visit http://gerrit.cloudera.org:8080/18316 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6974dae7fb429599847165614adc4eaaf338f744 Gerrit-Change-Number: 18316 Gerrit-PatchSet: 7 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 08:27:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11429: Set table owner after creating an Iceberg table
Hello Tamas Mate, Gergely Fürnstáhl, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18837 to look at the new patch set (#7). Change subject: IMPALA-11429: Set table owner after creating an Iceberg table .. IMPALA-11429: Set table owner after creating an Iceberg table Iceberg tables are created using Apache Iceberg's API. However, currently Iceberg gets the owner of the process running Iceberg for the owner of the newly created tables. In our case it's the user running catalogd and not the user running the CREATE TABLE statement. Until the Iceberg API is enhanced to accept an owner parameter for table creation, as a workaround this patch adds an extra alter table step right after Iceberg table creation. TestIcebergTable.test_drop_incomplete_table test had to be skipped with this implementation because this would run into a known bug where Impala runs into an infinite loop when the table location is being dropped somewhere during the execution of the first command after CREATE TABLE. In this case the ALTER TABLE SET OWNER is going to be the first command, so IMPALA-11509 would be triggered in case we dropped the table location as this test would. Testing: - Manually creating Iceberg tables and checking the owner. - Added one automated test to create Iceberg tables with different users and checking the output of DESCRIBE FORMATTED to verify the owner. - Turned off TestIcebergTable.test_drop_incomplete_table as described above. Change-Id: I5cac198a4a53be3599cb582864ee5f8c269202c0 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/query_test/test_iceberg.py 2 files changed, 63 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/18837/7 -- To view, visit http://gerrit.cloudera.org:8080/18837 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5cac198a4a53be3599cb582864ee5f8c269202c0 Gerrit-Change-Number: 18837 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-11429: Set table owner after creating an Iceberg table
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/18837 )
Change subject: IMPALA-11429: Set table owner after creating an Iceberg table
..
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18837/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/18837/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3633
PS5, Line 3633: throw new ImpalaRuntimeException("Failed to set table
owner after creating " +
: "Iceberg table but the table has been created
successfully", e);
> I was thinking about a rollback, which is probably a drop table in this cas
Rollback would in fact be a good choice here but I don't think I should
implement that at this point for the following reasons:
- If the ALTER TABLE part fails then we are where we were without this patch so
actually we haven't made made anything worse, but with the ALTER TABLE part
succeeding we get to the state we wanted to.
- This ALTER TABLE step is a temporary one until we manage to enhance the
Iceberg API to also accept a table owner during table creation.
- Adding one more step would result a CREATE TABLE as a 3 step operation in
some use cases and I feel that over complicated and too error prone.
- If the "automatic" ALTER TABLE fails there is still the opportunity to run a
manual ALTER TABLE with a user with enough permissions to fix the table.
http://gerrit.cloudera.org:8080/#/c/18837/5/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:
http://gerrit.cloudera.org:8080/#/c/18837/5/tests/query_test/test_iceberg.py@130
PS5, Line 130: loading fails
> Thank you for explaining. Could you add this Jira to the above comment as w
Done
--
To view, visit http://gerrit.cloudera.org:8080/18837
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5cac198a4a53be3599cb582864ee5f8c269202c0
Gerrit-Change-Number: 18837
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Comment-Date: Tue, 23 Aug 2022 08:47:08 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11429: Set table owner after creating an Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18837 ) Change subject: IMPALA-11429: Set table owner after creating an Iceberg table .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11208/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18837 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5cac198a4a53be3599cb582864ee5f8c269202c0 Gerrit-Change-Number: 18837 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 23 Aug 2022 09:07:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11521: Skip test binary type for HBase
Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18895 Change subject: IMPALA-11521: Skip test_binary_type for HBase .. IMPALA-11521: Skip test_binary_type for HBase Skip this test for builds that don't have HBase. Change-Id: Id41049094b632af6326f6ee9f3886577d1fc5ee6 --- M tests/query_test/test_scanners.py 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/18895/1 -- To view, visit http://gerrit.cloudera.org:8080/18895 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id41049094b632af6326f6ee9f3886577d1fc5ee6 Gerrit-Change-Number: 18895 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer
[Impala-ASF-CR](branch-4.1.1) IMPALA-11447: Fix crash when fetching arrays/structs with result caching
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18889 ) Change subject: IMPALA-11447: Fix crash when fetching arrays/structs with result caching .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8485/ -- To view, visit http://gerrit.cloudera.org:8080/18889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dd8f76187dc3555207e2d30d46d811e0a7a126 Gerrit-Change-Number: 18889 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 09:17:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11394: Update jackson-databind to 2.12.6.1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18891 ) Change subject: IMPALA-11394: Update jackson-databind to 2.12.6.1 .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8487/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18891 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8db403a102097a22c48f5d9d42ced3b85930078f Gerrit-Change-Number: 18891 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 09:17:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11521: Skip test binary type for HBase
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18895 ) Change subject: IMPALA-11521: Skip test_binary_type for HBase .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11209/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18895 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id41049094b632af6326f6ee9f3886577d1fc5ee6 Gerrit-Change-Number: 18895 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Aug 2022 09:33:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11520: Remove functional.unsupported types misc test
Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18896 Change subject: IMPALA-11520: Remove functional.unsupported_types misc test .. IMPALA-11520: Remove functional.unsupported_types misc test IMPALA-9482 added support to the remaining Hive types and removed the functional.unsupported_types table. There was a misc exhaustive test left, this commit removes this test. Change-Id: I65b6ea5ac742fbcc427ad41741d347558cb7d110 --- M testdata/workloads/functional-query/queries/QueryTest/misc.test 1 file changed, 0 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/18896/1 -- To view, visit http://gerrit.cloudera.org:8080/18896 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I65b6ea5ac742fbcc427ad41741d347558cb7d110 Gerrit-Change-Number: 18896 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer
[Impala-ASF-CR] IMPALA-11520: Remove functional.unsupported types misc test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18896 ) Change subject: IMPALA-11520: Remove functional.unsupported_types misc test .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11210/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18896 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I65b6ea5ac742fbcc427ad41741d347558cb7d110 Gerrit-Change-Number: 18896 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Aug 2022 09:52:29 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11489: Fix int overflow in >2GB ORC files
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18886 ) Change subject: IMPALA-11489: Fix int overflow in >2GB ORC files .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Id8438d3512b0ebaafd319f369c1f6871d980a599 Gerrit-Change-Number: 18886 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Aug 2022 10:06:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11520: Remove functional.unsupported types misc test
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18896 ) Change subject: IMPALA-11520: Remove functional.unsupported_types misc test .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/18896/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18896/1//COMMIT_MSG@10 PS1, Line 10: There was a misc exhaustive test : left This was a bit sneaky, as the test is not explicitly "exhaustive only", but it it is skipped for several file formats, including Parquet,while the test suite only runs with Parquet in core runs. Can you add a TODO about this to https://github.com/apache/impala/blob/master/tests/query_test/test_queries.py#L186 ? -- To view, visit http://gerrit.cloudera.org:8080/18896 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I65b6ea5ac742fbcc427ad41741d347558cb7d110 Gerrit-Change-Number: 18896 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Aug 2022 10:25:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11521: Skip test binary type for HBase
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/18895 )
Change subject: IMPALA-11521: Skip test_binary_type for HBase
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18895/1/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:
http://gerrit.cloudera.org:8080/#/c/18895/1/tests/query_test/test_scanners.py@1835
PS1, Line 1835: @SkipIfFS.hbase
: def test_binary_type(self, vector):
: self.run_test_case('QueryTest/binary-type', vector)
Can you change the binary-type.test?
I think that this was a typo of mine:
https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/binary-type.test#L139
It always uses functional_hbase for one of the sides of the join without any
good reason.
--
To view, visit http://gerrit.cloudera.org:8080/18895
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id41049094b632af6326f6ee9f3886577d1fc5ee6
Gerrit-Change-Number: 18895
Gerrit-PatchSet: 1
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Tue, 23 Aug 2022 10:28:09 +
Gerrit-HasComments: Yes
[Impala-ASF-CR](branch-4.1.1) IMPALA-11447: Fix crash when fetching arrays/structs with result caching
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18889 ) Change subject: IMPALA-11447: Fix crash when fetching arrays/structs with result caching .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dd8f76187dc3555207e2d30d46d811e0a7a126 Gerrit-Change-Number: 18889 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 10:34:07 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11447: Fix crash when fetching arrays/structs with result caching
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18889 ) Change subject: IMPALA-11447: Fix crash when fetching arrays/structs with result caching .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8488/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dd8f76187dc3555207e2d30d46d811e0a7a126 Gerrit-Change-Number: 18889 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 10:57:34 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11447: Fix crash when fetching arrays/structs with result caching
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18889 ) Change subject: IMPALA-11447: Fix crash when fetching arrays/structs with result caching .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dd8f76187dc3555207e2d30d46d811e0a7a126 Gerrit-Change-Number: 18889 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 10:57:33 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11489: Fix int overflow in >2GB ORC files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18886 ) Change subject: IMPALA-11489: Fix int overflow in >2GB ORC files .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8489/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Id8438d3512b0ebaafd319f369c1f6871d980a599 Gerrit-Change-Number: 18886 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Aug 2022 10:58:07 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11489: Fix int overflow in >2GB ORC files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18886 ) Change subject: IMPALA-11489: Fix int overflow in >2GB ORC files .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Id8438d3512b0ebaafd319f369c1f6871d980a599 Gerrit-Change-Number: 18886 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Aug 2022 10:58:06 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11447: Fix crash when fetching arrays/structs with result caching
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18889 ) Change subject: IMPALA-11447: Fix crash when fetching arrays/structs with result caching .. Patch Set 1: > Patch Set 1: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8485/ The failure is in bin/gen-cscope.sh: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/17358/testReport/junit/generate_junitxml/build/dd4ed8bcce2f4f6482c62052304e5da4/ I think it's unrelated. Retriggering the job. -- To view, visit http://gerrit.cloudera.org:8080/18889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dd8f76187dc3555207e2d30d46d811e0a7a126 Gerrit-Change-Number: 18889 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 10:57:11 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11295: Deflake TestParquet.test multiple blocks mt dop
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18892 ) Change subject: IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4 Gerrit-Change-Number: 18892 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Aug 2022 11:04:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10918: Allow map type in SELECT list
Daniel Becker has uploaded a new patch set (#12). (
http://gerrit.cloudera.org:8080/18736 )
Change subject: IMPALA-10918: Allow map type in SELECT list
..
IMPALA-10918: Allow map type in SELECT list
Adding support for MAP types in the select list.
An example of how maps are printed:
{"k1":2,"k2":null}
Nested collection types (maps and arrays) are supported in any
combination. However, structs in collections and collections in structs
are not supported.
Limitations (other than map support) as described in the commit for
IMPALA-9498 still apply, the following are to be implemented later:
- Unify HS2 / Beeswax logic with the way STRUCTs are handled.
This could be done in a "final" logic that can handle
STRUCTS/ARRAYS nested to each other
- Implement "deep copy" and "deep serialize" for collections in BE.
This would enable all operators, e.g. ORDER BY and UNION.
Testing:
- modified the FE tests that checked that maps were not allowed in the
select list - now the test expect maps are allowed there
- added FE and EE tests involving maps based on the array tests
Change-Id: I921c647f1779add36e7f5df4ce6ca237dcfaf001
---
M be/src/codegen/codegen-anyval.cc
M be/src/runtime/raw-value.cc
M be/src/runtime/raw-value.h
M be/src/runtime/types.h
M be/src/service/hs2-util.cc
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-server.h
M be/src/service/query-result-set.cc
M be/src/service/query-result-set.h
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeUpsertStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M
testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
A
testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
M
testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_complex_types.test
M
testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_struct_in_select_list.test
M
testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test
M tests/authorization/test_ranger.py
M tests/query_test/test_nested_types.py
26 files changed, 652 insertions(+), 140 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/18736/12
--
To view, visit http://gerrit.cloudera.org:8080/18736
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I921c647f1779add36e7f5df4ce6ca237dcfaf001
Gerrit-Change-Number: 18736
Gerrit-PatchSet: 12
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10918: Allow map type in SELECT list
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/18736 ) Change subject: IMPALA-10918: Allow map type in SELECT list .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/18736/12/testdata/datasets/functional/functional_schema_template.sql File testdata/datasets/functional/functional_schema_template.sql: http://gerrit.cloudera.org:8080/#/c/18736/12/testdata/datasets/functional/functional_schema_template.sql@3529 PS12, Line 3529:) : ); Fixed a parenthesis error here. -- To view, visit http://gerrit.cloudera.org:8080/18736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I921c647f1779add36e7f5df4ce6ca237dcfaf001 Gerrit-Change-Number: 18736 Gerrit-PatchSet: 12 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Aug 2022 11:10:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10918: Allow map type in SELECT list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18736 ) Change subject: IMPALA-10918: Allow map type in SELECT list .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11211/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I921c647f1779add36e7f5df4ce6ca237dcfaf001 Gerrit-Change-Number: 18736 Gerrit-PatchSet: 12 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Aug 2022 11:32:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10069: Support TLS 1.3 ciphersuites
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18316 ) Change subject: IMPALA-10069: Support TLS 1.3 ciphersuites .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18316 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6974dae7fb429599847165614adc4eaaf338f744 Gerrit-Change-Number: 18316 Gerrit-PatchSet: 7 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 12:21:36 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11317/IMPALA-11316/IMPALA-11315: impala-shell Python 3 fixes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18885 ) Change subject: IMPALA-11317/IMPALA-11316/IMPALA-11315: impala-shell Python 3 fixes .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8490/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ie58380a17d7e011f4ce96b27d34717509a0b80a6 Gerrit-Change-Number: 18885 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 13:51:46 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18887 ) Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column .. Patch Set 2: TestIcebergTable.test_migrated_tables fails: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/17356/ https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/6199/ I'll try to fix them. -- To view, visit http://gerrit.cloudera.org:8080/18887 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4 Gerrit-Change-Number: 18887 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 23 Aug 2022 13:48:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11521: Fix test binary type
Tamas Mate has posted comments on this change. (
http://gerrit.cloudera.org:8080/18895 )
Change subject: IMPALA-11521: Fix test_binary_type
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18895/1/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:
http://gerrit.cloudera.org:8080/#/c/18895/1/tests/query_test/test_scanners.py@1835
PS1, Line 1835: @SkipIfFS.hbase
: def test_binary_type(self, vector):
: self.run_test_case('QueryTest/binary-type', vector)
> Can you change the binary-type.test?
Oh, I see, thanks! :)
--
To view, visit http://gerrit.cloudera.org:8080/18895
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id41049094b632af6326f6ee9f3886577d1fc5ee6
Gerrit-Change-Number: 18895
Gerrit-PatchSet: 1
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Comment-Date: Tue, 23 Aug 2022 13:53:29 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11521: Fix test binary type
Tamas Mate has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18895 ) Change subject: IMPALA-11521: Fix test_binary_type .. IMPALA-11521: Fix test_binary_type Fix test_binary_type typo, it should not reference to an HBase table. Change-Id: Id41049094b632af6326f6ee9f3886577d1fc5ee6 --- M testdata/workloads/functional-query/queries/QueryTest/binary-type.test 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/18895/2 -- To view, visit http://gerrit.cloudera.org:8080/18895 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id41049094b632af6326f6ee9f3886577d1fc5ee6 Gerrit-Change-Number: 18895 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18887 ) Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column .. Patch Set 2: Probably we should backport IMPALA-9410 as well. -- To view, visit http://gerrit.cloudera.org:8080/18887 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4 Gerrit-Change-Number: 18887 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 23 Aug 2022 13:49:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11394: Update jackson-databind to 2.12.6.1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18891 ) Change subject: IMPALA-11394: Update jackson-databind to 2.12.6.1 .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18891 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8db403a102097a22c48f5d9d42ced3b85930078f Gerrit-Change-Number: 18891 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 14:10:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11521: Fix test binary type
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18895 ) Change subject: IMPALA-11521: Fix test_binary_type .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11212/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18895 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id41049094b632af6326f6ee9f3886577d1fc5ee6 Gerrit-Change-Number: 18895 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 23 Aug 2022 14:14:29 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-11454: part-2: create kudu binaries symlinks to reduce package size.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18880 ) Change subject: IMPALA-11454: part-2: create kudu binaries symlinks to reduce package size. .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc1904be06f84acebf056a079dc1b6694032de87 Gerrit-Change-Number: 18880 Gerrit-PatchSet: 1 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Tue, 23 Aug 2022 14:26:12 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11489: Fix int overflow in >2GB ORC files
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18886 ) Change subject: IMPALA-11489: Fix int overflow in >2GB ORC files .. IMPALA-11489: Fix int overflow in >2GB ORC files This can lead to returning an error "Invalid read len.", but can't lead to other problems like reading corrupt data or crashing. The fix is trivial, but creating automatic tests for it seems hard, so no regression test is added. Change-Id: Id8438d3512b0ebaafd319f369c1f6871d980a599 Reviewed-on: http://gerrit.cloudera.org:8080/18827 Reviewed-by: Impala Public Jenkins Reviewed-by: Csaba Ringhofer Tested-by: Csaba Ringhofer Reviewed-on: http://gerrit.cloudera.org:8080/18886 Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-orc-scanner.cc 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: merged Gerrit-Change-Id: Id8438d3512b0ebaafd319f369c1f6871d980a599 Gerrit-Change-Number: 18886 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR](branch-4.1.1) IMPALA-11489: Fix int overflow in >2GB ORC files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18886 ) Change subject: IMPALA-11489: Fix int overflow in >2GB ORC files .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Id8438d3512b0ebaafd319f369c1f6871d980a599 Gerrit-Change-Number: 18886 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Aug 2022 15:28:41 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11447: Fix crash when fetching arrays/structs with result caching
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18889 ) Change subject: IMPALA-11447: Fix crash when fetching arrays/structs with result caching .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8488/ -- To view, visit http://gerrit.cloudera.org:8080/18889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dd8f76187dc3555207e2d30d46d811e0a7a126 Gerrit-Change-Number: 18889 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 15:29:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10069: Support TLS 1.3 ciphersuites
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18316 ) Change subject: IMPALA-10069: Support TLS 1.3 ciphersuites .. IMPALA-10069: Support TLS 1.3 ciphersuites OpenSSL 1.1.1 adds support for TLS 1.3. TLS 1.3 has a new set of cipher suites that are maintained separately from the TLS 1.2 ciphers. This caused test failures on tests that expected failures when setting invalid TLS 1.2 ciphers. It also rendered some success test cases invalid, because the TLS 1.3 ciphers would work even if TLS 1.2 didn't. This adds the tls_ciphersuites startup parameter, which customizes the TLS 1.3 cipher suites. tls_ciphersuites is only effective when Impala is built for a system with OpenSSL 1.1.1 or above. This uses tls_ciphersuites to fix the existing TLS 1.2 cipher tests. It also adds a set of tests for TLS 1.3 cipher suites. KRPC and the webserver now support ssl_minimum_version=tlsv1.3. However, Thrift does not support this configuration yet, so this is not supported for impalad yet. To support TLS 1.3 tests on Thrift, this adds a disable_tls12 option to ThriftServer and ThriftClient. This will be removed when ssl_minimum_version=tlsv1.3 is supported. Testing: - Ran the backend tests with TLS checks (rpc-mgr-test, rpc-mgr-kerberized-test, webserver-test, and thrift-server-test) on Ubuntu 18 and Ubuntu 20 - Added tests for tls_ciphersuites in rpc-mgr-test and thrift-server-test - Ran a core test on Centos 7 - Ran a core test on Ubuntu 16 Change-Id: I6974dae7fb429599847165614adc4eaaf338f744 Reviewed-on: http://gerrit.cloudera.org:8080/18316 Reviewed-by: Wenzhe Zhou Tested-by: Impala Public Jenkins --- M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/thrift-client.cc M be/src/rpc/thrift-client.h M be/src/rpc/thrift-server-test.cc M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/service/impala-server.cc M be/src/thirdparty/squeasel/squeasel.c M be/src/util/webserver-test.cc M be/src/util/webserver.cc 14 files changed, 523 insertions(+), 84 deletions(-) Approvals: Wenzhe Zhou: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/18316 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6974dae7fb429599847165614adc4eaaf338f744 Gerrit-Change-Number: 18316 Gerrit-PatchSet: 8 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11513: Upgrade postregresql packge to 42.4.1
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18890 ) Change subject: IMPALA-11513: Upgrade postregresql packge to 42.4.1 .. Patch Set 1: Verified+1 Top of stack (with jackson-databind version bump) passed GVO -- To view, visit http://gerrit.cloudera.org:8080/18890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic891b75a8f3383c6bb1229a41033896873441e21 Gerrit-Change-Number: 18890 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 15:50:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11394: Update jackson-databind to 2.12.6.1
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18891 ) Change subject: IMPALA-11394: Update jackson-databind to 2.12.6.1 .. IMPALA-11394: Update jackson-databind to 2.12.6.1 This updates jackson-databind to address CVE-2020-36518. Testing: - Ran a core job Change-Id: I8db403a102097a22c48f5d9d42ced3b85930078f Reviewed-on: http://gerrit.cloudera.org:8080/18891 Reviewed-by: Wenzhe Zhou Tested-by: Impala Public Jenkins --- M java/pom.xml 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Wenzhe Zhou: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/18891 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8db403a102097a22c48f5d9d42ced3b85930078f Gerrit-Change-Number: 18891 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11513: Upgrade postregresql packge to 42.4.1
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18890 ) Change subject: IMPALA-11513: Upgrade postregresql packge to 42.4.1 .. IMPALA-11513: Upgrade postregresql packge to 42.4.1 This upgrades postgresql to address CVE-2022-31197. Testing: - Ran core job Change-Id: Ic891b75a8f3383c6bb1229a41033896873441e21 Reviewed-on: http://gerrit.cloudera.org:8080/18890 Reviewed-by: Wenzhe Zhou Tested-by: Joe McDonnell --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Wenzhe Zhou: Looks good to me, approved Joe McDonnell: Verified -- To view, visit http://gerrit.cloudera.org:8080/18890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic891b75a8f3383c6bb1229a41033896873441e21 Gerrit-Change-Number: 18890 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11519: [DOCS] add UTF-8 requirements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18897 ) Change subject: IMPALA-11519: [DOCS] add UTF-8 requirements .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/667/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/18897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7a21c787c66868219c7bd64aa31f772de2f850 Gerrit-Change-Number: 18897 Gerrit-PatchSet: 1 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Aug 2022 16:04:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11519: [DOCS] add UTF-8 requirements
Shajini Thayasingh has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18897 Change subject: IMPALA-11519: [DOCS] add UTF-8 requirements .. IMPALA-11519: [DOCS] add UTF-8 requirements added a note about Glibc version and en_US.UTF-8 locale Change-Id: I4d7a21c787c66868219c7bd64aa31f772de2f850 --- M docs/topics/impala_components.xml M docs/topics/impala_utf_8.xml 2 files changed, 43 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/18897/1 -- To view, visit http://gerrit.cloudera.org:8080/18897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4d7a21c787c66868219c7bd64aa31f772de2f850 Gerrit-Change-Number: 18897 Gerrit-PatchSet: 1 Gerrit-Owner: Shajini Thayasingh
[Impala-ASF-CR] IMPALA-11519: [DOCS] add UTF-8 requirements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18897 ) Change subject: IMPALA-11519: [DOCS] add UTF-8 requirements .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/667/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/18897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7a21c787c66868219c7bd64aa31f772de2f850 Gerrit-Change-Number: 18897 Gerrit-PatchSet: 1 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Aug 2022 16:13:19 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11317/IMPALA-11316/IMPALA-11315: impala-shell Python 3 fixes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18885 ) Change subject: IMPALA-11317/IMPALA-11316/IMPALA-11315: impala-shell Python 3 fixes .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ie58380a17d7e011f4ce96b27d34717509a0b80a6 Gerrit-Change-Number: 18885 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 18:29:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Yida Wu has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/17979 ) Change subject: IMPALA-10791 Add batch reading for remote temporary files .. IMPALA-10791 Add batch reading for remote temporary files The patch adds a feature to batch read from a remote temporary file in order to improve the reading performance for the spilled remote data. Originally, the design is to use the local disk file as the buffer for batch read from the remote file. But in practice, it doesn't help to improve the performance. Therefore, the design is changed to use the memory as the read buffer. Currently, each TmpFileRemote has two DiskFile, one is for the remote, and one is for the local buffer. The patch adds MemBlocks to the local buffer file. Each local buffer file is divided into several MemBlocks evenly. Moreover, in order to guarantee a single page not being cut into two parts in different blocks, the block size could be a little different to each other in practice. The default block size is the minimum value between the default file size and MAX_REMOTE_READ_MEM_BLOCK_THRESHOLD_BYTES, which is 16MB. When pinning a page, the system will detect if there is enough memory for the block that holds the page, if not, we will go reading the page directly and disable this block, because it may be good to avoid duplicated reads from the remote fs for the same content. If the system decides to fetch a block, the block will be stored in the memory until all of the pages in the block are read or the query ends. One challenge of the read buffer is where to get the extra memory for it, because when impala starts to spill data, it means the process lacks of memory to use. By default, impala process will reserve 20% of the total system memory as unused memory, and here we decide to use this unused memory for the read buffer because it is reasonable to use it for the emergency case like spilling. The maximum bytes of the read buffer memory are 10% of the process memory limit, which is a conservative number compared to 20% of the total system memory. The read buffer size plus process memory limit should not exceed the total system memory. Two start options have been added for the new feature. 1. remote_batch_read. Default is false. If set true, the batch read is enabled. 2. remote_read_memory_buffer_size. Default is 1G. The maximum memory that can be used by the read buffer. The number is also restricted by the process memory limit, which can not exceed 10% of the process memory limit. Added metrics ScratchReadsUseMem/ScratchBytesReadUseMem/ ScratchBytesReadUseLocalDisk to the query profile. The patch also increases the MAX_REMOTE_TMPFILE_SIZE_THRESHOLD_MB from 256 to 512. Tests: Ran core and exhaustive tests. Added and ran TmpFileMgrTest::TestBatchReadingFromRemote. Added e2e test test_scratch_dirs_batch_reading. Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b --- M be/src/runtime/io/CMakeLists.txt A be/src/runtime/io/disk-file-test.cc M be/src/runtime/io/disk-file.cc M be/src/runtime/io/disk-file.h M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/request-context.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/mem-info.cc M be/src/util/mem-info.h M be/src/util/metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_scratch_disk.py 19 files changed, 1,587 insertions(+), 161 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/17979/16 -- To view, visit http://gerrit.cloudera.org:8080/17979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Gerrit-Change-Number: 17979 Gerrit-PatchSet: 16 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17979 ) Change subject: IMPALA-10791 Add batch reading for remote temporary files .. Patch Set 16: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11213/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Gerrit-Change-Number: 17979 Gerrit-PatchSet: 16 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Aug 2022 21:24:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17979 ) Change subject: IMPALA-10791 Add batch reading for remote temporary files .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8491/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Gerrit-Change-Number: 17979 Gerrit-PatchSet: 17 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Aug 2022 22:03:29 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11447: Fix crash when fetching arrays/structs with result caching
Quanlong Huang has removed a vote on this change. Change subject: IMPALA-11447: Fix crash when fetching arrays/structs with result caching .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/18889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ia4dd8f76187dc3555207e2d30d46d811e0a7a126 Gerrit-Change-Number: 18889 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR](branch-4.1.1) IMPALA-11447: Fix crash when fetching arrays/structs with result caching
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18889 ) Change subject: IMPALA-11447: Fix crash when fetching arrays/structs with result caching .. IMPALA-11447: Fix crash when fetching arrays/structs with result caching Some parts of HS2ColumnarResultSet were not prepared for returning non-scalar types. This code only runs if impala.resultset.cache.size is set, which is not the case in most of tests. The issue was caught with Hue, which uses result caching. Testing: - Added a regression test in test_fetch_first.py, which contained other tests that used result caching. - It turned out that some tests in the file did not run at all, as @needs_session() needs the parenthesis at the end. For this reason some test fixes were added to run them correctly, though these changes are totally unrelated to the current issue. Backport issue: - Test fails due to STRUCT in SelectList not supported on Parquet. Changed to use the corresponding ORC table. Change-Id: Ia4dd8f76187dc3555207e2d30d46d811e0a7a126 Reviewed-on: http://gerrit.cloudera.org:8080/18768 Reviewed-by: Impala Public Jenkins Reviewed-by: Wenzhe Zhou Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/18889 Tested-by: Quanlong Huang --- M be/src/service/query-result-set.cc M tests/hs2/test_fetch_first.py 2 files changed, 49 insertions(+), 25 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved Quanlong Huang: Verified -- To view, visit http://gerrit.cloudera.org:8080/18889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: merged Gerrit-Change-Id: Ia4dd8f76187dc3555207e2d30d46d811e0a7a126 Gerrit-Change-Number: 18889 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11519: [DOCS] add UTF-8 requirements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18897 ) Change subject: IMPALA-11519: [DOCS] add UTF-8 requirements .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/668/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/18897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7a21c787c66868219c7bd64aa31f772de2f850 Gerrit-Change-Number: 18897 Gerrit-PatchSet: 2 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Aug 2022 22:24:47 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11447: Fix crash when fetching arrays/structs with result caching
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18889 ) Change subject: IMPALA-11447: Fix crash when fetching arrays/structs with result caching .. Patch Set 2: Verified+1 > Patch Set 2: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8488/ The failed test is TestAcid.test_acid_compute_stats: https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/6203 It's tracked in IMPALA-11160 and unrelated to this patch. Merging this. -- To view, visit http://gerrit.cloudera.org:8080/18889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dd8f76187dc3555207e2d30d46d811e0a7a126 Gerrit-Change-Number: 18889 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 22:20:34 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11295: Deflake TestParquet.test multiple blocks mt dop
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18892 ) Change subject: IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4 Gerrit-Change-Number: 18892 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Aug 2022 22:21:11 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11295: Deflake TestParquet.test multiple blocks mt dop
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18892 ) Change subject: IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop .. Patch Set 1: Verified in https://jenkins.impala.io/job/gerrit-verify-dryrun/8490/ -- To view, visit http://gerrit.cloudera.org:8080/18892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4 Gerrit-Change-Number: 18892 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Aug 2022 22:21:44 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11391: Fixed race condition in test drop managed kudu table
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18893 ) Change subject: IMPALA-11391: Fixed race condition in test_drop_managed_kudu_table .. Patch Set 1: Verified+1 Verified in https://jenkins.impala.io/job/gerrit-verify-dryrun/8490/ -- To view, visit http://gerrit.cloudera.org:8080/18893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: I857098c87fcd44d945dd33108bcfdfaa2ca939df Gerrit-Change-Number: 18893 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 23 Aug 2022 22:22:00 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11391: Fixed race condition in test drop managed kudu table
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18893 ) Change subject: IMPALA-11391: Fixed race condition in test_drop_managed_kudu_table .. Patch Set 1: Code-Review+2 +2 since this is a trivial test fix and clean cherrypick -- To view, visit http://gerrit.cloudera.org:8080/18893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: I857098c87fcd44d945dd33108bcfdfaa2ca939df Gerrit-Change-Number: 18893 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 23 Aug 2022 22:22:59 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11317/IMPALA-11316/IMPALA-11315: impala-shell Python 3 fixes
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18885 ) Change subject: IMPALA-11317/IMPALA-11316/IMPALA-11315: impala-shell Python 3 fixes .. Patch Set 2: Code-Review+2 Carry Wenzhe's +2 -- To view, visit http://gerrit.cloudera.org:8080/18885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: comment Gerrit-Change-Id: Ie58380a17d7e011f4ce96b27d34717509a0b80a6 Gerrit-Change-Number: 18885 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Aug 2022 22:23:17 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11391: Fixed race condition in test drop managed kudu table
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18893 ) Change subject: IMPALA-11391: Fixed race condition in test_drop_managed_kudu_table .. IMPALA-11391: Fixed race condition in test_drop_managed_kudu_table test_drop_managed_kudu_table uses exception to verify the deleted table is really missing. Depending on timing, this exception could have been raised in several control pathes with different content. Now the test waits for event processing, meaning Analyzer will consistently catch the missing table and raise the same exception. Change-Id: I857098c87fcd44d945dd33108bcfdfaa2ca939df Reviewed-on: http://gerrit.cloudera.org:8080/18667 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/18893 Tested-by: Quanlong Huang Reviewed-by: Quanlong Huang --- M tests/custom_cluster/test_kudu.py 1 file changed, 7 insertions(+), 2 deletions(-) Approvals: Quanlong Huang: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: merged Gerrit-Change-Id: I857098c87fcd44d945dd33108bcfdfaa2ca939df Gerrit-Change-Number: 18893 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR](branch-4.1.1) IMPALA-11295: Deflake TestParquet.test multiple blocks mt dop
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18892 ) Change subject: IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop .. IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop TestParquet.test_multiple_blocks_mt_dop runs a query on 6 scan ranges using mt_dop=2. It then verifies the sum of ranges read on a backend is 2 (6/3). The test assumes that counters of the 2 instances on the same host are printed consecutively. However, this is not always true. They could be interleaving. This patch makes the test more robust by grouping the counters based on the host. Test - I can't reproduce the issue locally. But I'm able to run the new test 100 times without any error. Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4 Reviewed-on: http://gerrit.cloudera.org:8080/18533 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/18892 Reviewed-by: Csaba Ringhofer Tested-by: Quanlong Huang --- M tests/query_test/test_scanners.py 1 file changed, 14 insertions(+), 4 deletions(-) Approvals: Csaba Ringhofer: Looks good to me, approved Quanlong Huang: Verified -- To view, visit http://gerrit.cloudera.org:8080/18892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: merged Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4 Gerrit-Change-Number: 18892 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR](branch-4.1.1) IMPALA-11317/IMPALA-11316/IMPALA-11315: impala-shell Python 3 fixes
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18885 ) Change subject: IMPALA-11317/IMPALA-11316/IMPALA-11315: impala-shell Python 3 fixes .. IMPALA-11317/IMPALA-11316/IMPALA-11315: impala-shell Python 3 fixes This fixes a few impala-shell Python 3 issues: 1. In ImpalaShell's do_history(), the decode() call needs to be avoided in Python 3, because in Python 3 the cmd is already a string and doesn't need further decoding. (IMPALA-11315) 2. TestImpalaShell.test_http_socket_timeout() gets a different error message in Python 3. It throws the "BlockingIOError" rather than "socker.error". (IMPALA-11316) 3. ImpalaHttpClient.py's code to retrieve the body when handling an HTTP error needs to have a decode() call for the body. Otherwise, the body remains bytes and causes TestImpalaShellInteractive.test_http_interactions_extra() to fail. (IMPALA-11317) Testing: - Ran shell tests in the standard way - Ran shell tests with the impala-shell executable coming from a Python 3 virtualenv using the PyPi package Change-Id: Ie58380a17d7e011f4ce96b27d34717509a0b80a6 Reviewed-on: http://gerrit.cloudera.org:8080/18556 Reviewed-by: Impala Public Jenkins Reviewed-by: Wenzhe Zhou Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/18885 Reviewed-by: Quanlong Huang --- M shell/ImpalaHttpClient.py M shell/impala_shell.py M tests/shell/test_shell_commandline.py 3 files changed, 13 insertions(+), 5 deletions(-) Approvals: Impala Public Jenkins: Verified Quanlong Huang: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-4.1.1 Gerrit-MessageType: merged Gerrit-Change-Id: Ie58380a17d7e011f4ce96b27d34717509a0b80a6 Gerrit-Change-Number: 18885 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11519: [DOCS] add UTF-8 requirements
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18897 ) Change subject: IMPALA-11519: [DOCS] add UTF-8 requirements .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7a21c787c66868219c7bd64aa31f772de2f850 Gerrit-Change-Number: 18897 Gerrit-PatchSet: 2 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Aug 2022 22:27:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11519: [DOCS] add UTF-8 requirements
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18897 to look at the new patch set (#2). Change subject: IMPALA-11519: [DOCS] add UTF-8 requirements .. IMPALA-11519: [DOCS] add UTF-8 requirements added a note about Glibc version and en_US.UTF-8 locale updated the notes in both topics Change-Id: I4d7a21c787c66868219c7bd64aa31f772de2f850 --- M docs/topics/impala_components.xml M docs/topics/impala_utf_8.xml 2 files changed, 43 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/18897/2 -- To view, visit http://gerrit.cloudera.org:8080/18897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4d7a21c787c66868219c7bd64aa31f772de2f850 Gerrit-Change-Number: 18897 Gerrit-PatchSet: 2 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11519: [DOCS] add UTF-8 requirements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18897 ) Change subject: IMPALA-11519: [DOCS] add UTF-8 requirements .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/377/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7a21c787c66868219c7bd64aa31f772de2f850 Gerrit-Change-Number: 18897 Gerrit-PatchSet: 2 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Aug 2022 22:29:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11519: [DOCS] add UTF-8 requirements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18897 ) Change subject: IMPALA-11519: [DOCS] add UTF-8 requirements .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/668/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/18897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7a21c787c66868219c7bd64aa31f772de2f850 Gerrit-Change-Number: 18897 Gerrit-PatchSet: 2 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Aug 2022 22:34:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11519: [DOCS] add UTF-8 requirements
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18897 ) Change subject: IMPALA-11519: [DOCS] add UTF-8 requirements .. IMPALA-11519: [DOCS] add UTF-8 requirements added a note about Glibc version and en_US.UTF-8 locale updated the notes in both topics Change-Id: I4d7a21c787c66868219c7bd64aa31f772de2f850 Reviewed-on: http://gerrit.cloudera.org:8080/18897 Reviewed-by: Quanlong Huang Tested-by: Impala Public Jenkins --- M docs/topics/impala_components.xml M docs/topics/impala_utf_8.xml 2 files changed, 43 insertions(+), 33 deletions(-) Approvals: Quanlong Huang: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/18897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4d7a21c787c66868219c7bd64aa31f772de2f850 Gerrit-Change-Number: 18897 Gerrit-PatchSet: 3 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Yida Wu has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/17979 ) Change subject: IMPALA-10791 Add batch reading for remote temporary files .. IMPALA-10791 Add batch reading for remote temporary files The patch adds a feature to batch read from a remote temporary file in order to improve the reading performance for the spilled remote data. Originally, the design is to use the local disk file as the buffer for batch read from the remote file. But in practice, it doesn't help to improve the performance. Therefore, the design is changed to use the memory as the read buffer. Currently, each TmpFileRemote has two DiskFile, one is for the remote, and one is for the local buffer. The patch adds MemBlocks to the local buffer file. Each local buffer file is divided into several MemBlocks evenly. Moreover, in order to guarantee a single page not being cut into two parts in different blocks, the block size could be a little different to each other in practice. The default block size is the minimum value between the default file size and MAX_REMOTE_READ_MEM_BLOCK_THRESHOLD_BYTES, which is 16MB. When pinning a page, the system will detect if there is enough memory for the block that holds the page, if not, we will go reading the page directly and disable this block, because it may be good to avoid duplicated reads from the remote fs for the same content. If the system decides to fetch a block, the block will be stored in the memory until all of the pages in the block are read or the query ends. One challenge of the read buffer is where to get the extra memory for it, because when impala starts to spill data, it means the process lacks of memory to use. By default, impala process will reserve 20% of the total system memory as unused memory, and here we decide to use this unused memory for the read buffer because it is reasonable to use it for the emergency case like spilling. The maximum bytes of the read buffer memory are 10% of the process memory limit, which is a conservative number compared to 20% of the total system memory. The read buffer size plus process memory limit should not exceed the total system memory. Two start options have been added for the new feature. 1. remote_batch_read. Default is false. If set true, the batch read is enabled. 2. remote_read_memory_buffer_size. Default is 1G. The maximum memory that can be used by the read buffer. The number is also restricted by the process memory limit, which can not exceed 10% of the process memory limit. Added metrics ScratchReadsUseMem/ScratchBytesReadUseMem/ ScratchBytesReadUseLocalDisk to the query profile. The patch also increases the MAX_REMOTE_TMPFILE_SIZE_THRESHOLD_MB from 256 to 512. Tests: Ran core and exhaustive tests. Added and ran TmpFileMgrTest::TestBatchReadingFromRemote. Added e2e test test_scratch_dirs_batch_reading. Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b --- M be/src/runtime/io/CMakeLists.txt A be/src/runtime/io/disk-file-test.cc M be/src/runtime/io/disk-file.cc M be/src/runtime/io/disk-file.h M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/request-context.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/mem-info.cc M be/src/util/mem-info.h M be/src/util/metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_scratch_disk.py 19 files changed, 1,587 insertions(+), 161 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/17979/18 -- To view, visit http://gerrit.cloudera.org:8080/17979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Gerrit-Change-Number: 17979 Gerrit-PatchSet: 18 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17979 ) Change subject: IMPALA-10791 Add batch reading for remote temporary files .. Patch Set 18: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11214/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Gerrit-Change-Number: 17979 Gerrit-PatchSet: 18 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Aug 2022 23:15:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/17979 ) Change subject: IMPALA-10791 Add batch reading for remote temporary files .. Patch Set 18: (3 comments) http://gerrit.cloudera.org:8080/#/c/17979/15/be/src/runtime/tmp-file-mgr.cc File be/src/runtime/tmp-file-mgr.cc: http://gerrit.cloudera.org:8080/#/c/17979/15/be/src/runtime/tmp-file-mgr.cc@604 PS15, Line 604: > I believe this is the memory limit for query processing, which is what I wo Thanks for pointing this out. Made a change on the max allowed bytes for read buffer. Now is 10% of the process bytes limit. Also modified the commit message. http://gerrit.cloudera.org:8080/#/c/17979/15/common/thrift/metrics.json File common/thrift/metrics.json: http://gerrit.cloudera.org:8080/#/c/17979/15/common/thrift/metrics.json@2424 PS15, Line 2424: The current total read memory buffer bytes for all scr > Maybe re-word this as follows to reflect this is current usage? Done http://gerrit.cloudera.org:8080/#/c/17979/15/common/thrift/metrics.json@2434 PS15, Line 2434: The high water mark for read memory buffer bytes of all scratch direct > Better wording to include the unit/bytes: Done -- To view, visit http://gerrit.cloudera.org:8080/17979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Gerrit-Change-Number: 17979 Gerrit-PatchSet: 18 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Aug 2022 23:19:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17979 ) Change subject: IMPALA-10791 Add batch reading for remote temporary files .. Patch Set 17: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8491/ -- To view, visit http://gerrit.cloudera.org:8080/17979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Gerrit-Change-Number: 17979 Gerrit-PatchSet: 17 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 24 Aug 2022 00:43:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11523: Fix test http socket timeout in Docker-based tests
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18899 Change subject: IMPALA-11523: Fix test_http_socket_timeout in Docker-based tests .. IMPALA-11523: Fix test_http_socket_timeout in Docker-based tests When running in the Docker-based tests, TestImpalaShell's test_http_socket_timeout fails with a mismatch in the error message. The test expected "Operation now in progress", but in Docker-based tests it throws "Cannot assign requested address". Since this is testing that a socket timeout of zero gets an error, it seems reasonable to tolerate this extra variant. This modifies the test to allow this error message. Testing: - TestImpalaShell.test_http_socket_timeout passes in the docker-based tests and in a normal core job Change-Id: If463f1100db673bb916b094c1402f1876342c80e --- M tests/shell/test_shell_commandline.py 1 file changed, 16 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/18899/1 -- To view, visit http://gerrit.cloudera.org:8080/18899 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If463f1100db673bb916b094c1402f1876342c80e Gerrit-Change-Number: 18899 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-11523: Fix test http socket timeout in Docker-based tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18899 ) Change subject: IMPALA-11523: Fix test_http_socket_timeout in Docker-based tests .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11215/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18899 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If463f1100db673bb916b094c1402f1876342c80e Gerrit-Change-Number: 18899 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Aug 2022 01:13:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11523: Fix test http socket timeout in Docker-based tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18899 ) Change subject: IMPALA-11523: Fix test_http_socket_timeout in Docker-based tests .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8492/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18899 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If463f1100db673bb916b094c1402f1876342c80e Gerrit-Change-Number: 18899 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Aug 2022 01:28:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11523: Fix test http socket timeout in Docker-based tests
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18899 ) Change subject: IMPALA-11523: Fix test_http_socket_timeout in Docker-based tests .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18899 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If463f1100db673bb916b094c1402f1876342c80e Gerrit-Change-Number: 18899 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 24 Aug 2022 02:21:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11454: Update toolchain to get smaller Kudu binaries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18901 ) Change subject: IMPALA-11454: Update toolchain to get smaller Kudu binaries .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8493/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id16a64668e9bd8ca046c1a5f1b70eb8a97186fcb Gerrit-Change-Number: 18901 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 24 Aug 2022 05:37:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11454: Update toolchain to get smaller Kudu binaries
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18901 Change subject: IMPALA-11454: Update toolchain to get smaller Kudu binaries .. IMPALA-11454: Update toolchain to get smaller Kudu binaries This updates the toolchain to use a native-toolchain with the fix for IMPALA-11454, which reduces the size of the Kudu tarball by symlinking the main Kudu binaries. Testing: - Ran core job Change-Id: Id16a64668e9bd8ca046c1a5f1b70eb8a97186fcb --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/18901/1 -- To view, visit http://gerrit.cloudera.org:8080/18901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id16a64668e9bd8ca046c1a5f1b70eb8a97186fcb Gerrit-Change-Number: 18901 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[native-toolchain-CR] IMPALA-11454: part-2: create kudu binaries symlinks to reduce package size.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18880 ) Change subject: IMPALA-11454: part-2: create kudu binaries symlinks to reduce package size. .. Patch Set 1: Code-Review+2 The toolchain build succeeded, and I put together this Impala side change with the new toolchain build (207-21f1a0057a). That change passed our core tests. So, this is good to go. When the Impala patch is ready to go, we'll merge them both at the same time. -- To view, visit http://gerrit.cloudera.org:8080/18880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc1904be06f84acebf056a079dc1b6694032de87 Gerrit-Change-Number: 18880 Gerrit-PatchSet: 1 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Wed, 24 Aug 2022 05:37:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11454: Update toolchain to get smaller Kudu binaries
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18901 ) Change subject: IMPALA-11454: Update toolchain to get smaller Kudu binaries .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id16a64668e9bd8ca046c1a5f1b70eb8a97186fcb Gerrit-Change-Number: 18901 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 24 Aug 2022 05:40:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11454: Update toolchain to get smaller Kudu binaries
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18901 ) Change subject: IMPALA-11454: Update toolchain to get smaller Kudu binaries .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id16a64668e9bd8ca046c1a5f1b70eb8a97186fcb Gerrit-Change-Number: 18901 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 24 Aug 2022 05:42:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11454: Update toolchain to get smaller Kudu binaries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18901 ) Change subject: IMPALA-11454: Update toolchain to get smaller Kudu binaries .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11216/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id16a64668e9bd8ca046c1a5f1b70eb8a97186fcb Gerrit-Change-Number: 18901 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 24 Aug 2022 05:56:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11523: Fix test http socket timeout in Docker-based tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18899 ) Change subject: IMPALA-11523: Fix test_http_socket_timeout in Docker-based tests .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18899 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If463f1100db673bb916b094c1402f1876342c80e Gerrit-Change-Number: 18899 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 24 Aug 2022 06:12:31 + Gerrit-HasComments: No
