[Impala-ASF-CR] IMPALA-12763: Union with string struct crashes in ASAN
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20970 ) Change subject: IMPALA-12763: Union with string struct crashes in ASAN .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4 Gerrit-Change-Number: 20970 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 30 Jan 2024 16:40:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12763: Union with string struct crashes in ASAN
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/20970 )
Change subject: IMPALA-12763: Union with string struct crashes in ASAN
..
IMPALA-12763: Union with string struct crashes in ASAN
In ASAN builds, if we UNION ALL an array containing a struct of a string
with itself, Impala crashes. This is how to reproduce it:
In Hive:
create table su (arr ARRAY>) stored as parquet;
insert into su values (array(named_struct("s", "A")));
In Impala:
select 1, arr from su
union all select 2, arr from su;
The ASAN error message indicates a heap-use-after-free.
Normally, UNIONs of structs are not supported yet (see IMPALA-10752),
but if the struct is inside an array it is allowed now. This was
probably not intentional and it leads to the above error, so this change
disables structs in unions completely, including embedded structs.
Testing:
- adjusted existing tests
- added a query that tests that types with embedded structs are not
allowed in a UNION statement, in mixed-collections-and-structs.test
Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4
Reviewed-on: http://gerrit.cloudera.org:8080/20970
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M
testdata/workloads/functional-query/queries/QueryTest/mixed-collections-and-structs.test
M
testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
M
testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
M
testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test
7 files changed, 44 insertions(+), 8 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/20970
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4
Gerrit-Change-Number: 20970
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Noemi Pap-Takacs
[Impala-ASF-CR] IMPALA-12763: Union with string struct crashes in ASAN
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20970 ) Change subject: IMPALA-12763: Union with string struct crashes in ASAN .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4 Gerrit-Change-Number: 20970 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 30 Jan 2024 12:09:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12763: Union with string struct crashes in ASAN
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20970 ) Change subject: IMPALA-12763: Union with string struct crashes in ASAN .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10214/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4 Gerrit-Change-Number: 20970 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 30 Jan 2024 12:09:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12763: Union with string struct crashes in ASAN
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20970 ) Change subject: IMPALA-12763: Union with string struct crashes in ASAN .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15101/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4 Gerrit-Change-Number: 20970 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 30 Jan 2024 11:09:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12763: Union with string struct crashes in ASAN
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20970 ) Change subject: IMPALA-12763: Union with string struct crashes in ASAN .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4 Gerrit-Change-Number: 20970 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 30 Jan 2024 10:49:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12763: Union with string struct crashes in ASAN
Daniel Becker has uploaded a new patch set (#2). (
http://gerrit.cloudera.org:8080/20970 )
Change subject: IMPALA-12763: Union with string struct crashes in ASAN
..
IMPALA-12763: Union with string struct crashes in ASAN
In ASAN builds, if we UNION ALL an array containing a struct of a string
with itself, Impala crashes. This is how to reproduce it:
In Hive:
create table su (arr ARRAY>) stored as parquet;
insert into su values (array(named_struct("s", "A")));
In Impala:
select 1, arr from su
union all select 2, arr from su;
The ASAN error message indicates a heap-use-after-free.
Normally, UNIONs of structs are not supported yet (see IMPALA-10752),
but if the struct is inside an array it is allowed now. This was
probably not intentional and it leads to the above error, so this change
disables structs in unions completely, including embedded structs.
Testing:
- adjusted existing tests
- added a query that tests that types with embedded structs are not
allowed in a UNION statement, in mixed-collections-and-structs.test
Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M
testdata/workloads/functional-query/queries/QueryTest/mixed-collections-and-structs.test
M
testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
M
testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
M
testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test
7 files changed, 44 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/20970/2
--
To view, visit http://gerrit.cloudera.org:8080/20970
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4
Gerrit-Change-Number: 20970
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Noemi Pap-Takacs
[Impala-ASF-CR] IMPALA-12763: Union with string struct crashes in ASAN
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/20970 )
Change subject: IMPALA-12763: Union with string struct crashes in ASAN
..
Patch Set 1: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20970/1/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/20970/1/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3468
PS1, Line 3468: private static boolean typeHasStruct(Type type) {
I think that Type.java would be a better place for this. There is already a
containsCollection function:
https://github.com/apache/impala/blob/eaa35b02507a834edd0d219343fd4bd075f21762/fe/src/main/java/org/apache/impala/catalog/Type.java#L251C18-L251C36
--
To view, visit http://gerrit.cloudera.org:8080/20970
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4
Gerrit-Change-Number: 20970
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Noemi Pap-Takacs
Gerrit-Comment-Date: Tue, 30 Jan 2024 09:11:00 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12763: Union with string struct crashes in ASAN
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20970 ) Change subject: IMPALA-12763: Union with string struct crashes in ASAN .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15086/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4 Gerrit-Change-Number: 20970 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 29 Jan 2024 15:50:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12763: Union with string struct crashes in ASAN
Daniel Becker has uploaded this change for review. (
http://gerrit.cloudera.org:8080/20970
Change subject: IMPALA-12763: Union with string struct crashes in ASAN
..
IMPALA-12763: Union with string struct crashes in ASAN
In ASAN builds, if we UNION ALL an array containing a struct of a string
with itself, Impala crashes. This is how to reproduce it:
In Hive:
create table su (arr ARRAY>) stored as parquet;
insert into su values (array(named_struct("s", "A")));
In Impala:
select 1, arr from su
union all select 2, arr from su;
The ASAN error message indicates a heap-use-after-free.
Normally, UNIONs of structs are not supported yet (see IMPALA-10752),
but if the struct is inside an array it is allowed now. This was
probably not intentional and it leads to the above error, so this change
disables structs in unions completely, including embedded structs.
Testing:
- adjusted existing tests
- added a query that tests that types with embedded structs are not
allowed in a UNION statement, in mixed-collections-and-structs.test
Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M
testdata/workloads/functional-query/queries/QueryTest/mixed-collections-and-structs.test
M
testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
M
testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
M
testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test
6 files changed, 38 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/20970/1
--
To view, visit http://gerrit.cloudera.org:8080/20970
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id728f1254b74636be594a33313a478b0b77c7ae4
Gerrit-Change-Number: 20970
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Becker
