[Impala-ASF-CR] IMPALA-11230: Add test for crash in partitioned Top-N codegen code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18389 ) Change subject: IMPALA-11230: Add test for crash in partitioned Top-N codegen code .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6909ef660b01ad3d301273deb8a8c31120445f79 Gerrit-Change-Number: 18389 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Apr 2022 05:39:30 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Add 3.4.1 change log
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18345 ) Change subject: Add 3.4.1 change log .. Patch Set 2: Verified+1 > Patch Set 2: > > Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/653/ > > Testing docs change - this change appears to modify docs/ and no code. This > is experimental - please report any issues to tarmstr...@cloudera.com or on > this JIRA: IMPALA-7317 The job only verifies normal doc patch. Removing the -1. -- To view, visit http://gerrit.cloudera.org:8080/18345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I7d91d97fed8ea496ff2a24f51b7328c58214292e Gerrit-Change-Number: 18345 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 07 Apr 2022 04:49:07 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Add 3.4.1 change log
Quanlong Huang has removed a vote on this change. Change subject: Add 3.4.1 change log .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/18345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: deleteVote Gerrit-Change-Id: I7d91d97fed8ea496ff2a24f51b7328c58214292e Gerrit-Change-Number: 18345 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR](asf-site) Add 3.4.1 change log
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18345 ) Change subject: Add 3.4.1 change log .. Patch Set 2: Verified-1 Build Failed https://jenkins.impala.io/job/gerrit-docs-auto-test/653/ : Doc tests failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I7d91d97fed8ea496ff2a24f51b7328c58214292e Gerrit-Change-Number: 18345 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 07 Apr 2022 04:46:01 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Add 3.4.1 change log
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18345 ) Change subject: Add 3.4.1 change log .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/653/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/18345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I7d91d97fed8ea496ff2a24f51b7328c58214292e Gerrit-Change-Number: 18345 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 07 Apr 2022 04:41:36 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Add 3.4.1 change log
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18345 to look at the new patch set (#2). Change subject: Add 3.4.1 change log .. Add 3.4.1 change log Previously we didn't mention the patch version, e.g. the last number of "3.4.1". This changes the web page to distinguish 3.4.1 vs 3.4.0. Also change 4.0 to 4.0.0. Added change log of 3.4.1 and mentioning that IMPALA-9242 is reverted. Change-Id: I7d91d97fed8ea496ff2a24f51b7328c58214292e --- A docs/changelog-3.4.1.html M impala-docs.html 2 files changed, 80 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/18345/2 -- To view, visit http://gerrit.cloudera.org:8080/18345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7d91d97fed8ea496ff2a24f51b7328c58214292e Gerrit-Change-Number: 18345 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10930: Bump the Java artifact versions to 4.1.0-SNAPSHOT
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18390 ) Change subject: IMPALA-10930: Bump the Java artifact versions to 4.1.0-SNAPSHOT .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10410/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18390 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10a589b4fbc15048199943a0e06d079f57840239 Gerrit-Change-Number: 18390 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Apr 2022 03:17:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10930: Bump the Java artifact versions to 4.1.0-SNAPSHOT
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18390 Change subject: IMPALA-10930: Bump the Java artifact versions to 4.1.0-SNAPSHOT .. IMPALA-10930: Bump the Java artifact versions to 4.1.0-SNAPSHOT This changes the Maven pom.xml files to use verison 4.1.0-SNAPSHOT rather than 4.0.0-SNAPSHOT. In the past, these versions were a fixed value, but that changed with IMPALA-10198. This is a new step that needs to happen on each release. Testing: - Ran a build Change-Id: I10a589b4fbc15048199943a0e06d079f57840239 --- M fe/pom.xml M java/TableFlattener/pom.xml M java/datagenerator/pom.xml M java/executor-deps/pom.xml M java/ext-data-source/api/pom.xml M java/ext-data-source/pom.xml M java/ext-data-source/sample/pom.xml M java/ext-data-source/test/pom.xml M java/pom.xml M java/query-event-hook-api/pom.xml M java/shaded-deps/hive-exec/pom.xml M java/shaded-deps/s3a-aws-sdk/pom.xml M java/test-hive-udfs/pom.xml M java/yarn-extras/pom.xml 14 files changed, 14 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/18390/1 -- To view, visit http://gerrit.cloudera.org:8080/18390 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I10a589b4fbc15048199943a0e06d079f57840239 Gerrit-Change-Number: 18390 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18385 ) Change subject: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile .. Patch Set 1: Code-Review+1 Thanks for fixing this quickly! Can we add a test using the query in the JIRA description? -- To view, visit http://gerrit.cloudera.org:8080/18385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f Gerrit-Change-Number: 18385 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 07 Apr 2022 02:01:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11227: FE OOM in TestParquetBloomFilter.test fallback from dict if no bloom tbl props
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/18387 )
Change subject: IMPALA-11227: FE OOM in
TestParquetBloomFilter.test_fallback_from_dict_if_no_bloom_tbl_props
..
Patch Set 1: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18387/1/tests/query_test/test_parquet_bloom_filter.py
File tests/query_test/test_parquet_bloom_filter.py:
http://gerrit.cloudera.org:8080/#/c/18387/1/tests/query_test/test_parquet_bloom_filter.py@214
PS1, Line 214: bl_create_stmt = \
: 'create table {db}.{tbl} ({col_name} BIGINT) stored as
textfile' .forma
> optional: I think that this could be made simpler by using an existing tabl
Ah, this is a cleaner approach! Maybe need an ORDER BY clause as well.
--
To view, visit http://gerrit.cloudera.org:8080/18387
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I923cc9ba4b6829a2f15e93365f2849b89248598b
Gerrit-Change-Number: 18387
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Thu, 07 Apr 2022 01:58:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11230: Add test for crash in partitioned Top-N codegen code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18389 ) Change subject: IMPALA-11230: Add test for crash in partitioned Top-N codegen code .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8029/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6909ef660b01ad3d301273deb8a8c31120445f79 Gerrit-Change-Number: 18389 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Apr 2022 01:13:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11230: Add test for crash in partitioned Top-N codegen code
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18389 ) Change subject: IMPALA-11230: Add test for crash in partitioned Top-N codegen code .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6909ef660b01ad3d301273deb8a8c31120445f79 Gerrit-Change-Number: 18389 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 06 Apr 2022 23:46:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/18385 ) Change subject: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile .. Patch Set 1: Code-Review+1 (1 comment) > Hi Riza! Capping messages to some sane limit makes sense, though I am not > sure that it should go to the same commit to make this easy to backport. Thank you, Csaba. That make sense. Let's track that in a separate patch. http://gerrit.cloudera.org:8080/#/c/18385/1/be/src/exec/exec-node.cc File be/src/exec/exec-node.cc: http://gerrit.cloudera.org:8080/#/c/18385/1/be/src/exec/exec-node.cc@331 PS1, Line 331: codegen_status_msgs_ We might want to cap the size of this array too next time, to avoid blowing up ExecOption string. -- To view, visit http://gerrit.cloudera.org:8080/18385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f Gerrit-Change-Number: 18385 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 06 Apr 2022 23:14:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11230: Add test for crash in partitioned Top-N codegen code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18389 ) Change subject: IMPALA-11230: Add test for crash in partitioned Top-N codegen code .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10409/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6909ef660b01ad3d301273deb8a8c31120445f79 Gerrit-Change-Number: 18389 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 06 Apr 2022 22:05:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11230: Add test for crash in partitioned Top-N codegen code
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18389 Change subject: IMPALA-11230: Add test for crash in partitioned Top-N codegen code .. IMPALA-11230: Add test for crash in partitioned Top-N codegen code User workloads hit a crash for certain queries that use partitioned Top-N operators. The crash occurred only when codegen is enabled. After investigation, the crash was due to a nullptr being passed into TupleRowComparator::Compare(). The issue was fixed as part of IMPALA-10961. This adds a test case with a SQL statement that triggers a crash if IMPALA-10961 is not present. Change-Id: I6909ef660b01ad3d301273deb8a8c31120445f79 --- M testdata/workloads/functional-query/queries/QueryTest/analytic-fns-tpcds-partitioned-topn.test 1 file changed, 14 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18389/1 -- To view, visit http://gerrit.cloudera.org:8080/18389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6909ef660b01ad3d301273deb8a8c31120445f79 Gerrit-Change-Number: 18389 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[native-toolchain-CR](CDH-7.1.8.x) IMPALA-11195: Disable SSL session renegotiation
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18382 ) Change subject: IMPALA-11195: Disable SSL session renegotiation .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: CDH-7.1.8.x Gerrit-MessageType: comment Gerrit-Change-Id: If3e12b6394134daf9c936268d4e60da6b4f3804e Gerrit-Change-Number: 18382 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal (Cloudera) Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 06 Apr 2022 21:16:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18385 ) Change subject: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10408/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f Gerrit-Change-Number: 18385 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 06 Apr 2022 19:38:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11227: FE OOM in TestParquetBloomFilter.test fallback from dict if no bloom tbl props
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/18387 )
Change subject: IMPALA-11227: FE OOM in
TestParquetBloomFilter.test_fallback_from_dict_if_no_bloom_tbl_props
..
Patch Set 1: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18387/1/tests/query_test/test_parquet_bloom_filter.py
File tests/query_test/test_parquet_bloom_filter.py:
http://gerrit.cloudera.org:8080/#/c/18387/1/tests/query_test/test_parquet_bloom_filter.py@214
PS1, Line 214: bl_create_stmt = \
: 'create table {db}.{tbl} ({col_name} BIGINT) stored as
textfile' .forma
optional: I think that this could be made simpler by using an existing table as
source, e.g.:
select o_orderkey * 2 from tpch_parquet.orders limit 40001;
(all values of o_orderkey are unique)
--
To view, visit http://gerrit.cloudera.org:8080/18387
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I923cc9ba4b6829a2f15e93365f2849b89248598b
Gerrit-Change-Number: 18387
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Wed, 06 Apr 2022 19:32:52 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18385 ) Change subject: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile .. Patch Set 1: > Hi Csaba, thanks for picking this up! > We hit similar issue in the past at IMPALA-9851, where the fix is > to truncate the message to 128kb. > Should we defensively add similar truncation/check in > RuntimeProfileBase::AddInfoStringInternal? Hi Riza! Capping messages to some sane limit makes sense, though I am not sure that it should go to the same commit to make this easy to backport. -- To view, visit http://gerrit.cloudera.org:8080/18385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f Gerrit-Change-Number: 18385 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 06 Apr 2022 19:20:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 ) Change subject: IMPALA-10723: Treat materialized view as a table instead of a view .. Patch Set 13: Code-Review+1 (1 comment) Looks good! http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test File testdata/workloads/functional-query/queries/QueryTest/compute-stats.test: http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test@1841 PS7, Line 1841: G > I will explore the syntax extension for the show stats command in a follow- Sounds like a plan. -- To view, visit http://gerrit.cloudera.org:8080/17595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff Gerrit-Change-Number: 17595 Gerrit-PatchSet: 13 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 18:13:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18289 ) Change subject: IMPALA-11153: Make lock wait time configurable for the users .. IMPALA-11153: Make lock wait time configurable for the users Currently Impala is using private static final values for lock retry/wait: // Number of retries to acquire an HMS ACID lock. private static final int LOCK_RETRIES = 10; // Time interval between retries of acquiring an HMS ACID lock private static final int LOCK_RETRY_WAIT_SECONDS = 3; This patch changes the logic for waiting on locks. It does an exponential backoff starting from 50 ms up to 30 seconds, and the users can configure a maximum total wait time for the locks. This total maximum wait time is 5 minutes (the earlier 30 seconds was too short in real use cases). Testing: * added e2e tests Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc Reviewed-on: http://gerrit.cloudera.org:8080/18289 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/catalog-op-executor.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CatalogService.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M tests/query_test/test_acid.py 13 files changed, 182 insertions(+), 58 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc Gerrit-Change-Number: 18289 Gerrit-PatchSet: 15 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 ) Change subject: IMPALA-11153: Make lock wait time configurable for the users .. Patch Set 14: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc Gerrit-Change-Number: 18289 Gerrit-PatchSet: 14 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 06 Apr 2022 16:46:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10838: Error when struct returned from WITH()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17847 ) Change subject: IMPALA-10838: Error when struct returned from WITH() .. Patch Set 20: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10407/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57 Gerrit-Change-Number: 17847 Gerrit-PatchSet: 20 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Wed, 06 Apr 2022 15:44:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10838: Error when struct returned from WITH()
Daniel Becker has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/17847 ) Change subject: IMPALA-10838: Error when struct returned from WITH() .. IMPALA-10838: Error when struct returned from WITH() The following query fails: ''' with sub as ( select id, outer_struct from functional_orc_def.complextypes_nested_structs) select sub.id, sub.outer_struct.inner_struct2 from sub; ''' with the following error: ''' ERROR: IllegalStateException: Illegal reference to non-materialized tuple: debugname=InlineViewRef sub alias=sub tid=6 ''' while if 'outer_struct.inner_struct2' is added to the select list of the inline view, the query works as expected. This change fixes the problem by two modifications: - if a field of a struct needs to be materialised, also materialise all of its enclosing structs (ancestors) - in InlineViewRef, struct fields are inserted into the 'smap' and 'baseTableSmap' with the appropriate inline view prefix TODO: Is this just a hack with the labels? This change also changes the way struct fields are materialised: until now, if a member of a struct was needed to be materialised, the whole struct, including other members of the struct were materialised. This behaviour can lead to using significantly more memory than necessary if we for example query a single member of a large struct. This change modifies this behaviour so that we only materialise the struct members that are actually needed. Tests: - added queries that are fixed by this change (including the one above) in nested-struct-in-select-list.test - added a planner test in fe/src/test/java/org/apache/impala/planner/PlannerTest.java that asserts that only the required parts of structs are materialised Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java M testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test 16 files changed, 813 insertions(+), 97 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/17847/20 -- To view, visit http://gerrit.cloudera.org:8080/17847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57 Gerrit-Change-Number: 17847 Gerrit-PatchSet: 20 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen
[Impala-ASF-CR] IMPALA-11227: FE OOM in TestParquetBloomFilter.test fallback from dict if no bloom tbl props
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18387 ) Change subject: IMPALA-11227: FE OOM in TestParquetBloomFilter.test_fallback_from_dict_if_no_bloom_tbl_props .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10406/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18387 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I923cc9ba4b6829a2f15e93365f2849b89248598b Gerrit-Change-Number: 18387 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 14:50:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11227: FE OOM in TestParquetBloomFilter.test fallback from dict if no bloom tbl props
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18387 ) Change subject: IMPALA-11227: FE OOM in TestParquetBloomFilter.test_fallback_from_dict_if_no_bloom_tbl_props .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/18387/1/tests/query_test/test_parquet_bloom_filter.py File tests/query_test/test_parquet_bloom_filter.py: http://gerrit.cloudera.org:8080/#/c/18387/1/tests/query_test/test_parquet_bloom_filter.py@216 PS1, Line 216: d flake8: E122 continuation line missing indentation or outdented http://gerrit.cloudera.org:8080/#/c/18387/1/tests/query_test/test_parquet_bloom_filter.py@236 PS1, Line 236: c flake8: E122 continuation line missing indentation or outdented -- To view, visit http://gerrit.cloudera.org:8080/18387 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I923cc9ba4b6829a2f15e93365f2849b89248598b Gerrit-Change-Number: 18387 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 14:31:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11227: FE OOM in TestParquetBloomFilter.test fallback from dict if no bloom tbl props
Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18387 Change subject: IMPALA-11227: FE OOM in TestParquetBloomFilter.test_fallback_from_dict_if_no_bloom_tbl_props .. IMPALA-11227: FE OOM in TestParquetBloomFilter.test_fallback_from_dict_if_no_bloom_tbl_props The huge values clause of the insert SQL statement in TestParquetBloomFilter.test_fallback_from_dict_if_no_bloom_tbl_props could cause an OutOfMemory error in the FE. We use a SQL statement with a huge values clause (more than 40 000 elements) to insert values into a parquet table in some tests, and the size of the SQL statement string sometimes causes an OOM error. After this change, when creating these parquet tables, instead of using a huge SQL statement to insert the values, we first create a text table with the same schema, write the values to a text file, copy the file into the HDFS directory of the text table and create a parquet table from the text table. Change-Id: I923cc9ba4b6829a2f15e93365f2849b89248598b --- M tests/query_test/test_parquet_bloom_filter.py 1 file changed, 30 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/18387/1 -- To view, visit http://gerrit.cloudera.org:8080/18387 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I923cc9ba4b6829a2f15e93365f2849b89248598b Gerrit-Change-Number: 18387 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8027/ -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 06 Apr 2022 13:41:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Tamas Mate has posted comments on this change. (
http://gerrit.cloudera.org:8080/18383 )
Change subject: IMPALA-11023: Raise error when delete file is found in an
Iceberg table
..
Patch Set 2:
(1 comment)
Thank you for the review Zoltan, updated the CR.
http://gerrit.cloudera.org:8080/#/c/18383/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/18383/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@566
PS2, Line 566: throw new TableLoadingException("Data file list collection
failed.");
> I think it would be useful to include the original exception as well, i.e.
agree, done
--
To view, visit http://gerrit.cloudera.org:8080/18383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e
Gerrit-Change-Number: 18383
Gerrit-PatchSet: 2
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 06 Apr 2022 12:57:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/18383 )
Change subject: IMPALA-11023: Raise error when delete file is found in an
Iceberg table
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18383/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/18383/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@566
PS2, Line 566: throw new TableLoadingException("Data file list collection
failed.");
I think it would be useful to include the original exception as well, i.e. use:
TableLoadingException(String s, Throwable cause)
--
To view, visit http://gerrit.cloudera.org:8080/18383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e
Gerrit-Change-Number: 18383
Gerrit-PatchSet: 2
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 06 Apr 2022 12:45:39 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 ) Change subject: IMPALA-11153: Make lock wait time configurable for the users .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8028/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc Gerrit-Change-Number: 18289 Gerrit-PatchSet: 14 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 06 Apr 2022 12:20:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 ) Change subject: IMPALA-11153: Make lock wait time configurable for the users .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc Gerrit-Change-Number: 18289 Gerrit-PatchSet: 14 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 06 Apr 2022 12:20:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 ) Change subject: IMPALA-11153: Make lock wait time configurable for the users .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc Gerrit-Change-Number: 18289 Gerrit-PatchSet: 13 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 06 Apr 2022 12:11:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8027/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 06 Apr 2022 12:00:59 + Gerrit-HasComments: No
[native-toolchain-CR](CDH-7.1.8.x) IMPALA-11195: Disable SSL session renegotiation
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18382 ) Change subject: IMPALA-11195: Disable SSL session renegotiation .. Patch Set 2: Verified+1 The internal build job has succeeded with this fix. -- To view, visit http://gerrit.cloudera.org:8080/18382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: CDH-7.1.8.x Gerrit-MessageType: comment Gerrit-Change-Id: If3e12b6394134daf9c936268d4e60da6b4f3804e Gerrit-Change-Number: 18382 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal (Cloudera) Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 06 Apr 2022 12:00:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 ) Change subject: IMPALA-10723: Treat materialized view as a table instead of a view .. Patch Set 13: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8026/ -- To view, visit http://gerrit.cloudera.org:8080/17595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff Gerrit-Change-Number: 17595 Gerrit-PatchSet: 13 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 11:08:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18373 ) Change subject: IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies .. IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies When Ranger authorization is enabled, analyzing a LOAD DATA statement may fail when checking access to the source file if the HDFS permissions is more restrict than the Ranger policies. This patch provides a quick fix, when Ranger authz is enabled, we invoke method hadoop.fs.FileSystem.access(Path path, FsAction mode) to check the actual access permission first. Testing: - existing tests - add FsPermissionCheckerTest.java to fe ut Change-Id: Ia0acd83e8e237ea6a1d5fe1e9e440db0dd68b09c Reviewed-on: http://gerrit.cloudera.org:8080/18373 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/util/FsPermissionChecker.java A fe/src/test/java/org/apache/impala/util/FsPermissionCheckerTest.java 3 files changed, 121 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia0acd83e8e237ea6a1d5fe1e9e440db0dd68b09c Gerrit-Change-Number: 18373 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18373 ) Change subject: IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0acd83e8e237ea6a1d5fe1e9e440db0dd68b09c Gerrit-Change-Number: 18373 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 10:42:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10405/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 06 Apr 2022 09:23:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Tamas Mate has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. IMPALA-11023: Raise error when delete file is found in an Iceberg table Iceberg V2 DeleteFiles are skipped during scans and the whole content of the DataFiles are returned. This commit adds an extra check to prevent scanning tables that have delete files to avoid unexpected results till merge on read is supported. Testing: - Added e2e test. Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e --- M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/0-0-fb178c51-b12a-4c5f-a66e-a8e9375daeba-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/8cbef400-daea-478a-858a-2baf2438f644-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/v2.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test 14 files changed, 199 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/18383/2 -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 ) Change subject: IMPALA-10723: Treat materialized view as a table instead of a view .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10404/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff Gerrit-Change-Number: 17595 Gerrit-PatchSet: 13 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 06:45:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 ) Change subject: IMPALA-10723: Treat materialized view as a table instead of a view .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8026/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff Gerrit-Change-Number: 17595 Gerrit-PatchSet: 13 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 06:44:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 ) Change subject: IMPALA-10723: Treat materialized view as a table instead of a view .. Patch Set 13: > Patch Set 11: > > (1 comment) Added the DEPENDENT_HIVE_LOAD section in the load script for the materialized views. Will test by doing a dry run on jenkins. -- To view, visit http://gerrit.cloudera.org:8080/17595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff Gerrit-Change-Number: 17595 Gerrit-PatchSet: 13 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 06:43:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view
Aman Sinha has posted comments on this change. (
http://gerrit.cloudera.org:8080/17595 )
Change subject: IMPALA-10723: Treat materialized view as a table instead of a
view
..
Patch Set 13:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@225
PS11, Line 225: vAuthExceptionM
> It seems the suffix is not added yet. BTW, I think we can remove the boolea
PS 12 was showing the suffix. In any case, in PS 13, I removed the boolean and
am using the null check as suggested.
http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@931
PS11, Line 931: resolvedTableRef.getTable() instanceof
MaterializedViewHdfsTable &&
> If the MV has masking policies, we won't go to the below check. Should we m
Good point. I moved the entire check for source table masking up into an
else-if block before going into the check for
tableMask.needsMaskingOrFiltering().
http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
File fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@322
PS11, Line 322: ((MaterializedViewHdfsTable)
tbl).addSrcTables(mvSrcTableNames);
> Can we define MV on some real views? In this case, 'mvSrcTableNames' could
MVs are not allowed to be defined on regular views since MVs expect the source
to be transactional entities and views are not in that category. So, in Hive,
the following fails (ran through beeline):
0: jdbc:hive2://localhost:11050/default> create view vv1 as select * from
tpch.nation;
0: jdbc:hive2://localhost:11050/default> create materialized view mvv1 as
select n_nationkey, count(*) from vv1 group by n_nationkey;
Error: Error while compiling statement: FAILED: SemanticException Automatic
rewriting for materialized view cannot be enabled if the materialized view uses
non-transactional tables (state=42000,code=4)
http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java@84
PS11, Line 84: }
> nit: need a blank line after this
Done
http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File
fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:
http://gerrit.cloudera.org:8080/#/c/17595/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3171
PS11, Line 3171: onTable("functional", "mv1_alltypes_jointbl",
TPrivilegeLevel.SELECT));
> Can we add an additional test that also add a masking policies on the MV? I
Yup, added a column masking policy on mv1_alltypes_jointbl and tested that the
same authorization exception is thrown (this after the other change to
Analyzer#resolveTableMask()).
http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
File testdata/workloads/functional-query/queries/QueryTest/compute-stats.test:
http://gerrit.cloudera.org:8080/#/c/17595/7/testdata/workloads/functional-query/queries/QueryTest/compute-stats.test@1841
PS7, Line 1841: G
> Thanks for the explanation.
I will explore the syntax extension for the show stats command in a follow-up
jira and we can discuss what would be the appropriate syntax.
--
To view, visit http://gerrit.cloudera.org:8080/17595
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff
Gerrit-Change-Number: 17595
Gerrit-PatchSet: 13
Gerrit-Owner: Aman Sinha
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Wed, 06 Apr 2022 06:40:41 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view
Hello Quanlong Huang, Qifan Chen, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17595 to look at the new patch set (#13). Change subject: IMPALA-10723: Treat materialized view as a table instead of a view .. IMPALA-10723: Treat materialized view as a table instead of a view The existing behavior is that materialized views are treated as views and therefore expanded similar to a view when one queries the MV directly (SELECT * FROM materialized_view). This is incorrect since an MV is a regular table with physical properties such as partitioning, clustering etc. and should be treated as such even though it has a view definition associated with it. This patch focuses on the use case where MVs are created as HDFS tables and makes the MVs a derived class of HdfsTable, therefore making it a Table object. It adds support for collecting and displaying statistics on materialized views and these statistics could be leveraged by an external frontend that supports MV based query rewrites (note that such a rewrite is not supported by Impala with or without this patch). Note that we are not introducing new syntax for MVs since DDL, DML operations on MVs are only supported through Hive. Directly querying a MV is permitted but inserts into MVs is not since MVs are supposed to be only modified through an external refresh when the source tables have modifications. If the source tables associated with a materialized view have column masking or row-filtering Ranger policies, querying the MV will throw an error. This behavior is consistent with that of Hive. Testing: - Added transactional tables for alltypes, jointbl and used them as source tables to create materialized view. - Added tests for compute stats, drop stats, show stats and simple select query on a materialized view. - Added test for select on a materialized view when the source table has a column mask. - Modified analyzer tests related to alter, insert, drop of materialized view. Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java A fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/service/FrontendTest.java M testdata/bin/compute-table-stats.sh M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/views.test M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test M testdata/workloads/functional-query/queries/QueryTest/views.test 17 files changed, 356 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17595/13 -- To view, visit http://gerrit.cloudera.org:8080/17595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff Gerrit-Change-Number: 17595 Gerrit-PatchSet: 13 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18373 ) Change subject: IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8025/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0acd83e8e237ea6a1d5fe1e9e440db0dd68b09c Gerrit-Change-Number: 18373 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 06:13:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18373 ) Change subject: IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0acd83e8e237ea6a1d5fe1e9e440db0dd68b09c Gerrit-Change-Number: 18373 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 06:13:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18373 ) Change subject: IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies .. Patch Set 4: Code-Review+2 Verified in a real cluster. Thanks for working on this! -- To view, visit http://gerrit.cloudera.org:8080/18373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0acd83e8e237ea6a1d5fe1e9e440db0dd68b09c Gerrit-Change-Number: 18373 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 06:13:03 + Gerrit-HasComments: No
