[Impala-ASF-CR] IMPALA-10598: Deflake test cache reload validation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17210 ) Change subject: IMPALA-10598: Deflake test_cache_reload_validation .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17210 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d491e90af461d5db3575a5840958d17ca90901c Gerrit-Change-Number: 17210 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Mar 2021 04:39:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10579: Fix usage of RemoteIterator in FileSystemUtil
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17171 ) Change subject: IMPALA-10579: Fix usage of RemoteIterator in FileSystemUtil .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/17171/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java: http://gerrit.cloudera.org:8080/#/c/17171/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@959 PS5, Line 959: curIter_ = subIter; > Should curIter_ be cleared first so it is not stale if there is an exceptio No, it's not stale. When newIterFunc_ throws an exception, it means we can't go into the subdirectory. We should still use curIter_ to get the next subdirectory or file. -- To view, visit http://gerrit.cloudera.org:8080/17171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I859bd4f976c51a34eb6a03cefd2ddcdf11656cea Gerrit-Change-Number: 17171 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 23 Mar 2021 01:18:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10598: Deflake test cache reload validation
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/17210 )
Change subject: IMPALA-10598: Deflake test_cache_reload_validation
..
Patch Set 1: Code-Review+2
(2 comments)
LGTM. Thanks for the fix!
http://gerrit.cloudera.org:8080/#/c/17210/1/tests/query_test/test_hdfs_caching.py
File tests/query_test/test_hdfs_caching.py:
http://gerrit.cloudera.org:8080/#/c/17210/1/tests/query_test/test_hdfs_caching.py@349
PS1, Line 349: line for line in stdout.split('\n') if
: line.strip(' \n') != ''
nit: I think this can be simplified as
[line for line in stdout.split('\n') if line.strip()]
http://gerrit.cloudera.org:8080/#/c/17210/1/tests/query_test/test_hdfs_caching.py@363
PS1, Line 363: else:
: count = len(stdout.split('\n'))
: return count
nit: return len(stdout.split('\n'))
--
To view, visit http://gerrit.cloudera.org:8080/17210
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d491e90af461d5db3575a5840958d17ca90901c
Gerrit-Change-Number: 17210
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Tue, 23 Mar 2021 01:06:17 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10564: Return error when inserting an invalid decimal value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17168 ) Change subject: IMPALA-10564: Return error when inserting an invalid decimal value .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8420/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64ce4ed194af81ef06401ffc1124e12f05b8da98 Gerrit-Change-Number: 17168 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Mar 2021 23:38:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10592: prevent pytest from hanging at exit.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17212 ) Change subject: IMPALA-10592: prevent pytest from hanging at exit. .. IMPALA-10592: prevent pytest from hanging at exit. In TestAdmissionControllerStress mark worker threads as daemons so that an exception in teardown() will not cause pytest to hang just after printing the test results. https://stackoverflow.com/questions/19219596/py-test-hangs-after-showing-test-results TESTING: Simulated the failure in IMPALA-10596 by throwing an exception during teardown(). Without this fix the pytest invocation hangs. Change-Id: I74cca8f577c7fbc4d394311e2f039cf4f68b08df Reviewed-on: http://gerrit.cloudera.org:8080/17212 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_admission_controller.py 1 file changed, 1 insertion(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17212 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I74cca8f577c7fbc4d394311e2f039cf4f68b08df Gerrit-Change-Number: 17212 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10592: prevent pytest from hanging at exit.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17212 ) Change subject: IMPALA-10592: prevent pytest from hanging at exit. .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17212 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74cca8f577c7fbc4d394311e2f039cf4f68b08df Gerrit-Change-Number: 17212 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 22 Mar 2021 23:33:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10564: Return error when inserting an invalid decimal value
Wenzhe Zhou has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/17168 ) Change subject: IMPALA-10564: Return error when inserting an invalid decimal value .. IMPALA-10564: Return error when inserting an invalid decimal value When using CTAS statements or INSERT-SELECT statements to insert rows to table with decimal columns, Impala insert NULL for overflowed decimal values, instead of returning error. This issue happens when the data expression for the decimal column in SELECT sub-query consists at least one alias. This issue is similar as IMPALA-6340, but IMPALA-6340 only fixed the issue for the cases with the data expression for the decimal columns as constants. This patch fixed the issue by checking the query status of RuntimeState in Table Writer when ScalarExprEvaluator return NULL for decimal column. If there is an invalid deciaml error, the query will be failed without inserting NULL for decimal column. We did not change the behaviour for decimal_v1. NULL will be inserted to the table for invalid decimal values with warning message. Tests: - Added unit-tests for INSERT-SELECT and CTAS statements with overflowed decimal values to be inserted into tables. The overflowed decimal values are expressed as a constant expression, or as an expression with aliases. Also added cases to verify behaviour of decimal_v1 is unchanged. - Passed core tests. Change-Id: I64ce4ed194af81ef06401ffc1124e12f05b8da98 --- M be/src/common/status.h M be/src/exec/hdfs-text-table-writer.cc M be/src/exec/kudu-table-sink.cc M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exprs/decimal-operators-ir.cc M be/src/runtime/runtime-state.h M be/src/udf/udf.cc M be/src/udf/udf.h M common/thrift/generate_error_codes.py A testdata/workloads/functional-query/queries/QueryTest/decimal-insert-overflow-exprs.test M tests/query_test/test_decimal_queries.py 11 files changed, 288 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/17168/8 -- To view, visit http://gerrit.cloudera.org:8080/17168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I64ce4ed194af81ef06401ffc1124e12f05b8da98 Gerrit-Change-Number: 17168 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10564: Return error when inserting an invalid decimal value
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17168 ) Change subject: IMPALA-10564: Return error when inserting an invalid decimal value .. Patch Set 7: aggressive fix means calling state->CheckQueryState() in the end of HdfsTableSink::Send() function. It may affect other query behavior. I will not fix the issue in this way. -- To view, visit http://gerrit.cloudera.org:8080/17168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64ce4ed194af81ef06401ffc1124e12f05b8da98 Gerrit-Change-Number: 17168 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Mar 2021 23:11:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10598: Deflake test cache reload validation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17210 ) Change subject: IMPALA-10598: Deflake test_cache_reload_validation .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6993/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17210 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d491e90af461d5db3575a5840958d17ca90901c Gerrit-Change-Number: 17210 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Mar 2021 22:44:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10494: Making use of the min/max column stats to improve min/max filters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17075 ) Change subject: IMPALA-10494: Making use of the min/max column stats to improve min/max filters .. Patch Set 24: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8419/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df Gerrit-Change-Number: 17075 Gerrit-PatchSet: 24 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Mar 2021 21:55:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10494: Making use of the min/max column stats to improve min/max filters
Qifan Chen has uploaded a new patch set (#24). ( http://gerrit.cloudera.org:8080/17075 ) Change subject: IMPALA-10494: Making use of the min/max column stats to improve min/max filters .. IMPALA-10494: Making use of the min/max column stats to improve min/max filters This patch adds the functionality to compute the minimal and the maximal value for column types of integer, float/double, or date for parquet tables, and to make use of the new stats to discard min/max filters, in both hash join builders and Parquet scanners, whose coverage are too close to the actual range defined by the column min and max. The computation and dislay of the new column min/max stats can be controlled by two new Boolean query options (default to false): 1. compute_column_minmax_stats 2. show_column_minmax_stats Usage examples. set compute_column_minmax_stats=true; compute stats tpcds_parquet.store_sales; set show_column_minmax_stats=true; show column stats tpcds_parquet.store_sales; +---+--+-...---+-+-+ | Column| Type | #Falses | Min | Max | +---+--+-...---+-+-+ | ss_sold_time_sk | INT | -1 | 28800 | 75599 | | ss_item_sk| BIGINT | -1 | 1 | 18000 | | ss_customer_sk| INT | -1 | 1 | 10 | | ss_cdemo_sk | INT | -1 | 15 | 1920797 | | ss_hdemo_sk | INT | -1 | 1 | 7200| | ss_addr_sk| INT | -1 | 1 | 5 | | ss_store_sk | INT | -1 | 1 | 10 | | ss_promo_sk | INT | -1 | 1 | 300 | | ss_ticket_number | BIGINT | -1 | 1 | 24 | | ss_quantity | INT | -1 | 1 | 100 | | ss_wholesale_cost | DECIMAL(7,2) | -1 | -1 | -1 | | ss_list_price | DECIMAL(7,2) | -1 | -1 | -1 | | ss_sales_price| DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_discount_amt | DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_sales_price| DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_wholesale_cost | DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_list_price | DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_tax| DECIMAL(7,2) | -1 | -1 | -1 | | ss_coupon_amt | DECIMAL(7,2) | -1 | -1 | -1 | | ss_net_paid | DECIMAL(7,2) | -1 | -1 | -1 | | ss_net_paid_inc_tax | DECIMAL(7,2) | -1 | -1 | -1 | | ss_net_profit | DECIMAL(7,2) | -1 | -1 | -1 | | ss_sold_date_sk | INT | -1 | 2450816 | 2452642 | +---+--+-...---+-+-+ Only the min/max values for non-partition columns are stored in HMS. The min/max values for partition columns are computed in coordinator. The min-max filters, in C++ class or protobuf form, are augmented to deal with the always true state better. Once always true is set, the actual min and max values in the filter are no longer populated. Testing: - Added new compute/show stats tests in compute-stats-column-minmax.test; - Added new tests in overlap_min_max_filters.test to demonstrate the usefulness of column stats to quickly disable useless filters in both hash join builder and Parquet scanner; - Added tests in min-max-filter-test.cc to demonstrate method Or(), ToProtobuf() and constructor can deal with always true flag well; - core tests. TODO: 1. Enable the feature for Iceberg tables with Parquet data files. Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df --- M be/src/exec/catalog-op-executor.cc M be/src/exec/filter-context.cc M be/src/exec/filter-context.h M be/src/exec/hdfs-scanner.h M be/src/exec/incr-stats-util-test.cc M be/src/exec/incr-stats-util.cc M be/src/exec/incr-stats-util.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/service/hs2-util.cc M be/src/service/hs2-util.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/min-max-filter-test.cc M be/src/util/min-max-filter.cc M be/src/util/min-max-filter.h M common/thrift/CatalogObjects.thrift M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M
[Impala-ASF-CR] IMPALA-10590: Introduce admission service heartbeat mechanism
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17194 ) Change subject: IMPALA-10590: Introduce admission service heartbeat mechanism .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6990/ -- To view, visit http://gerrit.cloudera.org:8080/17194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia528d92268cea487ada20b476935a81166f5ad34 Gerrit-Change-Number: 17194 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Mar 2021 21:26:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10594: Handle failed coordinators in admissiond
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/17209 ) Change subject: IMPALA-10594: Handle failed coordinators in admissiond .. Patch Set 2: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/17209/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17209/2//COMMIT_MSG@13 PS2, Line 13: > There won't be any overadmission because the executors also monitor the sta Got it. Thanks for the explanation. -- To view, visit http://gerrit.cloudera.org:8080/17209 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I883f323bb765680ef24b3c3f51fb209dea15f0b0 Gerrit-Change-Number: 17209 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Mar 2021 20:08:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Update Python requests package to 2.20.0
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/17215 ) Change subject: Update Python requests package to 2.20.0 .. Patch Set 2: Hi, all! This failed with "Could not find a version that satisfies the requirement chardet<3.1.0,>=3.0.2". Do you know how to manage that? I can't recall how python deps are managed and what would be causing this if pypi clearly has 3.0.2 available: https://pypi.org/project/chardet/3.0.2/ -- To view, visit http://gerrit.cloudera.org:8080/17215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I092a641a12f38cdb45b0062c31ffb51c0c664800 Gerrit-Change-Number: 17215 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Mar 2021 19:59:52 + Gerrit-HasComments: No
[Impala-ASF-CR] Update Python requests package to 2.20.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17215 ) Change subject: Update Python requests package to 2.20.0 .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6992/ -- To view, visit http://gerrit.cloudera.org:8080/17215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I092a641a12f38cdb45b0062c31ffb51c0c664800 Gerrit-Change-Number: 17215 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Mar 2021 19:46:16 + Gerrit-HasComments: No
[Impala-ASF-CR] Update Python requests package to 2.20.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17215 ) Change subject: Update Python requests package to 2.20.0 .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/8418/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I092a641a12f38cdb45b0062c31ffb51c0c664800 Gerrit-Change-Number: 17215 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Mar 2021 19:41:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Update Python requests package to 2.20.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17215 ) Change subject: Update Python requests package to 2.20.0 .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6992/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I092a641a12f38cdb45b0062c31ffb51c0c664800 Gerrit-Change-Number: 17215 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Mar 2021 19:36:58 + Gerrit-HasComments: No
[Impala-ASF-CR] Update Python requests package to 2.20.0
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17215 Change subject: Update Python requests package to 2.20.0 .. Update Python requests package to 2.20.0 See https://2.python-requests.org/en/master/community/updates/#id8. This is currently only used in the tests, but it's best to fix this now. Change-Id: I092a641a12f38cdb45b0062c31ffb51c0c664800 --- M infra/python/deps/requirements.txt 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/17215/1 -- To view, visit http://gerrit.cloudera.org:8080/17215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I092a641a12f38cdb45b0062c31ffb51c0c664800 Gerrit-Change-Number: 17215 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] Update Python requests package to 2.20.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17215 ) Change subject: Update Python requests package to 2.20.0 .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I092a641a12f38cdb45b0062c31ffb51c0c664800 Gerrit-Change-Number: 17215 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Mar 2021 19:36:58 + Gerrit-HasComments: No
[Impala-ASF-CR] Update Paramiko to 2.4.2.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17214 ) Change subject: Update Paramiko to 2.4.2. .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9cc9ea4a0763c8b5303ca4e8482761ee2f53efa Gerrit-Change-Number: 17214 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Mar 2021 19:33:59 + Gerrit-HasComments: No
[Impala-ASF-CR] Update Paramiko to 2.4.2.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17214 ) Change subject: Update Paramiko to 2.4.2. .. Update Paramiko to 2.4.2. See https://www.paramiko.org/changelog.html#2.4.2. This shouldn't directly apply to Impala deployments, but it is best to fix this in test now. Change-Id: If9cc9ea4a0763c8b5303ca4e8482761ee2f53efa Reviewed-on: http://gerrit.cloudera.org:8080/17214 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M infra/python/deps/extended-test-requirements.txt 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If9cc9ea4a0763c8b5303ca4e8482761ee2f53efa Gerrit-Change-Number: 17214 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10579: Fix usage of RemoteIterator in FileSystemUtil
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/17171 ) Change subject: IMPALA-10579: Fix usage of RemoteIterator in FileSystemUtil .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/17171/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java: http://gerrit.cloudera.org:8080/#/c/17171/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@959 PS5, Line 959: curIter_ = subIter; Should curIter_ be cleared first so it is not stale if there is an exception? -- To view, visit http://gerrit.cloudera.org:8080/17171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I859bd4f976c51a34eb6a03cefd2ddcdf11656cea Gerrit-Change-Number: 17171 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 22 Mar 2021 18:07:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10564: Return error when inserting an invalid decimal value
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17168 ) Change subject: IMPALA-10564: Return error when inserting an invalid decimal value .. Patch Set 7: > Patch Set 7: > > When decimal_v2 is set as false, Impala return a warning and insert NULL. So > we don't need to introduce another query option. The bug is not related with > ABORT_ON_ERROR. Even ABORT_ON_ERROR is set false, we still should abort the > query with an error for overflowed decimal value. In first two cases, current > code abort the query with an error for invalid decimal value when > ABORT_ON_ERROR is set false. To fix the issue, we just return error in table > writer if we found got NULL for decimal columns as my patch set 2. Or fix it > more aggressively, by replacing "return Status:OK()" with "return > state->CheckQueryState())" in HdfsTableSink::Send() and KuduTableSink::Send(). Since decimal_v2 is the default, we only need to focus on that. Based on your 3 examples it is clear that we are already returning the Decimal overflow error in the common cases of 1 and 2. For the not common case 3, if you just ran the SELECT portion of the CTAS, it will also return the same error. So, it makes sense to return error for the CTAS as well. I agree that a new query option is not needed. For case 3, when running the CTAS, the log file does show 'UDF ERROR: Decimal expression overflowed' in the stack trace for HdfsTableSink::Send() but the execution continues and the row is inserted. Is this what you were referring to for the aggressive fix ? -- To view, visit http://gerrit.cloudera.org:8080/17168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64ce4ed194af81ef06401ffc1124e12f05b8da98 Gerrit-Change-Number: 17168 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Mar 2021 17:57:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10592: prevent pytest from hanging at exit.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17212 ) Change subject: IMPALA-10592: prevent pytest from hanging at exit. .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6991/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17212 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74cca8f577c7fbc4d394311e2f039cf4f68b08df Gerrit-Change-Number: 17212 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 22 Mar 2021 17:42:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10592: prevent pytest from hanging at exit.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17212 ) Change subject: IMPALA-10592: prevent pytest from hanging at exit. .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17212 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74cca8f577c7fbc4d394311e2f039cf4f68b08df Gerrit-Change-Number: 17212 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 22 Mar 2021 17:42:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10592: prevent pytest from hanging at exit.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17212 ) Change subject: IMPALA-10592: prevent pytest from hanging at exit. .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17212 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74cca8f577c7fbc4d394311e2f039cf4f68b08df Gerrit-Change-Number: 17212 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 22 Mar 2021 17:27:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10564: Return error when inserting an invalid decimal value
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17168 ) Change subject: IMPALA-10564: Return error when inserting an invalid decimal value .. Patch Set 7: When decimal_v2 is set as false, Impala return a warning and insert NULL. So we don't need to introduce another query option. The bug is not related with ABORT_ON_ERROR. Even ABORT_ON_ERROR is set false, we still should abort the query with an error for overflowed decimal value. In first two cases, current code abort the query with an error for invalid decimal value when ABORT_ON_ERROR is set false. To fix the issue, we just return error in table writer if we found got NULL for decimal columns as my patch set 2. Or fix it more aggressively, by replacing "return Status:OK()" with "return state->CheckQueryState())" in HdfsTableSink::Send() and KuduTableSink::Send(). -- To view, visit http://gerrit.cloudera.org:8080/17168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64ce4ed194af81ef06401ffc1124e12f05b8da98 Gerrit-Change-Number: 17168 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Mar 2021 17:26:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10564: Return error when inserting an invalid decimal value
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17168 ) Change subject: IMPALA-10564: Return error when inserting an invalid decimal value .. Patch Set 7: > Patch Set 7: > > (1 comment) > It is not like the intended behavior by design. Instead, it's more like a bug > that RuntimeState::CheckQueryState() is not called in a corner case. This does seem to be the underlying issue right ? Is this why when you set ABORT_ON_ERROR = true, the error was not propagated back to caller ? If we can fix this behavior, then yes I would agree we don't need to introduce a new query option. -- To view, visit http://gerrit.cloudera.org:8080/17168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64ce4ed194af81ef06401ffc1124e12f05b8da98 Gerrit-Change-Number: 17168 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Mar 2021 16:25:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10483: Support subqueries in Ranger masking policies
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17185 ) Change subject: IMPALA-10483: Support subqueries in Ranger masking policies .. IMPALA-10483: Support subqueries in Ranger masking policies This patch adds support for using subqueries in Ranger masking policies, i.e. column-masking/row-filtering policies. The subquery can reference either the current table or other tables. However, masking policies on these tables won't be applied recursively. This is consistent with Hive. One motivation is to avoid infinitely masking if it references the same table. Another motivation I think is to simplify the masking behavior, so when the admin is setting a masking expression, it can be considered as running in the admin's perspective (i.e. no masking). Implementation Before analyzing the query, the coordinator loads the metadata of all possibly used tables into the query's StmtTableCache. Table masking takes place after the analyzing phase. If the subquery filter introduces any new tables, the analyzer will fail to resolve them since their metadata is not loaded in the StmtTableCache. This patch modified the StmtMetadataLoader to also load those tables introduced by masking policies. So they can be resolved correctly. Tests - Add more complex tests in test_row_filtering Change-Id: I254df9f684c95c660f402abd99ca12dded7e764f Reviewed-on: http://gerrit.cloudera.org:8080/17185 Reviewed-by: Aman Sinha Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/authorization/TableMask.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/service/Frontend.java M testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test M testdata/workloads/functional-query/queries/QueryTest/ranger_row_filtering.test M tests/authorization/test_ranger.py 8 files changed, 298 insertions(+), 59 deletions(-) Approvals: Aman Sinha: Looks good to me, approved Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I254df9f684c95c660f402abd99ca12dded7e764f Gerrit-Change-Number: 17185 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10483: Support subqueries in Ranger masking policies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17185 ) Change subject: IMPALA-10483: Support subqueries in Ranger masking policies .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I254df9f684c95c660f402abd99ca12dded7e764f Gerrit-Change-Number: 17185 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 22 Mar 2021 15:52:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10564: Return error when inserting an invalid decimal value
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17168 ) Change subject: IMPALA-10564: Return error when inserting an invalid decimal value .. Patch Set 7: > Patch Set 7: > > (1 comment) Thanks Wenzhe for the update. Taking another look at your comments. -- To view, visit http://gerrit.cloudera.org:8080/17168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64ce4ed194af81ef06401ffc1124e12f05b8da98 Gerrit-Change-Number: 17168 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Mar 2021 15:43:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10590: Introduce admission service heartbeat mechanism
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17194 ) Change subject: IMPALA-10590: Introduce admission service heartbeat mechanism .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6990/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia528d92268cea487ada20b476935a81166f5ad34 Gerrit-Change-Number: 17194 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Mar 2021 15:37:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10590: Introduce admission service heartbeat mechanism
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17194 ) Change subject: IMPALA-10590: Introduce admission service heartbeat mechanism .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia528d92268cea487ada20b476935a81166f5ad34 Gerrit-Change-Number: 17194 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Mar 2021 15:37:49 + Gerrit-HasComments: No
[Impala-ASF-CR] Update Paramiko to 2.4.2.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17214 ) Change subject: Update Paramiko to 2.4.2. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8417/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9cc9ea4a0763c8b5303ca4e8482761ee2f53efa Gerrit-Change-Number: 17214 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Mar 2021 14:09:04 + Gerrit-HasComments: No
[Impala-ASF-CR] Update Paramiko to 2.4.2.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17214 ) Change subject: Update Paramiko to 2.4.2. .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9cc9ea4a0763c8b5303ca4e8482761ee2f53efa Gerrit-Change-Number: 17214 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Mar 2021 13:50:40 + Gerrit-HasComments: No
[Impala-ASF-CR] Update Paramiko to 2.4.2.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17214 ) Change subject: Update Paramiko to 2.4.2. .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6989/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9cc9ea4a0763c8b5303ca4e8482761ee2f53efa Gerrit-Change-Number: 17214 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Mar 2021 13:50:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Update Paramiko to 2.4.2.
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17214 Change subject: Update Paramiko to 2.4.2. .. Update Paramiko to 2.4.2. See https://www.paramiko.org/changelog.html#2.4.2. This shouldn't directly apply to Impala deployments, but it is best to fix this in test now. Change-Id: If9cc9ea4a0763c8b5303ca4e8482761ee2f53efa --- M infra/python/deps/extended-test-requirements.txt 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/17214/1 -- To view, visit http://gerrit.cloudera.org:8080/17214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If9cc9ea4a0763c8b5303ca4e8482761ee2f53efa Gerrit-Change-Number: 17214 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10483: Support subqueries in Ranger masking policies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17185 ) Change subject: IMPALA-10483: Support subqueries in Ranger masking policies .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8416/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I254df9f684c95c660f402abd99ca12dded7e764f Gerrit-Change-Number: 17185 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 22 Mar 2021 10:28:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10580: Implement ds theta union f() function
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/17179 )
Change subject: IMPALA-10580: Implement ds_theta_union_f() function
..
Patch Set 3:
(2 comments)
Thanks for these changes! Do you plan to implement ds_theta_union() as an
aggregate function as well?
http://gerrit.cloudera.org:8080/#/c/17179/3/be/src/exprs/datasketches-functions-ir.cc
File be/src/exprs/datasketches-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/17179/3/be/src/exprs/datasketches-functions-ir.cc@167
PS3, Line 167: if (!first_serialized_sketch.is_null &&
first_serialized_sketch.len > 0) {
: datasketches::theta_sketch::unique_ptr first_sketch_ptr;
: if (!DeserializeDsSketch(first_serialized_sketch,
_sketch_ptr)) {
: LogSketchDeserializationError(ctx);
: return StringVal::null();
: }
: union_sketch.update(*first_sketch_ptr);
: }
This part seems pretty similar to L175-182. Have you considered introducing a
function for this purpose? (e.g. update_sketch_toThera_union or such)
http://gerrit.cloudera.org:8080/#/c/17179/3/testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test
File
testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test:
http://gerrit.cloudera.org:8080/#/c/17179/3/testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test@436
PS3, Line 436: # Checks that ds_theta_union_f() returns an empty sketch for
NULL inputs.
Shouldn't this return null for null inputs? Have you checked the behaviour in
Hive?
--
To view, visit http://gerrit.cloudera.org:8080/17179
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8329979b81ceeaad739a43fab79768ca9c2916fa
Gerrit-Change-Number: 17179
Gerrit-PatchSet: 3
Gerrit-Owner: Fucun Chu
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Mon, 22 Mar 2021 10:26:49 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10597: Enable setting Iceberg table properties
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17207 ) Change subject: IMPALA-10597: Enable setting Iceberg table properties .. Patch Set 1: Thanks WangSheng for your comment. It's mainly for tables that are created via HiveCatalog. These tables will have the proper Input/Output format, SerDe, etc., but don't have 'iceberg.catalog'. But in this case Impala correctly assume that HiveCatalog was used, and will be able to load the table. After that the user can set e.g. 'iceberg.file_format'='orc'. Tables created by HadoopTables or HadoopCatalog are not necesserily have an HMS table associated with, so in those cases Impala users can just create an external table. -- To view, visit http://gerrit.cloudera.org:8080/17207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4b3506be4562a1ace3e6435867aadb3bdde7a8e2 Gerrit-Change-Number: 17207 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 22 Mar 2021 10:24:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10483: Support subqueries in Ranger masking policies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17185 ) Change subject: IMPALA-10483: Support subqueries in Ranger masking policies .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I254df9f684c95c660f402abd99ca12dded7e764f Gerrit-Change-Number: 17185 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 22 Mar 2021 10:07:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10483: Support subqueries in Ranger masking policies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17185 ) Change subject: IMPALA-10483: Support subqueries in Ranger masking policies .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6988/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I254df9f684c95c660f402abd99ca12dded7e764f Gerrit-Change-Number: 17185 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 22 Mar 2021 10:07:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9661: Avoid introducing unused columns in table masking view
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17199 ) Change subject: IMPALA-9661: Avoid introducing unused columns in table masking view .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17199 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib015a8ab528065907b27fbdceb8e2818deb814e1 Gerrit-Change-Number: 17199 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 22 Mar 2021 08:40:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9661: Avoid introducing unused columns in table masking view
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17199 ) Change subject: IMPALA-9661: Avoid introducing unused columns in table masking view .. IMPALA-9661: Avoid introducing unused columns in table masking view Previously, if a table has column masking policies, we replace its unanalyzed TableRef with an analyzed InlineViewRef (table masking view) in FromClause.analyze(). However, we can't detect which columns are actually used in the original query at this point. In fact, analyze() for SelectList, WhereClause, GroupByClause and other clauses containing SlotRefs happen after FromClause.analyze(). After the whole query block is analyzed, we can get the exact set of required columns. This patch refactor the codes to do table masking after analyze() to avoid introducing unused columns. Referenced columns of a TableRef are registered in analyze(), which helps to figure out what columns are actually needed. With this, we don't need to revert table masking in FromClause.reset(). The doTableMasking flag in AST is also removed since now the table mask is resolved once after analyze(). Tests: - Add more e2e tests in test_ranger.py - Run CORE tests Change-Id: Ib015a8ab528065907b27fbdceb8e2818deb814e1 Reviewed-on: http://gerrit.cloudera.org:8080/17199 Reviewed-by: Aman Sinha Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/StmtNode.java M fe/src/main/java/org/apache/impala/analysis/Subquery.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/main/java/org/apache/impala/authorization/TableMask.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_and_row_filtering.test M testdata/workloads/functional-query/queries/QueryTest/ranger_row_filtering.test M tests/authorization/test_ranger.py 23 files changed, 518 insertions(+), 239 deletions(-) Approvals: Aman Sinha: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/17199 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib015a8ab528065907b27fbdceb8e2818deb814e1 Gerrit-Change-Number: 17199 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
