[Impala-ASF-CR] IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17136 ) Change subject: IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6934/ -- To view, visit http://gerrit.cloudera.org:8080/17136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Gerrit-Change-Number: 17136 Gerrit-PatchSet: 6 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 03 Mar 2021 06:05:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10503: testdata load hits hive memory limit errors during hive inserts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17061 ) Change subject: IMPALA-10503: testdata load hits hive memory limit errors during hive inserts .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6936/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idac5f054e814070b983f7f57aef4ea9d54252bb2 Gerrit-Change-Number: 17061 Gerrit-PatchSet: 7 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Comment-Date: Wed, 03 Mar 2021 03:32:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10503: testdata load hits hive memory limit errors during hive inserts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17061 ) Change subject: IMPALA-10503: testdata load hits hive memory limit errors during hive inserts .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6935/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idac5f054e814070b983f7f57aef4ea9d54252bb2 Gerrit-Change-Number: 17061 Gerrit-PatchSet: 7 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Comment-Date: Wed, 03 Mar 2021 03:16:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/17116 ) Change subject: IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad .. Patch Set 14: (2 comments) http://gerrit.cloudera.org:8080/#/c/17116/14/be/src/service/impala-hs2-server.cc File be/src/service/impala-hs2-server.cc: http://gerrit.cloudera.org:8080/#/c/17116/14/be/src/service/impala-hs2-server.cc@1190 PS14, Line 1190: HS2_RETURN_IF_ERROR(return_val, THandleIdentifierToTUniqueId( > I think you may have misunderstood my previous comment here, sorry if I was This RPC uses common Client and Session logic on the external frontend to connect to Impalad. Originally it was not clear if we might need authentication for external frontend connections. The external port was added later but then rebased before this change to make it more clear how security would work and avoid having an interim security hole. I'd prefer to leave this as-is for now. http://gerrit.cloudera.org:8080/#/c/17116/14/be/src/util/backend-gflag-util.cc File be/src/util/backend-gflag-util.cc: http://gerrit.cloudera.org:8080/#/c/17116/14/be/src/util/backend-gflag-util.cc@258 PS14, Line 258: string saml2_private_key_password; > As is, this patch will return things from GetBackendConfig that are both se The intention with external frontends is to share Impala frontend code and functionality. As such, we have not limited access to backend metadata which may be needed in unforeseen places, especially as more functionality is added. As it stands, this mechanism simply mirrors what the Impala FE is getting, excepted pulled rather than pushed. The log case as easier to make as restricting access there is not going to cause functional problems. -- To view, visit http://gerrit.cloudera.org:8080/17116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14a3cee29f1fc91f4431b7ea89053bb3fbfa5e69 Gerrit-Change-Number: 17116 Gerrit-PatchSet: 14 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 03 Mar 2021 02:57:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6933/ -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 10 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 03 Mar 2021 02:21:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10522: Support external use of frontend libraries
Kurt Deschler has abandoned this change. ( http://gerrit.cloudera.org:8080/17146 ) Change subject: IMPALA-10522: Support external use of frontend libraries .. Abandoned Accidentally changed Change-Id -- To view, visit http://gerrit.cloudera.org:8080/17146 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I51775ef13acbdca02531ddb37296890675a0a2b9 Gerrit-Change-Number: 17146 Gerrit-PatchSet: 4 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10522: Support external use of frontend libraries
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/17115 ) Change subject: IMPALA-10522: Support external use of frontend libraries .. Patch Set 8: Looks like I changed the Change-I by accident. Will fix that. -- To view, visit http://gerrit.cloudera.org:8080/17115 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e3a84721ba196ec00773ce2923b19610b90edd9 Gerrit-Change-Number: 17115 Gerrit-PatchSet: 8 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 03 Mar 2021 01:49:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10494: Making use of the min/max column stats to improve min/max filters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17075 ) Change subject: IMPALA-10494: Making use of the min/max column stats to improve min/max filters .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8296/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df Gerrit-Change-Number: 17075 Gerrit-PatchSet: 14 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Wed, 03 Mar 2021 01:10:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10494: Making use of the min/max column stats to improve min/max filters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17075 ) Change subject: IMPALA-10494: Making use of the min/max column stats to improve min/max filters .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8295/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df Gerrit-Change-Number: 17075 Gerrit-PatchSet: 13 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Wed, 03 Mar 2021 01:02:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10494: Making use of the min/max column stats to improve min/max filters
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17075 ) Change subject: IMPALA-10494: Making use of the min/max column stats to improve min/max filters .. Patch Set 14: Fix incorrectly spelled query option name in overlap_min_max_filters.test. -- To view, visit http://gerrit.cloudera.org:8080/17075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df Gerrit-Change-Number: 17075 Gerrit-PatchSet: 14 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Wed, 03 Mar 2021 00:51:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10494: Making use of the min/max column stats to improve min/max filters
Qifan Chen has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/17075 ) Change subject: IMPALA-10494: Making use of the min/max column stats to improve min/max filters .. IMPALA-10494: Making use of the min/max column stats to improve min/max filters This patch adds the functionality to compute the minimal and the maximal value for a column of type integers, float or double for parquet tables, and to make use of the new stats to discard the min/max filters whose coverage are too close to the actual range. The computation and dislay of the new column min/max stats can be controlled by two new Boolean query options (default to false): 1. compute_column_minmax_stats 2. show_column_minmax_stats When enabled, two new columns 'Min' and 'Max' are added in the output of the show column command as shown below. set show_column_minmax_stats=true; show column stats tpcds_parquet.store_sales; +---+--+-...---+-+-+ | Column| Type | #Falses | Min | Max | +---+--+-...---+-+-+ | ss_sold_time_sk | INT | -1 | 28800 | 75599 | | ss_item_sk| BIGINT | -1 | 1 | 18000 | | ss_customer_sk| INT | -1 | 1 | 10 | | ss_cdemo_sk | INT | -1 | 15 | 1920797 | | ss_hdemo_sk | INT | -1 | 1 | 7200| | ss_addr_sk| INT | -1 | 1 | 5 | | ss_store_sk | INT | -1 | 1 | 10 | | ss_promo_sk | INT | -1 | 1 | 300 | | ss_ticket_number | BIGINT | -1 | 1 | 24 | | ss_quantity | INT | -1 | 1 | 100 | | ss_wholesale_cost | DECIMAL(7,2) | -1 | -1 | -1 | | ss_list_price | DECIMAL(7,2) | -1 | -1 | -1 | | ss_sales_price| DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_discount_amt | DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_sales_price| DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_wholesale_cost | DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_list_price | DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_tax| DECIMAL(7,2) | -1 | -1 | -1 | | ss_coupon_amt | DECIMAL(7,2) | -1 | -1 | -1 | | ss_net_paid | DECIMAL(7,2) | -1 | -1 | -1 | | ss_net_paid_inc_tax | DECIMAL(7,2) | -1 | -1 | -1 | | ss_net_profit | DECIMAL(7,2) | -1 | -1 | -1 | | ss_sold_date_sk | INT | -1 | 2450816 | 2452642 | +---+--+-...---+-+-+ Only the min/max values for non-partition columns are stored in HMS. The min/max values for partition columns are computed in coordinator. Testing: - Added TestLowAndHighValueShort and TestLowAndHighValueInt to IncrStatsUtilTest; - Add new tests in overlap_min_max_filters.test to demonstrate the usefulness of column stats to quickly disable useless filters; - Tested compute/show stats for integers, float and double column data types; - core tests. TODO: 1. Test compute stats for timestamp and date columns; 2. Add logic to disable min/max filters inside HJ builder via the column stats. Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df --- M be/src/exec/catalog-op-executor.cc M be/src/exec/filter-context.cc M be/src/exec/filter-context.h M be/src/exec/hdfs-scanner.h M be/src/exec/incr-stats-util-test.cc M be/src/exec/incr-stats-util.cc M be/src/exec/incr-stats-util.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/service/hs2-util.cc M be/src/service/hs2-util.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/min-max-filter.h M common/thrift/CatalogObjects.thrift M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java A
[Impala-ASF-CR] IMPALA-10494: Making use of the min/max column stats to improve min/max filters
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17075 ) Change subject: IMPALA-10494: Making use of the min/max column stats to improve min/max filters .. Patch Set 13: Added the following: 1. New test overlap_min_max_filters.test 2. Query option show_column_minmax_stats to facilitate shield existing column stats from format change 3. A new counter NumColumnStatsRejectedRowGroups. -- To view, visit http://gerrit.cloudera.org:8080/17075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df Gerrit-Change-Number: 17075 Gerrit-PatchSet: 13 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Wed, 03 Mar 2021 00:45:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10494: Making use of the min/max column stats to improve min/max filters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17075 ) Change subject: IMPALA-10494: Making use of the min/max column stats to improve min/max filters .. Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/17075/13/tests/metadata/test_compute_stats.py File tests/metadata/test_compute_stats.py: http://gerrit.cloudera.org:8080/#/c/17075/13/tests/metadata/test_compute_stats.py@430 PS13, Line 430: class TestParquetComputeColumnMinMax(ImpalaTestSuite): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/17075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df Gerrit-Change-Number: 17075 Gerrit-PatchSet: 13 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Wed, 03 Mar 2021 00:43:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10494: Making use of the min/max column stats to improve min/max filters
Qifan Chen has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/17075 ) Change subject: IMPALA-10494: Making use of the min/max column stats to improve min/max filters .. IMPALA-10494: Making use of the min/max column stats to improve min/max filters This patch adds the functionality to compute the minimal and the maximal value for a column of type integers, float or double for parquet tables, and to make use of the new stats to discard the min/max filters whose coverage are too close to the actual range. The computation and dislay of the new column min/max stats can be controlled by two new Boolean query options (default to false): 1. compute_column_minmax_stats 2. show_column_minmax_stats When enabled, two new columns 'Min' and 'Max' are added in the output of the show column command as shown below. set show_column_minmax_stats=true; show column stats tpcds_parquet.store_sales; +---+--+-...---+-+-+ | Column| Type | #Falses | Min | Max | +---+--+-...---+-+-+ | ss_sold_time_sk | INT | -1 | 28800 | 75599 | | ss_item_sk| BIGINT | -1 | 1 | 18000 | | ss_customer_sk| INT | -1 | 1 | 10 | | ss_cdemo_sk | INT | -1 | 15 | 1920797 | | ss_hdemo_sk | INT | -1 | 1 | 7200| | ss_addr_sk| INT | -1 | 1 | 5 | | ss_store_sk | INT | -1 | 1 | 10 | | ss_promo_sk | INT | -1 | 1 | 300 | | ss_ticket_number | BIGINT | -1 | 1 | 24 | | ss_quantity | INT | -1 | 1 | 100 | | ss_wholesale_cost | DECIMAL(7,2) | -1 | -1 | -1 | | ss_list_price | DECIMAL(7,2) | -1 | -1 | -1 | | ss_sales_price| DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_discount_amt | DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_sales_price| DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_wholesale_cost | DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_list_price | DECIMAL(7,2) | -1 | -1 | -1 | | ss_ext_tax| DECIMAL(7,2) | -1 | -1 | -1 | | ss_coupon_amt | DECIMAL(7,2) | -1 | -1 | -1 | | ss_net_paid | DECIMAL(7,2) | -1 | -1 | -1 | | ss_net_paid_inc_tax | DECIMAL(7,2) | -1 | -1 | -1 | | ss_net_profit | DECIMAL(7,2) | -1 | -1 | -1 | | ss_sold_date_sk | INT | -1 | 2450816 | 2452642 | +---+--+-...---+-+-+ Only the min/max values for non-partition columns are stored in HMS. The min/max values for partition columns are computed in coordinator. Testing: - Added TestLowAndHighValueShort and TestLowAndHighValueInt to IncrStatsUtilTest; - Add new tests in overlap_min_max_filters.test to demonstrate the usefulness of column stats to quickly disable useless filters; - Tested compute/show stats for integers, float and double column data types; - core tests. TODO: 1. Test compute stats for timestamp and date columns; 2. Add logic to disable min/max filters inside HJ builder via the column stats. Change-Id: I08581b44419bb8da5940cbf98502132acd1c86df --- M be/src/exec/catalog-op-executor.cc M be/src/exec/filter-context.cc M be/src/exec/filter-context.h M be/src/exec/hdfs-scanner.h M be/src/exec/incr-stats-util-test.cc M be/src/exec/incr-stats-util.cc M be/src/exec/incr-stats-util.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/service/hs2-util.cc M be/src/service/hs2-util.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/min-max-filter.h M common/thrift/CatalogObjects.thrift M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java A
[Impala-ASF-CR] IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17136 ) Change subject: IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6934/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Gerrit-Change-Number: 17136 Gerrit-PatchSet: 6 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 03 Mar 2021 00:02:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17136 ) Change subject: IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Gerrit-Change-Number: 17136 Gerrit-PatchSet: 6 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 03 Mar 2021 00:02:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10555: Fix Hit DCHECK in TmpFileGroup::RecoverWriteError
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17140 ) Change subject: IMPALA-10555: Fix Hit DCHECK in TmpFileGroup::RecoverWriteError .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6931/ -- To view, visit http://gerrit.cloudera.org:8080/17140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd9aea4bf2fff634ea9a30bf6e87987be4e1c611 Gerrit-Change-Number: 17140 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 02 Mar 2021 23:29:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17136 ) Change subject: IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6932/ -- To view, visit http://gerrit.cloudera.org:8080/17136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Gerrit-Change-Number: 17136 Gerrit-PatchSet: 5 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 02 Mar 2021 22:43:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8294/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 02 Mar 2021 22:04:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString This patch adds a wrapper around ThriftDebugString method provided in the Thrift library. The thrift's method can throw exceptions like (bad_alloc or TProtocolException) when the object cannot be serialized into a string representation. This exception is not caught on the catalogd side and it crashes the catalogd. The error was specifically seen in the catalogd's debug UI which provides a way to display a Table object. An exception thrown when rendering the table on the UI would have crashed the catalogd before the patch. In order to simulate this crash a new debug action called EXCEPTION was added. A new custom cluster test was added which simulates a exception thrown in this method and makes sure that fetching the table from catalogd's debug UI does not crash the catalogd. Tests: 1. Added a new custom cluster test which reproduces the crash. 2. Created a large table which has ~270K partitions and reduced the memory of the catalogd to 16GB. This configuration throws bad_alloc exception in the ThriftDebugString method and crashes the catalogd. After the patch the crash is averted and we see a error message on the debug UI instead. I also looped around the catalog web UI call for more than an hour to see if there are any other stability issues. I could not see any problems. Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Reviewed-on: http://gerrit.cloudera.org:8080/17110 Reviewed-by: Vihang Karajgaonkar Tested-by: Vihang Karajgaonkar --- M be/src/catalog/catalog-server.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h A be/src/util/thrift-debug-util.h A tests/custom_cluster/test_thrift_debug_string_exception.py 5 files changed, 121 insertions(+), 6 deletions(-) Approvals: Vihang Karajgaonkar: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 10 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 9: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 02 Mar 2021 21:44:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString This patch adds a wrapper around ThriftDebugString method provided in the Thrift library. The thrift's method can throw exceptions like (bad_alloc or TProtocolException) when the object cannot be serialized into a string representation. This exception is not caught on the catalogd side and it crashes the catalogd. The error was specifically seen in the catalogd's debug UI which provides a way to display a Table object. An exception thrown when rendering the table on the UI would have crashed the catalogd before the patch. In order to simulate this crash a new debug action called EXCEPTION was added. A new custom cluster test was added which simulates a exception thrown in this method and makes sure that fetching the table from catalogd's debug UI does not crash the catalogd. Tests: 1. Added a new custom cluster test which reproduces the crash. 2. Created a large table which has ~270K partitions and reduced the memory of the catalogd to 16GB. This configuration throws bad_alloc exception in the ThriftDebugString method and crashes the catalogd. After the patch the crash is averted and we see a error message on the debug UI instead. I also looped around the catalog web UI call for more than an hour to see if there are any other stability issues. I could not see any problems. Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 --- M be/src/catalog/catalog-server.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h A be/src/util/thrift-debug-util.h A tests/custom_cluster/test_thrift_debug_string_exception.py 5 files changed, 121 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/17110/9 -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8293/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 12 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 21:32:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10503: testdata load hits hive memory limit errors during hive inserts
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/17061 ) Change subject: IMPALA-10503: testdata load hits hive memory limit errors during hive inserts .. Patch Set 6: gerrit-verify-dryrun-external -- To view, visit http://gerrit.cloudera.org:8080/17061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idac5f054e814070b983f7f57aef4ea9d54252bb2 Gerrit-Change-Number: 17061 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Comment-Date: Tue, 02 Mar 2021 21:19:30 + Gerrit-HasComments: No
[Impala-ASF-CR] CDPD-17512: Impala testdata load hits hive memory limit errors during hive inserts
Kurt Deschler has abandoned this change. ( http://gerrit.cloudera.org:8080/17147 ) Change subject: CDPD-17512: Impala testdata load hits hive memory limit errors during hive inserts .. Abandoned Accidental push -- To view, visit http://gerrit.cloudera.org:8080/17147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I60a193ea5911d3c0e03fd6fc4a7036c415b20bb2 Gerrit-Change-Number: 17147 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] CDPD-17512: Impala testdata load hits hive memory limit errors during hive inserts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17147 ) Change subject: CDPD-17512: Impala testdata load hits hive memory limit errors during hive inserts .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8292/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60a193ea5911d3c0e03fd6fc4a7036c415b20bb2 Gerrit-Change-Number: 17147 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 02 Mar 2021 21:15:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Kurt Deschler has uploaded a new patch set (#12) to the change originally created by John Sherman. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - Added enable_external_fe_support option to start-impala-cluster.py - which when detected will start impala clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 137 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/17125/12 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 12 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8291/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 11 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 21:02:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8290/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 11 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 21:01:58 + Gerrit-HasComments: No
[Impala-ASF-CR] CDPD-17512: Impala testdata load hits hive memory limit errors during hive inserts
Hello Fang-Yu Rao, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/17147 to review the following change. Change subject: CDPD-17512: Impala testdata load hits hive memory limit errors during hive inserts .. CDPD-17512: Impala testdata load hits hive memory limit errors during hive inserts Changed the following hive settings to avoid hitting Hive container limit errors: hive.tez.container.size: 2048 hive.tez.java.opts: -Xmx1700m With these settings, testdata load completes without errors on a 32GB host. Change-Id: I60a193ea5911d3c0e03fd6fc4a7036c415b20bb2 Reviewed-on: https://gerrit.sjc.cloudera.com/c/cdh/impala/+/103294 Tested-by: Jenkins User CDH-Build: Jenkins User Reviewed-by: Fang-Yu Rao API-Compat: Kurt Deschler Quasar-L0: Kurt Deschler --- M fe/src/test/resources/hive-site.xml.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/17147/1 -- To view, visit http://gerrit.cloudera.org:8080/17147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I60a193ea5911d3c0e03fd6fc4a7036c415b20bb2 Gerrit-Change-Number: 17147 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Fang-Yu Rao
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello Thomas Tauber-Marshall, Joe McDonnell, John Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#11). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 989 insertions(+), 760 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/11 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 11 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Kurt Deschler has uploaded a new patch set (#11) to the change originally created by John Sherman. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - Added enable_external_fe_support option to start-impala-cluster.py - which when detected will start impala clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 136 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/17125/11 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 11 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 10 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 20:38:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6933/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 10 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 20:38:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 9 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 20:37:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10503: testdata load hits hive memory limit errors during hive inserts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17061 ) Change subject: IMPALA-10503: testdata load hits hive memory limit errors during hive inserts .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8289/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idac5f054e814070b983f7f57aef4ea9d54252bb2 Gerrit-Change-Number: 17061 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Comment-Date: Tue, 02 Mar 2021 20:32:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10503: testdata load hits hive memory limit errors during hive inserts
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/17061 ) Change subject: IMPALA-10503: testdata load hits hive memory limit errors during hive inserts .. Patch Set 6: gerrit-verify-dryrun-external -- To view, visit http://gerrit.cloudera.org:8080/17061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idac5f054e814070b983f7f57aef4ea9d54252bb2 Gerrit-Change-Number: 17061 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 02 Mar 2021 20:17:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10503: testdata load hits hive memory limit errors during hive inserts
Kurt Deschler has removed Tim Armstrong from this change. ( http://gerrit.cloudera.org:8080/17061 ) Change subject: IMPALA-10503: testdata load hits hive memory limit errors during hive inserts .. Removed reviewer Tim Armstrong. -- To view, visit http://gerrit.cloudera.org:8080/17061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: Idac5f054e814070b983f7f57aef4ea9d54252bb2 Gerrit-Change-Number: 17061 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler
[Impala-ASF-CR] IMPALA-10503: testdata load hits hive memory limit errors during hive inserts
Hello Aman Sinha, Fang-Yu Rao, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17061 to look at the new patch set (#6). Change subject: IMPALA-10503: testdata load hits hive memory limit errors during hive inserts .. IMPALA-10503: testdata load hits hive memory limit errors during hive inserts Changed the following hive settings to avoid hitting Hive container limit errors: hive.tez.container.size: 2048 hive.tez.java.opts: -Xmx1700m With these settings, testdata load completes without errors on a 32GB host. Reviewed-by: Fang-Yu Rao Change-Id: Idac5f054e814070b983f7f57aef4ea9d54252bb2 --- M fe/src/test/resources/hive-site.xml.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/17061/6 -- To view, visit http://gerrit.cloudera.org:8080/17061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idac5f054e814070b983f7f57aef4ea9d54252bb2 Gerrit-Change-Number: 17061 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10522: Support external use of frontend libraries
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17115 ) Change subject: IMPALA-10522: Support external use of frontend libraries .. Patch Set 8: I think you forgot to push the updated version of this patch? -- To view, visit http://gerrit.cloudera.org:8080/17115 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e3a84721ba196ec00773ce2923b19610b90edd9 Gerrit-Change-Number: 17115 Gerrit-PatchSet: 8 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 19:28:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17116 ) Change subject: IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad .. Patch Set 14: (2 comments) http://gerrit.cloudera.org:8080/#/c/17116/14/be/src/service/impala-hs2-server.cc File be/src/service/impala-hs2-server.cc: http://gerrit.cloudera.org:8080/#/c/17116/14/be/src/service/impala-hs2-server.cc@1190 PS14, Line 1190: HS2_RETURN_IF_ERROR(return_val, THandleIdentifierToTUniqueId( I think you may have misunderstood my previous comment here, sorry if I was unclear. Is there any reason for this rpc to take 'sessionHandle' at all, i.e. TGetBackendConfigReq.sessionHandle? I don't think it will ever be the case that the value of backend flags will be session-dependent, so removing TGetBackendConfigReq.sessionHandle would increase flexibility - eg. presumably this will just be getting called once at startup, when there isn't an actual user session anyways, so removing 'sessionHandle' would mean the external FE doesn't have to make a fake session just to call this. I guess one consideration is authentication - maybe we only want users that can create sessions to be able to call this - but I don't think that comes into play here since this will only be accessible over the external FE interface where authentication is always turned off. I may be missing something about how this will eventually evolve, and its not that big of a deal since like I said this will presumably only be called once at startup so perf doesn't really matter, so I'm fine if you prefer to leave it as is. http://gerrit.cloudera.org:8080/#/c/17116/14/be/src/util/backend-gflag-util.cc File be/src/util/backend-gflag-util.cc: http://gerrit.cloudera.org:8080/#/c/17116/14/be/src/util/backend-gflag-util.cc@258 PS14, Line 258: string saml2_private_key_password; As is, this patch will return things from GetBackendConfig that are both sensitive and (I assume) unnecessary for the external FE, such as passwords. Since we're restricting this to use by the external FE, which we implicitly trust, maybe that's fine, but it would be nice if we had some way to remove this kind of stuff from what gets returned, eg. by depending on the TAG_FLAG(sensitive)/CheckFlagAndRedact construct like we do to prevent these sorts of things from being printed in the logs or output on the debug webui, eg. see: https://github.com/apache/impala/blob/master/be/src/util/default-path-handlers.cc#L105 -- To view, visit http://gerrit.cloudera.org:8080/17116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14a3cee29f1fc91f4431b7ea89053bb3fbfa5e69 Gerrit-Change-Number: 17116 Gerrit-PatchSet: 14 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 19:28:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10503: testdata load hits hive memory limit errors during hive inserts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17061 ) Change subject: IMPALA-10503: testdata load hits hive memory limit errors during hive inserts .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8288/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idac5f054e814070b983f7f57aef4ea9d54252bb2 Gerrit-Change-Number: 17061 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 02 Mar 2021 19:26:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10503: testdata load hits hive memory limit errors during hive inserts
Hello Aman Sinha, Fang-Yu Rao, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17061 to look at the new patch set (#5). Change subject: IMPALA-10503: testdata load hits hive memory limit errors during hive inserts .. IMPALA-10503: testdata load hits hive memory limit errors during hive inserts Changed the following hive settings to avoid hitting Hive container limit errors: hive.tez.container.size: 2048 hive.tez.java.opts: -Xmx1700m With these settings, testdata load completes without errors on a 32GB host. Reviewed-by: Fang-Yu Rao Change-Id: Idac5f054e814070b983f7f57aef4ea9d54252bb2 --- M fe/src/test/resources/hive-site.xml.py M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test 2 files changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/17061/5 -- To view, visit http://gerrit.cloudera.org:8080/17061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idac5f054e814070b983f7f57aef4ea9d54252bb2 Gerrit-Change-Number: 17061 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 10: (3 comments) http://gerrit.cloudera.org:8080/#/c/17104/10/be/src/service/impala-server.h File be/src/service/impala-server.h: http://gerrit.cloudera.org:8080/#/c/17104/10/be/src/service/impala-server.h@679 PS10, Line 679: external_exec_request is a statement that was prepared by an external frontend using : /// Impala PlanNodes. or null if the external frontend isn't being used http://gerrit.cloudera.org:8080/#/c/17104/10/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17104/10/be/src/service/impala-server.cc@1207 PS10, Line 1207: (*query_handle)->SetExecRequestKrpcAddress(ExecEnv::GetInstance()->krpc_address()); I think you misunderstood my previous comment here, sorry if I was unclear. I was suggesting moving all of these calls to set other values on the TExecRequest into SetExternalPlan() (not just the call to set local_time, which you did move). If you do that, then I think you can eliminate most of the extra ClientRequestState functions that you added - SetExecRequestQueryId/SetExecRequestHostname/etc. http://gerrit.cloudera.org:8080/#/c/17104/10/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/17104/10/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1080 PS10, Line 1080: false could you add a "/* must_exist */" before this, for readability, ie: tbl = getTable(tblName.getDb(), tblName.getTbl(), /* must_exist */ false); here and elsewhere in this patch -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 10 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 18:55:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17136 ) Change subject: IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Gerrit-Change-Number: 17136 Gerrit-PatchSet: 5 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 02 Mar 2021 18:29:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17136 ) Change subject: IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6932/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Gerrit-Change-Number: 17136 Gerrit-PatchSet: 5 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 02 Mar 2021 18:29:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17136 ) Change subject: IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Gerrit-Change-Number: 17136 Gerrit-PatchSet: 4 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 02 Mar 2021 17:58:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9661: [WIP] Avoid introducing unused columns in table masking view
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15838 ) Change subject: IMPALA-9661: [WIP] Avoid introducing unused columns in table masking view .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/8287/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15838 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iedc7f15347a50626a9f2aff549bc49338cf73831 Gerrit-Change-Number: 15838 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 02 Mar 2021 17:57:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17136 ) Change subject: IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/17136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Gerrit-Change-Number: 17136 Gerrit-PatchSet: 4 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 02 Mar 2021 17:48:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10555: Fix Hit DCHECK in TmpFileGroup::RecoverWriteError
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17140 ) Change subject: IMPALA-10555: Fix Hit DCHECK in TmpFileGroup::RecoverWriteError .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd9aea4bf2fff634ea9a30bf6e87987be4e1c611 Gerrit-Change-Number: 17140 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 02 Mar 2021 17:47:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10555: Fix Hit DCHECK in TmpFileGroup::RecoverWriteError
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17140 ) Change subject: IMPALA-10555: Fix Hit DCHECK in TmpFileGroup::RecoverWriteError .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6931/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd9aea4bf2fff634ea9a30bf6e87987be4e1c611 Gerrit-Change-Number: 17140 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 02 Mar 2021 17:47:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10555: Fix Hit DCHECK in TmpFileGroup::RecoverWriteError
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17140 ) Change subject: IMPALA-10555: Fix Hit DCHECK in TmpFileGroup::RecoverWriteError .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd9aea4bf2fff634ea9a30bf6e87987be4e1c611 Gerrit-Change-Number: 17140 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 02 Mar 2021 17:46:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9661: [WIP] Avoid introducing unused columns in table masking view
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15838 ) Change subject: IMPALA-9661: [WIP] Avoid introducing unused columns in table masking view .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/15838/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/15838/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@721 PS2, Line 721: // Fang-Yu: Could consider making the following block a method, e.g., computeResolvedPath(). line too long (96 > 90) http://gerrit.cloudera.org:8080/#/c/15838/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@774 PS2, Line 774: // // factory supports column masking. If both of these are false, return the unmasked line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/15838/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/15838/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@279 PS2, Line 279: // tblRef.analyze(analyzer) in FromClause#analyze() (inside that for-loop). line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/15838 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iedc7f15347a50626a9f2aff549bc49338cf73831 Gerrit-Change-Number: 15838 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 02 Mar 2021 17:32:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9661: [WIP] Avoid introducing unused columns in table masking view
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15838 Change subject: IMPALA-9661: [WIP] Avoid introducing unused columns in table masking view .. IMPALA-9661: [WIP] Avoid introducing unused columns in table masking view Need to figure out why the following query could not be analyzed correctly. with t as (select id, bool_col, int_col, string_col from default.my_alltypestiny) select id, bool_col, string_col from t; Change-Id: Iedc7f15347a50626a9f2aff549bc49338cf73831 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java 3 files changed, 145 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/15838/2 -- To view, visit http://gerrit.cloudera.org:8080/15838 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iedc7f15347a50626a9f2aff549bc49338cf73831 Gerrit-Change-Number: 15838 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8286/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 9 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 17:17:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Hello Aman Sinha, Thomas Tauber-Marshall, Kurt Deschler, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17125 to look at the new patch set (#9). Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - Added enable_external_fe_support option to start-impala-cluster.py - which when detected will start impala clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 137 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/17125/9 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 9 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8285/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 10 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 15:54:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17116 ) Change subject: IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8283/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14a3cee29f1fc91f4431b7ea89053bb3fbfa5e69 Gerrit-Change-Number: 17116 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 15:52:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10549: Register transactions from external frontend DML
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17122 ) Change subject: IMPALA-10549: Register transactions from external frontend DML .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8284/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db Gerrit-Change-Number: 17122 Gerrit-PatchSet: 8 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 15:51:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8282/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 15:49:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Kurt Deschler has posted comments on this change. (
http://gerrit.cloudera.org:8080/17104 )
Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of
externally compiled statements
..
Patch Set 7:
(13 comments)
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/runtime/query-driver.cc
File be/src/runtime/query-driver.cc:
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/runtime/query-driver.cc@67
PS7, Line 67: exec_request
> Nit: I think it would be clearer to call this 'external_exec_request'
Done
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-hs2-server.cc
File be/src/service/impala-hs2-server.cc:
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-hs2-server.cc@458
PS7, Line 458: execRequest
> nit: exec_request (here and several other places in this patch)
Done
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.h
File be/src/service/impala-server.h:
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.h@678
PS7, Line 678: /// have been checked out.
> comment might benefit from a brief mention of what 'exec_request' is
Done
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@319
PS7, Line 319:
"TExecRequest-{internal|external}.{query_id.hi}-{query_id.lo}");
> might be worth explicitly saying this is for debugging
Done
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1114
PS7, Line 1114: if (FLAGS_dump_exec_request_path.empty())
: return;
> nit: this can go on one line (and if it couldn't, Impala always uses braces
Done
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1118
PS7, Line 1118: FLAGS_dump_exec_request_path + "/TExecRequest-" + dumpType +
"." +
: std::to_string(queryID.hi) + "-" +
std::to_string(queryID.lo)
> For constructing the filename, I think it would be cleaner to use the Subst
Done
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1140
PS7, Line 1140: exec_request
> It might be clearer for this to be 'external_exec_request' to emphasize tha
Done
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1191
PS7, Line 1191: exec_request
> Nit: From a style point, we prefer explicit checks against nullptr ("exec_r
Done
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1198
PS7, Line 1198: exec_request
> Nit: Same here (use explicit nullptr checks)
Done
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1202
PS7, Line 1202: // Update the externally provided exec_request with Impala's
query_id()
: (*query_handle)->SetExecRequestQueryId(query_id);
: // Update coordinator related internal addresses
: (*query_handle)->SetExecRequestHostname(
:
ExecEnv::GetInstance()->configured_backend_address().hostname);
:
(*query_handle)->SetExecRequestKrpcAddress(ExecEnv::GetInstance()->krpc_address());
: // Update the field of 'local_time_zone' of 'query_ctx'.
:
(*query_handle)->SetExecRequestLocalTimeZone(query_ctx.local_time_zone);
> might be nice to move this into QueryDriver::SetExternalPlan()
Done
http://gerrit.cloudera.org:8080/#/c/17104/7/be/src/service/impala-server.cc@1213
PS7, Line 1213:
RETURN_IF_ERROR((*query_handle)->UpdateQueryStatus(exec_status));
> What's the point of this call? I think it will always be called with OK and
Removed
http://gerrit.cloudera.org:8080/#/c/17104/7/common/thrift/CMakeLists.txt
File common/thrift/CMakeLists.txt:
http://gerrit.cloudera.org:8080/#/c/17104/7/common/thrift/CMakeLists.txt@70
PS7, Line 70: # Also do not generate ImpalaService.thrift because the
generated code doesn't
: # compile with hive if the thrift version in hive is 0.9.0
> Nit: We can remove this part of the comment
Done
http://gerrit.cloudera.org:8080/#/c/17104/7/fe/pom.xml
File fe/pom.xml:
http://gerrit.cloudera.org:8080/#/c/17104/7/fe/pom.xml@434
PS7, Line 434: org.apache.hive
: hive-classification
: ${hive.version}
> Does this bring in any dependencies that we need to exclude?
I checked both and only hive-classification.jar was added
--
To view, visit http://gerrit.cloudera.org:8080/17104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa
Gerrit-Change-Number: 17104
Gerrit-PatchSet: 7
Gerrit-Owner: Kurt Deschler
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: John
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello Thomas Tauber-Marshall, Joe McDonnell, John Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#10). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 989 insertions(+), 760 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/10 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 10 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10549: Register transactions from external frontend DML
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/17122 )
Change subject: IMPALA-10549: Register transactions from external frontend DML
..
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17122/8/fe/src/main/java/org/apache/impala/service/JniFrontend.java
File fe/src/main/java/org/apache/impala/service/JniFrontend.java:
http://gerrit.cloudera.org:8080/#/c/17122/8/fe/src/main/java/org/apache/impala/service/JniFrontend.java@695
PS8, Line 695: public void addTransaction(byte[] thriftQueryContext) throws
TransactionException, ImpalaException {
line too long (102 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/17122
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db
Gerrit-Change-Number: 17122
Gerrit-PatchSet: 8
Gerrit-Owner: Kurt Deschler
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: John Sherman
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Comment-Date: Tue, 02 Mar 2021 15:31:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10549: Register transactions from external frontend DML
Hello Thomas Tauber-Marshall, Joe McDonnell, John Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17122 to look at the new patch set (#8). Change subject: IMPALA-10549: Register transactions from external frontend DML .. IMPALA-10549: Register transactions from external frontend DML This change registers transactions that were started by an external frontend so that coordinator keepalive can track them properly. Testing: manually tested using DMLs from external frontend Reviewed-by: John Sherman Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db --- M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java 5 files changed, 38 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/17122/8 -- To view, visit http://gerrit.cloudera.org:8080/17122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db Gerrit-Change-Number: 17122 Gerrit-PatchSet: 8 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad
Hello Thomas Tauber-Marshall, Joe McDonnell, John Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17116 to look at the new patch set (#13). Change subject: IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad .. IMPALA-10546: Add ImpalaServer interface to retrieve BackendConfig from impalad This patch add a new interface ImpalaServer::GetBackendConfig() that returns the current TBackendGflags from impalad. Testing: Called new interface from external frontend. Verified that TBackendGflags were populated correctly. Reviewed-by: John Sherman Change-Id: I14a3cee29f1fc91f4431b7ea89053bb3fbfa5e69 --- M be/src/catalog/catalog.cc M be/src/rpc/hs2-http-test.cc M be/src/service/frontend.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.h M be/src/util/backend-gflag-util.cc M be/src/util/backend-gflag-util.h M be/src/util/logging-support.cc M common/thrift/ImpalaService.thrift M tests/hs2/test_hs2.py 10 files changed, 64 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/17116/13 -- To view, visit http://gerrit.cloudera.org:8080/17116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I14a3cee29f1fc91f4431b7ea89053bb3fbfa5e69 Gerrit-Change-Number: 17116 Gerrit-PatchSet: 13 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17104 ) Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/17104/9/be/src/runtime/query-driver.cc File be/src/runtime/query-driver.cc: http://gerrit.cloudera.org:8080/#/c/17104/9/be/src/runtime/query-driver.cc@76 PS9, Line 76: exec_request_->query_exec_request.query_ctx.__set_local_time_zone(query_ctx.local_time_zone); line too long (95 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 15:29:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements
Hello Thomas Tauber-Marshall, Joe McDonnell, John Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17104 to look at the new patch set (#9). Change subject: IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements .. IMPALA-10535: Add interface to ImpalaServer for execution of externally compiled statements The ExecutePlannedStatement interface allows an externally supplied TExecRequest to be executed by impalad. The TExecRequest must be fully populated and will be sent directly to the backend for execution. The following fields in the TExecRequest are updated by the coordinator: - Hostname - KRPC address - Local Timezone In order to add the interface to ImpalaInternalService.thrift, several of the thrift classes were moved to Query.thrift to avoid a circular dependency with Frontend.thrift. Added functionality to format and dump TExecRequest structures to path specified in debug flag dump_exec_request_path. A start timestamp field has been added to TExecRequest to represent the interval in the query profile between when the request was sent by the external frontend and handled by the backend. A local timestamp field has been added to the Ping result struct to return the current backend timestamp. This is used by the external to frontend to populate the start timestamp. Also included is a change to avoid generating silent AnalysisExceptions during table resolution. Tested with TExecRequest structures populated by external frontend. Local timezone change tested withe INT64 TIMESTAMP datatype Reviewed-by: John Sherman Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/rpc/hs2-http-test.cc M be/src/runtime/debug-options.h M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A common/thrift/Query.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 21 files changed, 988 insertions(+), 760 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/17104/9 -- To view, visit http://gerrit.cloudera.org:8080/17104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace716dd67290f08441857dc02d2428b0e335eaa Gerrit-Change-Number: 17104 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
John Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 8: I'll investigate the failure(s). Initial look was it is unit test - which I should have ran locally in the first place (but evidently did not). -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 8 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 15:20:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10222: CREATE TABLE AS SELECT for Iceberg tables
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/17130 )
Change subject: IMPALA-10222: CREATE TABLE AS SELECT for Iceberg tables
..
Patch Set 1:
(5 comments)
Nice work, Zoltan! I left some comments, nothing too serious.
http://gerrit.cloudera.org:8080/#/c/17130/1/be/src/exec/hdfs-table-sink.cc
File be/src/exec/hdfs-table-sink.cc:
http://gerrit.cloudera.org:8080/#/c/17130/1/be/src/exec/hdfs-table-sink.cc@545
PS1, Line 545: new HdfsParquetTableWriter(
I'm a bit confused here. This is for Iceberg with Parquet as file format, but
don't we support other formats such as ORC?
http://gerrit.cloudera.org:8080/#/c/17130/1/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
File fe/src/main/java/org/apache/impala/analysis/InsertStmt.java:
http://gerrit.cloudera.org:8080/#/c/17130/1/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@514
PS1, Line 514: if (isStaticPartitioning) {
: throw new AnalysisException("Static partitioning is
not supported for " +
: "Iceberg tables.");
: }
You can move this check inside the for loop above.
http://gerrit.cloudera.org:8080/#/c/17130/1/fe/src/main/java/org/apache/impala/catalog/CtasTargetTable.java
File fe/src/main/java/org/apache/impala/catalog/CtasTargetTable.java:
http://gerrit.cloudera.org:8080/#/c/17130/1/fe/src/main/java/org/apache/impala/catalog/CtasTargetTable.java@48
PS1, Line 48: // colsByPos[i] refers to the ith column in the table. The
first numClusteringCols are
nit: indentation
http://gerrit.cloudera.org:8080/#/c/17130/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java
File fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java:
http://gerrit.cloudera.org:8080/#/c/17130/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java@74
PS1, Line 74: private final FeDb db_;
: private final org.apache.hadoop.hive.metastore.api.Table
msTable_;
CtasTargetTble already has members with the same name, abs as I see they get
the same value as well. Are these needed here?
http://gerrit.cloudera.org:8080/#/c/17130/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java@222
PS1, Line 222: public void addColumn(Column col) {
Can't you use IcebergColumn type for the param? You could get rid of the DCHECK
in L223 then.
--
To view, visit http://gerrit.cloudera.org:8080/17130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81d2084e401b9fa74d5ad161b51fd3e2aa3fcc67
Gerrit-Change-Number: 17130
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Tue, 02 Mar 2021 15:19:43 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/17136 ) Change subject: IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/17136/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17136/3//COMMIT_MSG@9 PS3, Line 9: “ > nit: non-ascii double quote " Done http://gerrit.cloudera.org:8080/#/c/17136/3//COMMIT_MSG@19 PS3, Line 19: ’ > nit: non-ascii single quote ' Done -- To view, visit http://gerrit.cloudera.org:8080/17136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Gerrit-Change-Number: 17136 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 02 Mar 2021 14:04:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
Hello Quanlong Huang, Thomas Tauber-Marshall, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17136 to look at the new patch set (#4). Change subject: IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build .. IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build For start option "scratch_dirs", it only considers local filesystem as the default filesystem, regardless of the setting of DefaultFS(for a remote scratch dir, it needs to explicitly set it with the remote fs prefix). However, the function AssignQueue() would assign the queue based on not only the path string but also the default filesystem setting. For example, if scratch_dirs is set as "/tmp", the scratch dir is supposed to be in the local filesystem, but the AssignQueue() would consider it as "s3a://xxx/tmp" if a s3 path is set as the default fs. To fix this, the solution is to add a bool variable to AssignQueue() to decide whether or not to check the default fs setting when parsing the file path. For all of the scratch dirs, AssignQueue() won't check the default fs. Tests: Added a unit testcase: TmpFileMgrTest::TestSpillingWithRemoteDefaultFS. Ran and Passed TmpFileMgrTest. Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 --- M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/runtime/test-env.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc 6 files changed, 56 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/17136/4 -- To view, visit http://gerrit.cloudera.org:8080/17136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Gerrit-Change-Number: 17136 Gerrit-PatchSet: 4 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] [WIP] IMPALA-7712: Support Google Cloud Storage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17121 ) Change subject: [WIP] IMPALA-7712: Support Google Cloud Storage .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8281/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia91ec956de3b620cccf6a1244b56b7da7a45b32b Gerrit-Change-Number: 17121 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 02 Mar 2021 13:52:35 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7712: Support Google Cloud Storage
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17121 ) Change subject: [WIP] IMPALA-7712: Support Google Cloud Storage .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/17121/2/tests/custom_cluster/test_hive_text_codec_interop.py File tests/custom_cluster/test_hive_text_codec_interop.py: http://gerrit.cloudera.org:8080/#/c/17121/2/tests/custom_cluster/test_hive_text_codec_interop.py@24 PS2, Line 24: from tests.common.skip import SkipIfS3, SkipIfGCS > flake8: F401 'tests.common.skip.SkipGCS' imported but unused Done http://gerrit.cloudera.org:8080/#/c/17121/2/tests/custom_cluster/test_hive_text_codec_interop.py@55 PS2, Line 55: S > flake8: F821 undefined name 'SkipIfGCS' Done -- To view, visit http://gerrit.cloudera.org:8080/17121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia91ec956de3b620cccf6a1244b56b7da7a45b32b Gerrit-Change-Number: 17121 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 02 Mar 2021 13:32:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-7712: Support Google Cloud Storage
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17121 to look at the new patch set (#3). Change subject: [WIP] IMPALA-7712: Support Google Cloud Storage .. [WIP] IMPALA-7712: Support Google Cloud Storage This patch adds support for GCS(Google Cloud Storage). Test steps: - Compile and create test data on a GCE instance. - Upload test data to a GCS bucket. - Modify the filesystem prefix of all locations in HMS DB to point to the GCS bucket. Remove some hdfs caching params. - TODO: Run CORE tests. Change-Id: Ia91ec956de3b620cccf6a1244b56b7da7a45b32b --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/bin/load-test-warehouse-snapshot.sh M testdata/bin/run-all.sh M tests/authorization/test_ranger.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_event_processing.py M tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/custom_cluster/test_hive_text_codec_interop.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_lineage.py M tests/custom_cluster/test_local_catalog.py M tests/custom_cluster/test_local_tz_conversion.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/custom_cluster/test_topic_update_frequency.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_catalogd_debug_actions.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_reset_metadata.py M tests/metadata/test_stale_metadata.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_acid.py M tests/query_test/test_aggregation.py M tests/query_test/test_date_queries.py M tests/query_test/test_hbase_queries.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_insert_permutation.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_resource_limits.py M tests/query_test/test_scanners.py M tests/stress/test_acid_stress.py M tests/stress/test_ddl_stress.py M tests/util/filesystem_utils.py 61 files changed, 263 insertions(+), 53 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/17121/3 -- To view, visit http://gerrit.cloudera.org:8080/17121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia91ec956de3b620cccf6a1244b56b7da7a45b32b Gerrit-Change-Number: 17121 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10523: Fix impala-shell crash in printing error messages that contain UTF-8 characters
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/17099 ) Change subject: IMPALA-10523: Fix impala-shell crash in printing error messages that contain UTF-8 characters .. Patch Set 2: (1 comment) Hi Quanlong, looks good to me, just one minor observation. http://gerrit.cloudera.org:8080/#/c/17099/2/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/17099/2/shell/impala_shell.py@1321 PS2, Line 1321: if sys.version_info.major == 2 and isinstance(msg, unicode): nit: could you add a short one line comment that explains this condition? -- To view, visit http://gerrit.cloudera.org:8080/17099 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie10f5b03ecc5877053c2fbada1afaf256b423a71 Gerrit-Change-Number: 17099 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 02 Mar 2021 12:33:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10520: Implement ds theta intersect() function
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/17088 ) Change subject: IMPALA-10520: Implement ds_theta_intersect() function .. Patch Set 3: Code-Review+1 (2 comments) Thanks for this patch! In overall this looks great, I just had some minor comments. http://gerrit.cloudera.org:8080/#/c/17088/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17088/3//COMMIT_MSG@14 PS3, Line 14: an nit: not needed http://gerrit.cloudera.org:8080/#/c/17088/3/testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test File testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test: http://gerrit.cloudera.org:8080/#/c/17088/3/testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test@271 PS3, Line 271: and checks if the intersection : # produces the same result as if these sketches were used separately to get the estimates Could you add tests that cover the second part of this sentence so that we can sew what ds_theta_intersect() gives when processing the sketches separately (and to see if they in fact match with the results of this test)? -- To view, visit http://gerrit.cloudera.org:8080/17088 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I80e68c2151c4604f0386d3dfb004c82b10293f97 Gerrit-Change-Number: 17088 Gerrit-PatchSet: 3 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 02 Mar 2021 10:13:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10377: Improve the accuracy of resource estimation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16842 ) Change subject: IMPALA-10377: Improve the accuracy of resource estimation .. Patch Set 20: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8280/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16842 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1 Gerrit-Change-Number: 16842 Gerrit-PatchSet: 20 Gerrit-Owner: liuyao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: liuyao Gerrit-Comment-Date: Tue, 02 Mar 2021 09:47:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10377: Improve the accuracy of resource estimation
Hello Aman Sinha, Zoltan Borok-Nagy, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16842 to look at the new patch set (#20). Change subject: IMPALA-10377: Improve the accuracy of resource estimation .. IMPALA-10377: Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate AggregationNode 1.MemoryEstimate = Ndv * (AvgRowSize + SizeOfBucket) 2.When estimating the Ndv of merge aggregation, Ndv should be divided only once. 3.If there is no grouping exprs, MemoryEstimate = MIN_PLAIN_AGG_MEM SortNode 1.MemoryEstimate = Cardinality * AvgRowSize. Memory used when there is enough memory HashJoinNode 1.MemoryEstimate= DataRows + Buckets + DuplicateNodes, DataRows = RightTableCardinality * AvgRowSize, Buckets= roundUpToPowerOf2(RightTableCardinality) * SizeOfBucket, DuplicateNodes = (RightTableCardinality - RightNdv) * SizeOfDuplicateNode KuduScanNode 1.MemoryEstimate = Columns * BytesPerColumn * MaxScannerThreads, Columns are scanned in query, not all the columns of the table UnitTest 1.CardinalityTest adds test cases to test memory estimation. Modify existing test cases related to memory estimation Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1 --- M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java M testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/constant-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection-hdfs-num-rows-est-enabled.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters-hdfs-num-rows-est-enabled.test M testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters.test M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering-disabled.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test M testdata/workloads/functional-planner/queries/PlannerTest/partition-pruning.test M testdata/workloads/functional-planner/queries/PlannerTest/preagg-bytes-limit.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/result-spooling.test M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-query-options.test M testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q01.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q04.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q06.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q07.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q08.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q09.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q11.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q12.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q13.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q15.test M
[Impala-ASF-CR] IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17136 ) Change subject: IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build .. Patch Set 3: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/17136/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17136/3//COMMIT_MSG@9 PS3, Line 9: “ nit: non-ascii double quote " http://gerrit.cloudera.org:8080/#/c/17136/3//COMMIT_MSG@19 PS3, Line 19: ’ nit: non-ascii single quote ' -- To view, visit http://gerrit.cloudera.org:8080/17136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53 Gerrit-Change-Number: 17136 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 02 Mar 2021 09:13:50 + Gerrit-HasComments: Yes
