[Impala-ASF-CR](2.x) Revert "IMPALA-7156: temporary revert to older Sentry version"
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10689 ) Change subject: Revert "IMPALA-7156: temporary revert to older Sentry version" .. Patch Set 1: Build looks good after updating the Hadoop components. Lars/Tim, can you review this? -- To view, visit http://gerrit.cloudera.org:8080/10689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I908a91f607f0a9ce8cb972ea8673771665ad88db Gerrit-Change-Number: 10689 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 12 Jun 2018 06:52:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7158: Fix HdfsScanNodeBase::progress 's init
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/10672 ) Change subject: IMPALA-7158: Fix HdfsScanNodeBase::progress_'s init .. Patch Set 2: Updated the commit message. -- To view, visit http://gerrit.cloudera.org:8080/10672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2a738edea80ff3fb13ff368b4093c8b4ef34df7 Gerrit-Change-Number: 10672 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 12 Jun 2018 06:49:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7158: Fix HdfsScanNodeBase::progress 's init
Hello Sailesh Mukil, Tim Armstrong, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10672 to look at the new patch set (#2). Change subject: IMPALA-7158: Fix HdfsScanNodeBase::progress_'s init .. IMPALA-7158: Fix HdfsScanNodeBase::progress_'s init (Testing) Verified that the correct node id is being logged with this patch and --v=2. Change-Id: Id2a738edea80ff3fb13ff368b4093c8b4ef34df7 --- M be/src/exec/hdfs-scan-node-base.cc 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/10672/2 -- To view, visit http://gerrit.cloudera.org:8080/10672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id2a738edea80ff3fb13ff368b4093c8b4ef34df7 Gerrit-Change-Number: 10672 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](2.x) Revert "IMPALA-7156: temporary revert to older Sentry version"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10689 ) Change subject: Revert "IMPALA-7156: temporary revert to older Sentry version" .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I908a91f607f0a9ce8cb972ea8673771665ad88db Gerrit-Change-Number: 10689 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 12 Jun 2018 06:25:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2634/ -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 12 Jun 2018 05:42:30 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Revert "IMPALA-7156: temporary revert to older Sentry version"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10689 ) Change subject: Revert "IMPALA-7156: temporary revert to older Sentry version" .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2636/ -- To view, visit http://gerrit.cloudera.org:8080/10689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I908a91f607f0a9ce8cb972ea8673771665ad88db Gerrit-Change-Number: 10689 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 12 Jun 2018 05:42:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5216: Make admission control queuing async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10060 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 18: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/41/ -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 18 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 12 Jun 2018 04:28:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5216: Make admission control queuing async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10060 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 19: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 19 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 12 Jun 2018 04:28:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5216: Make admission control queuing async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10060 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 19: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/41/ -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 19 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 12 Jun 2018 04:28:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5216: Make admission control queuing async
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10060 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 18: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 18 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 12 Jun 2018 04:28:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5216: Make admission control queuing async
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10060 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 17: Fixed TSAN and clang-tidy compile warnings. -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 17 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 12 Jun 2018 04:28:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5216: Make admission control queuing async
Tim Armstrong has uploaded a new patch set (#18) to the change originally created by Bikramjeet Vig. ( http://gerrit.cloudera.org:8080/10060 ) Change subject: IMPALA-5216: Make admission control queuing async .. IMPALA-5216: Make admission control queuing async Implement asynchronous admission control queuing. This is achieved by running the admission control code-path in a separate thread. Major changes include: propagating cancellation to the admission control thread and dequeuing thread, and adding a new Query Operation State called "PENDING" that represents the state between completion of planning and starting of query execution. Testing: - Added a deterministic end to end test and a session expiry test. - Ran multiple stress tests successfully with a cancellation probability of 60% and with different values for the following parameters: max_requests, queue_wait_timeout_ms. Ensured that the impalad was in a valid state afterwards (no orphan fragments or wrong metrics). - Ran all exhaustive tests and ASAN core tests successfully. - Ran data load successfully. Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf --- M be/src/common/atomic.h M be/src/common/logging.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/query-state.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M be/src/util/promise-test.cc M be/src/util/promise.h M common/thrift/ImpalaService.thrift M tests/authorization/test_authorization.py M tests/beeswax/impala_beeswax.py M tests/common/impala_connection.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_krpc_mem_usage.py M tests/custom_cluster/test_session_expiration.py M tests/hs2/hs2_test_suite.py M tests/hs2/test_hs2.py M tests/query_test/test_observability.py M www/query_backends.tmpl 30 files changed, 678 insertions(+), 225 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/10060/18 -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 18 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](2.x) Revert "IMPALA-7156: temporary revert to older Sentry version"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10689 ) Change subject: Revert "IMPALA-7156: temporary revert to older Sentry version" .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2635/ -- To view, visit http://gerrit.cloudera.org:8080/10689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I908a91f607f0a9ce8cb972ea8673771665ad88db Gerrit-Change-Number: 10689 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 12 Jun 2018 02:43:38 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Revert "IMPALA-7156: temporary revert to older Sentry version"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10689 ) Change subject: Revert "IMPALA-7156: temporary revert to older Sentry version" .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2635/ -- To view, visit http://gerrit.cloudera.org:8080/10689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I908a91f607f0a9ce8cb972ea8673771665ad88db Gerrit-Change-Number: 10689 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 12 Jun 2018 02:43:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2634/ -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 12 Jun 2018 02:25:31 + Gerrit-HasComments: No
[Impala-ASF-CR] Optimize dependencies for Codegen
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10688 ) Change subject: Optimize dependencies for Codegen .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2633/ -- To view, visit http://gerrit.cloudera.org:8080/10688 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3294cd27c2d35388a04934440a1d2b0ba3a0dd9 Gerrit-Change-Number: 10688 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 12 Jun 2018 02:25:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2634/ -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 12 Jun 2018 02:25:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/10571/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java: http://gerrit.cloudera.org:8080/#/c/10571/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@117 PS5, Line 117: stmt.sqlString_ = null; > Sorry for being unclear. What I was trying to say is that the between behav So removing stmt.sqlString_ = null breaks this test. SQL: select * from functional.alltypes where id = (select 1 + 1)) a Expected :SELECT * FROM (SELECT * FROM functional.alltypes LEFT SEMI JOIN (SELECT 2) `$a$1` (`$c$1`) ON id = `$a$1`.`$c$1`) a Actual: SELECT * FROM (SELECT * FROM functional.alltypes LEFT SEMI JOIN (SELECT 1 + 1) `$a$1` (`$c$1`) ON id = `$a$1`.`$c$1`) a For between predicates, we have a rewritten rule that can't be turned off: https://github.com/apache/impala/blob/06fe3210509424cf50fdd3cc919466a1ce6bc842/fe/src/main/java/org/apache/impala/analysis/Analyzer.java#L322 -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 12 Jun 2018 02:24:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) IMPALA-5931: Generates scan ranges in planner for s3/adls
Hello Impala Public Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10692 to review the following change. Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. IMPALA-5931: Generates scan ranges in planner for s3/adls Currently, for filesystems that do not include physical block information (e.g., block replica locations, caching), synthetic blocks are generated and stored in the catalog when metadata is loaded. Example file systems for which this is done includes S3, ADLS, and local fs. This change avoids generating these blocks when metadata is loaded. Instead, scan ranges are directly generated from such files by the backend coordinator. Previously, all scan ranges were produced by the planner in HDFSScanNode in the frontend. Now, those files without block information are sent to the coordinator represented by a split specification that determines how the coordinator will create scan ranges to send to executors. This change reduces the space needed in the catalog and reduces the scan range data structures that are passed from the frontend to the backend when planning and coordinating a query. In addition a bug is avoided where non-splittable files were being split anyways to support the query parameter that places a limit on scan ranges. Testing: - added backend scheduler tests - mixed-filesystems test covers tables/queries with multiple fs's. - local fs tests cover the code paths in this change - all core tests pass when configured with s3 - manually tried larger local filesystem tables (tpch) with multiple partitions and observed the same scan ranges. - TODO: adls testing Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Reviewed-on: http://gerrit.cloudera.org:8080/8523 Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins --- M CMakeLists.txt M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/util/CMakeLists.txt A be/src/util/flat_buffer.cc A be/src/util/flat_buffer.h M common/thrift/Frontend.thrift M common/thrift/PlanNodes.thrift M common/thrift/Planner.thrift M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 22 files changed, 677 insertions(+), 245 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10692/1 -- To view, visit http://gerrit.cloudera.org:8080/10692 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 10692 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5216: Make admission control queuing async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10060 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 17: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/40/ -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 17 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 12 Jun 2018 01:30:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/10571/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java: http://gerrit.cloudera.org:8080/#/c/10571/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@117 PS5, Line 117: stmt.sqlString_ = null; > Although I agree that "between y and x" is essentially the same as "<= x an Sorry for being unclear. What I was trying to say is that the between behavior doesn't mimic the user queries right now. And it will mimic the user queries if this strange "stmt.sqlString_ = null;" line is removed, so it makes more sense to remove it. Right now for a "BETWEEN" query, impala will print ">= x and <= y" if the query analysis fails, which is bad behavior caused by this strange line which I don't know what it's for. The 3 tests are all broken in the same way AFAIK, which is why I think it would make sense to clean them up in this change. See: https://github.com/apache/impala/blob/06fe3210509424cf50fdd3cc919466a1ce6bc842/fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java#L709 -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 12 Jun 2018 01:20:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5706: Spilling sort optimisations
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9943 ) Change subject: IMPALA-5706: Spilling sort optimisations .. IMPALA-5706: Spilling sort optimisations This patch covers multiple changes with the purpose of optimizing spilling sort mechanism: - Remove the hard-coded maximum limit of buffers that can be used for merging the sorted runs. Instead this number is calculated based on the available memory through buffer pool. - The already sorted runs are distributed more optimally between the last intermediate merge and the final merge to avoid that a heavy intermediate merge is followed by a light final merge. - Right before starting the merging phase Sorter tries to allocate additional memory through the buffer pool. - An output run is not allocated anymore for the final merge. Note, double-buffering the runs during a merge was also planned with this patch. However, performance testing showed that except some exotic queries with unreasonably small amount of buffer pool memory available double-buffering doesn't add to the overall performance. It's basically because the half of the available buffers have to be sacrificed to do double-buffering and as a result the merge tree can get deeper. In addition the amount of I/O wait time is not reaching the level where double-buffering could countervail the reduced number of runs during a particular merge. Performance measurements were made during manual testing to verify that this is in fact an optimization: - In case doing a sort on top of a join when working with a restricted amount of memory then the Sort node successfully allocates additional memory right before the merging phase. This is feasible because once Join finishes sending new input data and calls InputDone() then it releases memory that can be picked up by the Sorter. This results in shallower merging trees (more runs grabbed for a merge). - On a multi-node cluster I verified that in cases when at least one merging step is done then this change reduces the execution time for sorts. - The more merging steps are done the bigger the performance gain is compared to the baseline. Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9 Reviewed-on: http://gerrit.cloudera.org:8080/9943 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/sorter.cc M be/src/runtime/sorter.h M tests/query_test/test_sort.py 3 files changed, 149 insertions(+), 120 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/9943 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9 Gerrit-Change-Number: 9943 Gerrit-PatchSet: 19 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5706: Spilling sort optimisations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9943 ) Change subject: IMPALA-5706: Spilling sort optimisations .. Patch Set 18: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9943 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9 Gerrit-Change-Number: 9943 Gerrit-PatchSet: 18 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 12 Jun 2018 01:15:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Fredy Wijaya has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. IMPALA-7106: Print rewritten SQL correctly when log trace is enabled toSql() method is used to print SQL string that is close to the original SQL string, which is something that users want to see. When debugging issues related to SQL rewrites, it can be very useful to be able to print/get the SQL string that is being rewritten. This patch adds a new method toSql(boolean rewritten) to get the rewritten SQL string. The LOG.trace statement that prints the rewritten SQL is also updated to use toSql(true). Testing: - Added FE test for the rewritten SQL string - Ran all FE tests Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java 9 files changed, 127 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10571/10 -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10571 )
Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is
enabled
..
Patch Set 10:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10571/8/fe/src/main/java/org/apache/impala/analysis/FromClause.java
File fe/src/main/java/org/apache/impala/analysis/FromClause.java:
http://gerrit.cloudera.org:8080/#/c/10571/8/fe/src/main/java/org/apache/impala/analysis/FromClause.java@129
PS8, Line 129:
> Call toSqlString(rewritten)?
Done
http://gerrit.cloudera.org:8080/#/c/10571/8/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/10571/8/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@973
PS8, Line 973: // Where clause
> Call toSqlString(rewritten) directly?
Done
http://gerrit.cloudera.org:8080/#/c/10571/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/10571/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@117
PS5, Line 117: stmt.sqlString_ = null;
> I tried removing it and it broke AnalyzeSubqueriesTest TestExistsSubqueries
Although I agree that "between y and x" is essentially the same as "<= x and >=
y", Isn't it better if the error messages mimic the user queries (with the
exception of case sensitivity and fully qualified names) to not confuse them?
Let me know what you think?
http://gerrit.cloudera.org:8080/#/c/10571/8/fe/src/main/java/org/apache/impala/analysis/TableRef.java
File fe/src/main/java/org/apache/impala/analysis/TableRef.java:
http://gerrit.cloudera.org:8080/#/c/10571/8/fe/src/main/java/org/apache/impala/analysis/TableRef.java@575
PS8, Line 575: protected String tableRefToSql(boolean rewritten) {
> Why this?
The idea is for InlineViewRef to override it. But I can also simplify it by
having tableRefToSql(boolean rewritten) and override it in InlineViewRef.
--
To view, visit http://gerrit.cloudera.org:8080/10571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53
Gerrit-Change-Number: 10571
Gerrit-PatchSet: 10
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Tue, 12 Jun 2018 00:45:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3816: (prep) Move TupleSorter to sorter-ir.cc
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10679 ) Change subject: IMPALA-3816: (prep) Move TupleSorter to sorter-ir.cc .. Patch Set 2: (1 comment) Seems good to me, but we'll have to rebase onto IMPALA-5706. I think there shouldn't be too many conflicts but they both change a lot of lines of code. http://gerrit.cloudera.org:8080/#/c/10679/2/be/src/runtime/sorter-internal.h File be/src/runtime/sorter-internal.h: http://gerrit.cloudera.org:8080/#/c/10679/2/be/src/runtime/sorter-internal.h@88 PS2, Line 88: unitialized "uninitialized" (typo was pre-existing). -- To view, visit http://gerrit.cloudera.org:8080/10679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaaf2b75c2f789002c42939865c018f728d29a113 Gerrit-Change-Number: 10679 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 12 Jun 2018 00:39:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7128 (part 2): add an interface for data sources
Hello Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10626 to look at the new patch set (#3). Change subject: IMPALA-7128 (part 2): add an interface for data sources .. IMPALA-7128 (part 2): add an interface for data sources This changes most of the usage of DataSource and DataSourceTable to use interfaces instead of implementation classes. There are still various usages of the implementation for functionality like creating and dropping data sources. We'll address those as part of IMPALA-7131 at a later date. Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e --- M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/catalog/DataSource.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java A fe/src/main/java/org/apache/impala/catalog/FeDataSource.java A fe/src/main/java/org/apache/impala/catalog/FeDataSourceTable.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java 14 files changed, 110 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/10626/3 -- To view, visit http://gerrit.cloudera.org:8080/10626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e Gerrit-Change-Number: 10626 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7137. Support configuring Frontend to use LocalCatalog
Hello Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10629 to look at the new patch set (#3). Change subject: IMPALA-7137. Support configuring Frontend to use LocalCatalog .. IMPALA-7137. Support configuring Frontend to use LocalCatalog This adds a new flag -use_local_catalog which is passed through to the frontend and causes it to use LocalCatalog instead of ImpaladCatalog. Additionally, when this flag is configured, the impalad does not subscribe to catalog topic updates from the statestore. The patch is slightly more complex than simply picking which class to instantiate, because the lifecycle is designed a bit differently between the two implementations: - LocalCatalog is instantiated once per query/request. - ImpaladCatalog is instantiated once and stateful across queries, except when a full catalog update is received. This maintains the current behavior for this implementation. In order to abstract this difference in lifecycle from the frontend, I introduced a new FeCatalogManager class with different implementations for the two lifecycles. I also had to add a simple test implementation since some tests rely on directly injecting a Catalog implementation into the Frontend. This patch also includes a few small changes to the local catalog implementation objects to enable the impalad to start and accept connections. With this patch, I was able to manually test as follows: I started just the statestore and the impalad in the new mode: - ./bin/start-statestored.sh - ./bin/start-impalad.sh --use_local_catalog I connected with impala-shell as usual and was able to run the most simple queries: - SHOW DATABASES; - USE functional; - SHOW TABLES; All other functionality results in error messages due to the various TODOs in the current skeleton implementation. Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 --- M be/src/runtime/exec-env.cc M be/src/service/impala-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java A fe/src/main/java/org/apache/impala/service/FeCatalogManager.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java 11 files changed, 238 insertions(+), 63 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/10629/3 -- To view, visit http://gerrit.cloudera.org:8080/10629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 Gerrit-Change-Number: 10629 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7135. Skeleton implementation of LocalCatalog
Hello Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10627 to look at the new patch set (#3). Change subject: IMPALA-7135. Skeleton implementation of LocalCatalog .. IMPALA-7135. Skeleton implementation of LocalCatalog This adds some of the high level classes for implementing the local catalog: - LocalCatalog is the top level implementation. The plan is to instantiate this once per query, so that no thread safety is required. - It loads metadata from a MetaProvider interface. The current implementation fetches directly from HMS and provides no caching. A future subtask will add a CachingMetaProvider implementation. Separating out caching will make it easier to experiment with different policies or storage mechanisms. - It instantiates LocalDb and LocalTable objects to implement FeDb and FeTable. These are mostly stubbed out except for the most basic functionality. Functionality will be filled in incrementally in further patches. Since it's not yet possible to hook this up to most of the existing tests, a very simple new unit test is included to cover the bits of functionality that are not stubbed out. I didn't concentrate on too much test coverage here, since once we've implemented more functionality we can switch over all of the existing tests to get coverage of the new implementation. Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java A fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java A fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java A fe/src/main/java/org/apache/impala/catalog/local/LocalCatalogException.java A fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java A fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java A fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java A fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 10 files changed, 794 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/10627/3 -- To view, visit http://gerrit.cloudera.org:8080/10627 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee Gerrit-Change-Number: 10627 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog
Hello Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10630 to look at the new patch set (#3). Change subject: IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog .. IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog This adds support for loading the Table object from HMS and parsing out the Column and partitioning information from it. With this change, I'm able to connect to an impalad running in "local catalog" mode and run DESCRIBE, DESCRIBE EXTENDED, and SHOW CREATE TABLE commands. Other commands like SHOW PARTITIONS don't work properly yet, and type-specific table functionality (eg views, HBase tables, etc) are not yet supported. Again a simple unit test is included to check that column information is loaded. More thorough testing is deferred until we've reached enough coverage that we can start running e2e tests against a cluster running in "local" mode. Change-Id: I640f27e36198955e057da62a3ce25a858406e496 --- A fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalogException.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 8 files changed, 254 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/10630/3 -- To view, visit http://gerrit.cloudera.org:8080/10630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I640f27e36198955e057da62a3ce25a858406e496 Gerrit-Change-Number: 10630 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Hello Impala Public Jenkins, Vuk Ercegovac, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10611 to look at the new patch set (#4). Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition This refactors out interfaces in the frontend for the interaction between the analysis/planning code and the classes that implement these catalog objects. This takes care of the most commonly used objects but defers some others (e.g. functions, cache pools, data sources, etc) to follow-on patches. There are a few spots remaining in the frontend that still downcast to implementation classes, particularly where the frontend actually makes modifications to catalog objects in-place. I left TODOs in those spots and will come back later as necessary. Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableAddDropRangePartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAddPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAddReplaceColsStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetFileFormatStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetRowFormatStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateFunctionStmtBase.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java M fe/src/main/java/org/apache/impala/analysis/DropFunctionStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/PartitionDef.java M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/analysis/PartitionSpecBase.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateFunctionStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java A fe/src/main/java/org/apache/impala/catalog/FeCatalog.java A fe/src/main/java/org/apache/impala/catalog/FeDb.java A fe/src/main/java/org/apache/impala/
[Impala-ASF-CR] IMPALA-7137. Support configuring Frontend to use LocalCatalog
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10629 ) Change subject: IMPALA-7137. Support configuring Frontend to use LocalCatalog .. Patch Set 2: (9 comments) http://gerrit.cloudera.org:8080/#/c/10629/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10629/2//COMMIT_MSG@21 PS2, Line 21: except when a full catalog update is received. > just to clarify, state that this is the behavior today. Done http://gerrit.cloudera.org:8080/#/c/10629/2/be/src/runtime/exec-env.cc File be/src/runtime/exec-env.cc: http://gerrit.cloudera.org:8080/#/c/10629/2/be/src/runtime/exec-env.cc@87 PS2, Line 87: required > nit: used it's also not required to be started. I'll clarify. http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java File fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java: http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@167 PS2, Line 167: name_.toLowerCase() > there have been a number of bugs with inconsistent case handling. just wond in the prior patch in this series I added preconditions to ensure that name_ is always lower case, so I"ll drop this. http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java File fe/src/main/java/org/apache/impala/service/FeCatalogManager.java: http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@30 PS2, Line 30: Managers > nit: Manages Done http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@30 PS2, Line 30: implementations > implementation? I assume its only one at a time. Done http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@126 PS2, Line 126: } > nit: add a newline for consistency Done http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/Frontend.java@168 PS2, Line 168: authzChecker_ > where is this set/used? in the constructor, by the policy reader task. http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/Frontend.java@187 PS2, Line 187: /** :* C'tor used by tests to pass in a custom ImpaladCatalog. :*/ > stale comment? this looks like the core constructor so should it be public? yea, you're right, this can be private since the above two constructors are the ones meant for external consumption. http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/Frontend.java@197 PS2, Line 197: policyReaderTask.run(); > why is this run here? I would think scheduling on L204 would do this. if ne I agree this is a little goofy, though I was trying to maintain the exact old behavior. In the old code, the constructor created the AuthorizationChecker on like 187 with code that duplicated the body of 'AuthorizationPolicyReader.run'. Instead of duplicating it I figured it would be easier to just call 'run' here, which takes care of setting the authzChecker_ value. The scheduling of the task sets a randomized initial delay to reload, but I think we always need to do one load right at startup to ensure that we have the policy loaded before beginning service, right? I was also confused about why it has the check 'if authzConfig_.isEnabled()' below before scheduling the reloader task, but unconditionally does the initial load. -- To view, visit http://gerrit.cloudera.org:8080/10629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 Gerrit-Change-Number: 10629 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 12 Jun 2018 00:25:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6835: Add table name and node id to Kudu scanner errors
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10671 ) Change subject: IMPALA-6835: Add table name and node id to Kudu scanner errors .. Patch Set 2: (2 comments) Looks good, just a couple of style nits. http://gerrit.cloudera.org:8080/#/c/10671/2/be/src/exec/kudu-scanner.h File be/src/exec/kudu-scanner.h: http://gerrit.cloudera.org:8080/#/c/10671/2/be/src/exec/kudu-scanner.h@93 PS2, Line 93: inline nit: inline isn't necessary, this isn't perf-critical code. I think it would be better to put this function definition in the .cc file as well - no reason to add more code to headers when not necessary, it hurts build times. http://gerrit.cloudera.org:8080/#/c/10671/2/be/src/exec/kudu-scanner.h@93 PS2, Line 93: AppendInfo Maybe this could be something more specific like "BuildErrorString"? -- To view, visit http://gerrit.cloudera.org:8080/10671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0377fc8591738dc45092d228fcf292ddbb367825 Gerrit-Change-Number: 10671 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 12 Jun 2018 00:21:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3816: (prep) Move TupleSorter to sorter-ir.cc
Tianyi Wang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/10679 ) Change subject: IMPALA-3816: (prep) Move TupleSorter to sorter-ir.cc .. IMPALA-3816: (prep) Move TupleSorter to sorter-ir.cc To inline calls to Compare() in TupleSorter, we need to cross compile TupleSorter to LLVM-IR first. This patch also adopted some clang-tidy suggestions including using nullptr. Change-Id: Iaaf2b75c2f789002c42939865c018f728d29a113 --- M be/src/codegen/impala-ir.cc M be/src/runtime/CMakeLists.txt A be/src/runtime/sorter-internal.h A be/src/runtime/sorter-ir.cc M be/src/runtime/sorter.cc 5 files changed, 914 insertions(+), 826 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/10679/2 -- To view, visit http://gerrit.cloudera.org:8080/10679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaaf2b75c2f789002c42939865c018f728d29a113 Gerrit-Change-Number: 10679 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-3816, IMPALA-4065: Remove the indirection to TupleRowComparator::Compare()
Tianyi Wang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/10680 ) Change subject: IMPALA-3816, IMPALA-4065: Remove the indirection to TupleRowComparator::Compare() .. IMPALA-3816, IMPALA-4065: Remove the indirection to TupleRowComparator::Compare() This patch removes the indirection of codegened TupleRowComparator:: Compare() at its call sites in sorter and topn node. It's implemented by cloning and modifying all the functions between the codegened entry function and Compare(). The call site in SortedRunMerger is still indirect. Change-Id: If4657ac09daf20408797856d94521d417d8cf171 --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/exchange-node.cc M be/src/exec/topn-node.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/sorted-run-merger.cc M be/src/runtime/sorter-internal.h M be/src/runtime/sorter.cc M be/src/util/tuple-row-compare.cc M be/src/util/tuple-row-compare.h A tests/custom_cluster/test_replace_tuple_row_compare.py 12 files changed, 414 insertions(+), 224 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/10680/2 -- To view, visit http://gerrit.cloudera.org:8080/10680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If4657ac09daf20408797856d94521d417d8cf171 Gerrit-Change-Number: 10680 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-7135. Skeleton implementation of LocalCatalog
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10627 ) Change subject: IMPALA-7135. Skeleton implementation of LocalCatalog .. Patch Set 2: (12 comments) http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java@49 PS2, Line 49: : new MetaStor > nit: single line Done http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java File fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java: http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java@53 PS2, Line 53: which does not interact with catalogd > Although it could, if a hypothetical CatalogdMetaProvider is implemented (a good point. I'll rephrase this to say that this implements a "pull-based" catalog design -- the source might be the direct metadata sources, or an intermediate node like catalogd. http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java@72 PS2, Line 72: metaProvider > precondition to assert non-null. Done http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java@76 PS2, Line 76: matcher > where's this used? woops, good catch! copy paste fail. Fixed and added tests. http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java@88 PS2, Line 88: list > nit: names (seems more intuitive) Done http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java File fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java: http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@44 PS2, Line 44: name_ > lowercase? added docs http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@53 PS2, Line 53: catalog, String dbName > precondition for non-null Done http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@71 PS2, Line 71: e); > nit: pull up to prev. line Done http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@85 PS2, Line 85: assert tb > use precondition, and check non-null. Done http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@122 PS2, Line 122: c > nit: C Done http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java: http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java@42 PS2, Line 42: name_ > lowercase? Done http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java: http://gerrit.cloudera.org:8080/#/c/10627/2/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java@33 PS2, Line 33: * or may include caching, etc. > so far, this covers a portion of hms. pls add a todo for covering other hms Done -- To view, visit http://gerrit.cloudera.org:8080/10627 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee Gerrit-Change-Number: 10627 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 12 Jun 2018 00:14:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10611/3/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/10611/3/fe/src/main/java/org/apache/impala/service/Frontend.java@723 PS3, Line 723: HdfsTable > FeFsTable? Done -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 12 Jun 2018 00:13:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10611/3/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/10611/3/fe/src/main/java/org/apache/impala/service/Frontend.java@723 PS3, Line 723: HdfsTable FeFsTable? -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 12 Jun 2018 00:00:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) Revert "IMPALA-7156: temporary revert to older Sentry version"
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10689 ) Change subject: Revert "IMPALA-7156: temporary revert to older Sentry version" .. Patch Set 1: Trying to run the Jenkins build: https://jenkins.impala.io/job/gerrit-verify-dryrun-external/170/ -- To view, visit http://gerrit.cloudera.org:8080/10689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I908a91f607f0a9ce8cb972ea8673771665ad88db Gerrit-Change-Number: 10689 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Comment-Date: Mon, 11 Jun 2018 23:53:16 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Revert "IMPALA-7156: temporary revert to older Sentry version"
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10689 Change subject: Revert "IMPALA-7156: temporary revert to older Sentry version" .. Revert "IMPALA-7156: temporary revert to older Sentry version" This reverts commit 908865a7e7ce38e9356057f5e0a7ab75e30fb2bc. Change-Id: I908a91f607f0a9ce8cb972ea8673771665ad88db --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/10689/1 -- To view, visit http://gerrit.cloudera.org:8080/10689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I908a91f607f0a9ce8cb972ea8673771665ad88db Gerrit-Change-Number: 10689 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-6835: Add table name and node id to Kudu scanner errors
Pooja Nilangekar has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/10671 ) Change subject: IMPALA-6835: Add table name and node id to Kudu scanner errors .. IMPALA-6835: Add table name and node id to Kudu scanner errors Previously, the error messages in KuduScanner only contained the reason for failure. They did not contain the KuduTable name or the TPlanNode id which made it inconveient to debug. This change adds the TPlanNode id to all error messages and the KuduTable name whenever applicable. This change was manually tested by explicitly returning failure while scanning kudu tables. Change-Id: I0377fc8591738dc45092d228fcf292ddbb367825 --- M be/src/exec/kudu-scanner.cc M be/src/exec/kudu-scanner.h M be/src/exec/kudu-util.h 3 files changed, 26 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10671/2 -- To view, visit http://gerrit.cloudera.org:8080/10671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0377fc8591738dc45092d228fcf292ddbb367825 Gerrit-Change-Number: 10671 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7128 (part 2): add an interface for data sources
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10626 ) Change subject: IMPALA-7128 (part 2): add an interface for data sources .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/10626/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10626/2//COMMIT_MSG@12 PS2, Line 12: IMPALA-7131 > pls mark these with todo's that reference this jira. Done http://gerrit.cloudera.org:8080/#/c/10626/2/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java File fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java: http://gerrit.cloudera.org:8080/#/c/10626/2/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@42 PS2, Line 42: Represents a table backed by an external data source. > pull this part up into the interface. Done http://gerrit.cloudera.org:8080/#/c/10626/2/fe/src/main/java/org/apache/impala/catalog/FeDataSource.java File fe/src/main/java/org/apache/impala/catalog/FeDataSource.java: http://gerrit.cloudera.org:8080/#/c/10626/2/fe/src/main/java/org/apache/impala/catalog/FeDataSource.java@21 PS2, Line 21: * Interface for interacting with data sources from the frontend. > nit: use consistent phrasing with the other interfaces Done -- To view, visit http://gerrit.cloudera.org:8080/10626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e Gerrit-Change-Number: 10626 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 11 Jun 2018 23:34:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding bit width
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10683 ) Change subject: IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding bit width .. IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding bit width Reading dictionary encoded Parquet data pages where the bit width is larger than the encoded type's size (e.g. coding 8 bit TINYINT with 16 bit dictionary indices) led to DCHECK error in debug builds. Impala does not create such parquet files (an N bit type can have maximum 2^N distinct values, so N bit dictionary indices are enough for a dictionary that contains every possible value), but the Parquet standard does not forbid to do so. These DCHECKs were probably introduced by a copy paste error (similar checks exist in the non-dictionary encoded bit reader functions, where they are valid). Testing: - a new test is added to check that these data pages can be decoded correctly Change-Id: I9ff3b00cbcab09dec11b3607d7d9a9c2c0025e1a Reviewed-on: http://gerrit.cloudera.org:8080/10683 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/util/bit-packing.inline.h M be/src/util/bit-stream-utils.inline.h M testdata/data/README A testdata/data/dict_encoding_with_large_bit_width.parquet M tests/query_test/test_scanners.py 5 files changed, 21 insertions(+), 3 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9ff3b00cbcab09dec11b3607d7d9a9c2c0025e1a Gerrit-Change-Number: 10683 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Optimize dependencies for Codegen
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10688 ) Change subject: Optimize dependencies for Codegen .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2633/ -- To view, visit http://gerrit.cloudera.org:8080/10688 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3294cd27c2d35388a04934440a1d2b0ba3a0dd9 Gerrit-Change-Number: 10688 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 11 Jun 2018 23:20:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding bit width
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10683 ) Change subject: IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding bit width .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ff3b00cbcab09dec11b3607d7d9a9c2c0025e1a Gerrit-Change-Number: 10683 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 23:19:57 + Gerrit-HasComments: No
[Impala-ASF-CR] Optimize dependencies for Codegen
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10688 ) Change subject: Optimize dependencies for Codegen .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2633/ -- To view, visit http://gerrit.cloudera.org:8080/10688 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3294cd27c2d35388a04934440a1d2b0ba3a0dd9 Gerrit-Change-Number: 10688 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 11 Jun 2018 23:20:01 + Gerrit-HasComments: No
[Impala-ASF-CR] Optimize dependencies for Codegen
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10688 Change subject: Optimize dependencies for Codegen .. Optimize dependencies for Codegen Pieces of the Codegen library (impala-(no-)sse-ir.cc) depend on Exec, Exprs, Runtime, Udf, and Util. These extra dependencies prevent Codegen from being built in parallel to these other components. This extends the build, especially on systems using high parallelism such as distcc. As far as I can tell, these dependencies are not necessary. This replaces those dependencies with gen-deps. Testing: - I forced all targets other than Codegen to wait 5 minutes after gen-deps completes to start building. This forces Codegen to build without any of the currently listed dependencies. It built successfully and I made sure that the following files were identical to a build without this change: - be/generated-sources/impala-ir/* - llvm-ir/* - I ran the core tests, and they were clean except for known unrelated issues. Change-Id: Ie3294cd27c2d35388a04934440a1d2b0ba3a0dd9 --- M be/src/codegen/CMakeLists.txt 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/10688/1 -- To view, visit http://gerrit.cloudera.org:8080/10688 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie3294cd27c2d35388a04934440a1d2b0ba3a0dd9 Gerrit-Change-Number: 10688 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-5216: Make admission control queuing async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10060 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/40/ -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 17 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 21:50:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5216: Make admission control queuing async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10060 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 17 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 21:50:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5216: Make admission control queuing async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10060 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 16: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/40/ -- To view, visit http://gerrit.cloudera.org:8080/10060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10060 Gerrit-PatchSet: 16 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 21:50:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5706: Spilling sort optimisations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9943 ) Change subject: IMPALA-5706: Spilling sort optimisations .. Patch Set 18: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/39/ -- To view, visit http://gerrit.cloudera.org:8080/9943 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9 Gerrit-Change-Number: 9943 Gerrit-PatchSet: 18 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 21:49:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5706: Spilling sort optimisations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9943 ) Change subject: IMPALA-5706: Spilling sort optimisations .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/39/ -- To view, visit http://gerrit.cloudera.org:8080/9943 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9 Gerrit-Change-Number: 9943 Gerrit-PatchSet: 17 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 21:49:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5706: Spilling sort optimisations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9943 ) Change subject: IMPALA-5706: Spilling sort optimisations .. Patch Set 18: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9943 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9 Gerrit-Change-Number: 9943 Gerrit-PatchSet: 18 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 21:49:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5706: Spilling sort optimisations
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9943 ) Change subject: IMPALA-5706: Spilling sort optimisations .. Patch Set 17: Let's get this one in. -- To view, visit http://gerrit.cloudera.org:8080/9943 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9 Gerrit-Change-Number: 9943 Gerrit-PatchSet: 17 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 21:49:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6835: Improve Kudu scanner error messages to include the table name and the plan node id
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10671 ) Change subject: IMPALA-6835: Improve Kudu scanner error messages to include the table name and the plan node id .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0377fc8591738dc45092d228fcf292ddbb367825 Gerrit-Change-Number: 10671 Gerrit-PatchSet: 1 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 20:09:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding bit width
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10683 ) Change subject: IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding bit width .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2632/ -- To view, visit http://gerrit.cloudera.org:8080/10683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ff3b00cbcab09dec11b3607d7d9a9c2c0025e1a Gerrit-Change-Number: 10683 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 20:09:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Tianyi Wang has posted comments on this change. (
http://gerrit.cloudera.org:8080/10571 )
Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is
enabled
..
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10571/8/fe/src/main/java/org/apache/impala/analysis/FromClause.java
File fe/src/main/java/org/apache/impala/analysis/FromClause.java:
http://gerrit.cloudera.org:8080/#/c/10571/8/fe/src/main/java/org/apache/impala/analysis/FromClause.java@129
PS8, Line 129: tableRefs_.get(i).toSql());
Call toSqlString(rewritten)?
http://gerrit.cloudera.org:8080/#/c/10571/8/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/10571/8/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@973
PS8, Line 973: strBuilder.append((rewritten) ?
fromClause_.toRewrittenSql() : fromClause_.toSql());
Call toSqlString(rewritten) directly?
http://gerrit.cloudera.org:8080/#/c/10571/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/10571/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@117
PS5, Line 117: stmt.sqlString_ = null;
> I did try removing this line and it didn't work. It also broke other tests.
I tried removing it and it broke AnalyzeSubqueriesTest TestExistsSubqueries and
ExprRewriterTest. All the broken cases have something to do with rewriting
"between y and x" into "<= x and >= y". I think we just need to fix those
tests, which seems pretty easy. The resulting string is "closer" to the
original query. And the code in StmtRewriter will be less confusing.
http://gerrit.cloudera.org:8080/#/c/10571/8/fe/src/main/java/org/apache/impala/analysis/TableRef.java
File fe/src/main/java/org/apache/impala/analysis/TableRef.java:
http://gerrit.cloudera.org:8080/#/c/10571/8/fe/src/main/java/org/apache/impala/analysis/TableRef.java@575
PS8, Line 575: protected String tableRefToRewrittenSql() {
Why this?
--
To view, visit http://gerrit.cloudera.org:8080/10571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53
Gerrit-Change-Number: 10571
Gerrit-PatchSet: 8
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Mon, 11 Jun 2018 19:11:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10687 ) Change subject: IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from .. IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from Change-Id: I706e2aa6791c88e41a2717c30f64619e7082c3d4 Reviewed-on: http://gerrit.cloudera.org:8080/10687 Reviewed-by: Sailesh Mukil Tested-by: Alex Rodoni --- M docs/changelog-3.0.html 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Sailesh Mukil: Looks good to me, approved Alex Rodoni: Verified -- To view, visit http://gerrit.cloudera.org:8080/10687 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: merged Gerrit-Change-Id: I706e2aa6791c88e41a2717c30f64619e7082c3d4 Gerrit-Change-Number: 10687 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR](asf-site) IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10687 ) Change subject: IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10687 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I706e2aa6791c88e41a2717c30f64619e7082c3d4 Gerrit-Change-Number: 10687 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Mon, 11 Jun 2018 19:06:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6835: Improve Kudu scanner error messages to include the table name and the plan node id
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/10671 )
Change subject: IMPALA-6835: Improve Kudu scanner error messages to include the
table name and the plan node id
..
Patch Set 1:
(3 comments)
Just some minor comments here.
http://gerrit.cloudera.org:8080/#/c/10671/1/be/src/exec/kudu-scanner.cc
File be/src/exec/kudu-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/10671/1/be/src/exec/kudu-scanner.cc@151
PS1, Line 151: PlanNode
I don't think we use the "PlanNode" terminology in user-facing error messages
yet. In other places we say things like "node with id 10", so would be better
to be consistent with that.
http://gerrit.cloudera.org:8080/#/c/10671/1/be/src/exec/kudu-scanner.cc@151
PS1, Line 151: KuduTable
Maybe "Kudu table"? That seems more human readable.
http://gerrit.cloudera.org:8080/#/c/10671/1/be/src/exec/kudu-scanner.cc@245
PS1, Line 245: Substitute("Unable to open scanner for PlanNode '$0'
with KuduTable '$1'",
We're depending on the fact that KUDU_RETURN_IF_ERROR lazily evaluates it's
second argument, right? Otherwise we'd be constructing a lot of unnecessary
strings.
If so, can you add a comment to KUDU_RETURN_IF_ERROR to mention this behaviour,
since it's something that other code depends on.
--
To view, visit http://gerrit.cloudera.org:8080/10671
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0377fc8591738dc45092d228fcf292ddbb367825
Gerrit-Change-Number: 10671
Gerrit-PatchSet: 1
Gerrit-Owner: Pooja Nilangekar
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Pooja Nilangekar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 11 Jun 2018 19:00:16 +
Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/10687 ) Change subject: IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10687 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I706e2aa6791c88e41a2717c30f64619e7082c3d4 Gerrit-Change-Number: 10687 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Mon, 11 Jun 2018 18:59:49 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10687 ) Change subject: IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10687/1/docs/changelog-3.0.html File docs/changelog-3.0.html: http://gerrit.cloudera.org:8080/#/c/10687/1/docs/changelog-3.0.html@8 PS1, Line 8: The changes in this log are in comparison to Impala 2.11. > nit: Since this is a non-standard message, let's format it to italics. Done -- To view, visit http://gerrit.cloudera.org:8080/10687 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I706e2aa6791c88e41a2717c30f64619e7082c3d4 Gerrit-Change-Number: 10687 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Mon, 11 Jun 2018 18:58:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from
Hello Lars Volker, Michael Brown, Sailesh Mukil, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10687 to look at the new patch set (#2). Change subject: IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from .. IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from Change-Id: I706e2aa6791c88e41a2717c30f64619e7082c3d4 --- M docs/changelog-3.0.html 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/10687/2 -- To view, visit http://gerrit.cloudera.org:8080/10687 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: newpatchset Gerrit-Change-Id: I706e2aa6791c88e41a2717c30f64619e7082c3d4 Gerrit-Change-Number: 10687 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR](asf-site) IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/10687 ) Change subject: IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10687/1/docs/changelog-3.0.html File docs/changelog-3.0.html: http://gerrit.cloudera.org:8080/#/c/10687/1/docs/changelog-3.0.html@8 PS1, Line 8: The changes in this log are in comparison to Impala 2.11. nit: Since this is a non-standard message, let's format it to italics. -- To view, visit http://gerrit.cloudera.org:8080/10687 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I706e2aa6791c88e41a2717c30f64619e7082c3d4 Gerrit-Change-Number: 10687 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Mon, 11 Jun 2018 18:51:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10687 Change subject: IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from .. IMPALA-7005: [DOCS] Noted which version the 3.0 changelog generated from Change-Id: I706e2aa6791c88e41a2717c30f64619e7082c3d4 --- M docs/changelog-3.0.html 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/10687/1 -- To view, visit http://gerrit.cloudera.org:8080/10687 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: newchange Gerrit-Change-Id: I706e2aa6791c88e41a2717c30f64619e7082c3d4 Gerrit-Change-Number: 10687 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7158: Fix HdfsScanNodeBase::progress 's init
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10672 ) Change subject: IMPALA-7158: Fix HdfsScanNodeBase::progress_'s init .. Patch Set 1: Please quickly mention how you tested the change. (maybe you looked at the logs?) -- To view, visit http://gerrit.cloudera.org:8080/10672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2a738edea80ff3fb13ff368b4093c8b4ef34df7 Gerrit-Change-Number: 10672 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 18:47:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7145: fix leak of OpenSSL context when spilling
Sailesh Mukil has posted comments on this change. (
http://gerrit.cloudera.org:8080/10666 )
Change subject: IMPALA-7145: fix leak of OpenSSL context when spilling
..
Patch Set 6:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10666/6/be/src/util/asan.h
File be/src/util/asan.h:
http://gerrit.cloudera.org:8080/#/c/10666/6/be/src/util/asan.h@a32
PS6, Line 32:
Could you add a comment to the commit message as to why you removed this?
http://gerrit.cloudera.org:8080/#/c/10666/6/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:
http://gerrit.cloudera.org:8080/#/c/10666/6/be/src/util/openssl-util.cc@96
PS6, Line 96: EVPCipherCtx
We usually prepend the names of these kinds of classes/structs with 'scoped'.
So renaming to ScopedEVPCipherCtx would be better.
http://gerrit.cloudera.org:8080/#/c/10666/6/be/src/util/openssl-util.cc@97
PS6, Line 97: EVPCipherCtx(int padding) {
Make 'explicit' to avoid implicit conversions.
http://gerrit.cloudera.org:8080/#/c/10666/6/be/src/util/openssl-util.cc@171
PS6, Line 171: ctx.ctx
nit: Maybe it would be cleaner to add a getter() to the newly added struct, to
get the context?
Your call though.
--
To view, visit http://gerrit.cloudera.org:8080/10666
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I98760ed8f31b18b489a156f945c29c95c9bf3184
Gerrit-Change-Number: 10666
Gerrit-PatchSet: 6
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Comment-Date: Mon, 11 Jun 2018 18:37:10 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10669 ) Change subject: IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0445950385fa6160764feaed9a993fa0e59b242 Gerrit-Change-Number: 10669 Gerrit-PatchSet: 2 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Mon, 11 Jun 2018 18:17:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance
Sailesh Mukil has posted comments on this change. (
http://gerrit.cloudera.org:8080/10669 )
Change subject: IMPALA-7157: Avoid unnecessarily pretty printing profiles per
fragment instance
..
Patch Set 1: Code-Review+1
(2 comments)
Thanks for the reviews.
Carry +1.
http://gerrit.cloudera.org:8080/#/c/10669/1/be/src/runtime/fragment-instance-state.cc
File be/src/runtime/fragment-instance-state.cc:
http://gerrit.cloudera.org:8080/#/c/10669/1/be/src/runtime/fragment-instance-state.cc@a371
PS1, Line 371:
:
:
> From git blame, it appears to be added in 368115cda. Probably was useful wh
It's actually from the following commit which is from 6 years ago:
https://github.com/apache/impala/commit/7725f25ff5219fb7440ac92d32802dd4ce3cb8f0
Commit 368115cda just moved it around. So, I feel that it's reasonable to
remove this now.
http://gerrit.cloudera.org:8080/#/c/10669/1/be/src/runtime/fragment-instance-state.cc@368
PS1, Line 368: if (VLOG_FILE_IS_ON) {
> you should just remove that if-stmt now. the VLOG_FILE macro will have the
Done
--
To view, visit http://gerrit.cloudera.org:8080/10669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0445950385fa6160764feaed9a993fa0e59b242
Gerrit-Change-Number: 10669
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Comment-Date: Mon, 11 Jun 2018 18:14:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance
Hello Michael Ho, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10669 to look at the new patch set (#2). Change subject: IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance .. IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance In SendReport(), if VLOG_FILE_IS_ON is 'true' (which is not the most verbose logging level, but is higher than default), we pretty print the profile for every fragment instance, which is a very expensive operation, as serializing the profile is non-trivial (look at RuntimeProfile::PrettyPrint()), and printing large amounts of information to the logs isn't cheap as well. Lastly, it is very noisy. This seems unnecessary since this will not benefit us, as all the profiles are merged at the coordinator side. We could argue that this might be necessary when an executor fails to send the profile to the coordinator, but that signifies a network issue which will not be reflected in the profile of any fragment instance. This will help reduce noise in the logs when the log level is bumped up to find other real issues that VLOG_FILE can help with. Change-Id: Ic0445950385fa6160764feaed9a993fa0e59b242 --- M be/src/runtime/fragment-instance-state.cc 1 file changed, 2 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/10669/2 -- To view, visit http://gerrit.cloudera.org:8080/10669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic0445950385fa6160764feaed9a993fa0e59b242 Gerrit-Change-Number: 10669 Gerrit-PatchSet: 2 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR] IMPALA-7137. Support configuring Frontend to use LocalCatalog
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10629 ) Change subject: IMPALA-7137. Support configuring Frontend to use LocalCatalog .. Patch Set 2: (9 comments) http://gerrit.cloudera.org:8080/#/c/10629/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10629/2//COMMIT_MSG@21 PS2, Line 21: except when a full catalog update is received. just to clarify, state that this is the behavior today. http://gerrit.cloudera.org:8080/#/c/10629/2/be/src/runtime/exec-env.cc File be/src/runtime/exec-env.cc: http://gerrit.cloudera.org:8080/#/c/10629/2/be/src/runtime/exec-env.cc@87 PS2, Line 87: required nit: used http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java File fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java: http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@167 PS2, Line 167: name_.toLowerCase() there have been a number of bugs with inconsistent case handling. just wondering why this tolower was used and whether we can make the choice more explicit. http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java File fe/src/main/java/org/apache/impala/service/FeCatalogManager.java: http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@30 PS2, Line 30: implementations implementation? I assume its only one at a time. http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@30 PS2, Line 30: Managers nit: Manages http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@126 PS2, Line 126: } nit: add a newline for consistency http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/Frontend.java@168 PS2, Line 168: authzChecker_ where is this set/used? http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/Frontend.java@187 PS2, Line 187: /** :* C'tor used by tests to pass in a custom ImpaladCatalog. :*/ stale comment? this looks like the core constructor so should it be public? looks like the standard entry point is L173 whereas for tests, we should use L182. I don't think this was clear before... the construction in JniFrontend used L174 and then used this "test" constructor. http://gerrit.cloudera.org:8080/#/c/10629/2/fe/src/main/java/org/apache/impala/service/Frontend.java@197 PS2, Line 197: policyReaderTask.run(); why is this run here? I would think scheduling on L204 would do this. if needed, perhaps it can be run in an else clause of L199. also, since authzChecker_ is not set here, something looks off. -- To view, visit http://gerrit.cloudera.org:8080/10629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 Gerrit-Change-Number: 10629 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 11 Jun 2018 18:03:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6835: Improve Kudu scanner error messages to include the table name and the plan node id
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/10671 )
Change subject: IMPALA-6835: Improve Kudu scanner error messages to include the
table name and the plan node id
..
Patch Set 1:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/10671/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10671/1//COMMIT_MSG@7
PS1, Line 7: IMPALA-6835: Improve Kudu scanner error messages to include the
table name and the plan node id
nit: it would be nice to find a shorter title without loosing too much
information - e.g. "Add table name and node id to Kudu scanner errors"
http://gerrit.cloudera.org:8080/#/c/10671/1//COMMIT_MSG@11
PS1, Line 11: TPlanNode id which made it inconveient to debug. This change add
nit: missing s
http://gerrit.cloudera.org:8080/#/c/10671/1/be/src/exec/kudu-scanner.cc
File be/src/exec/kudu-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/10671/1/be/src/exec/kudu-scanner.cc@151
PS1, Line 151: Substitute("Unable to deserialize scan token for PlanNode
'$0' with KuduTable '$1'",
:scan_node_->id(),
scan_node_->table_->name()));
This made me think about ways to make it less verbose - a function like "string
KuduScanner::AppendInfo(const char* msg)" could be created to add the node id
and table name to a message. It is up to you, but I think that it's worth to
add +1 function to make the code a bit shorter and ensure consistent messages.
http://gerrit.cloudera.org:8080/#/c/10671/1/be/src/exec/kudu-scanner.cc@167
PS1, Line 167: scan_node_->id()
I am not sure about this, but I would add table name for these errors too.
http://gerrit.cloudera.org:8080/#/c/10671/1/be/src/exec/kudu-scanner.cc@168
PS1, Line 168: VLOG_ROW << "Starting KuduScanner with ReadMode=" << mode << "
timeout=" <<
It may be useful do add the same info to logs.
--
To view, visit http://gerrit.cloudera.org:8080/10671
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0377fc8591738dc45092d228fcf292ddbb367825
Gerrit-Change-Number: 10671
Gerrit-PatchSet: 1
Gerrit-Owner: Pooja Nilangekar
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Pooja Nilangekar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 11 Jun 2018 17:44:52 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3307: Add support for IANA time-zone db
Attila Jeges has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/9986 ) Change subject: IMPALA-3307: Add support for IANA time-zone db .. IMPALA-3307: Add support for IANA time-zone db Impala currently uses two different libraries for timestamp manipulations: boost and glibc. Issues with boost: - Time-zone database is currently hard coded in timezone_db.cc. Impala admins cannot update it without upgrading Impala. - Time-zone database is flat, therefore can’t track year-to-year changes. - Time-zone database is not updated on a regular basis. Issues with glibc: - Uses /usr/share/zoneinfo/ database which could be out of sync on some of the nodes in the Impala cluster. - Uses the host system’s local time-zone. Different nodes in the Impala cluster might use a different local time-zone. - Conversion functions take a global lock, which causes severe performance degradation. In addition to the issues above, the fact that /usr/share/zoneinfo/ and the hard-coded boost time-zone database are both in use is a source of inconsistency in itself. This patch makes the following changes: - Instead of boost and glibc, impalad uses Google's CCTZ to implement time-zone conversions. - Introduces a new startup flag (--hdfs_zone_info_zip) to impalad to specify an HDFS/S3/ADLS path to a zip archive that contains the shared compiled IANA time-zone database. If the startup flag is set, impalad will use the specified time-zone database. Otherwise, impalad will use the default /usr/share/zoneinfo time-zone database. - Introduces a new startup flag (--hdfs_zone_alias_conf) to impalad to specify an HDFS/S3/ADLS path to a shared config file that contains definitions for non-standard time-zone aliases. - impalad reads the entire time-zone database into an in-memory map on startup for fast lookups. - The name of the coordinator node’s local time-zone is saved to the query context when preparing query execution. This time-zone is used whenever the current time-zone is referred afterwards in an execution node. - Adds a new ZipUtil class to extract files from a zip archive. The implementation is not vulnerable to Zip Slip. Cherry-picks: not for 2.x. Change-Id: I93c1fbffe81f067919706e30db0a34d0e58e7e77 --- M CMakeLists.txt M be/CMakeLists.txt M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/benchmarks/CMakeLists.txt A be/src/benchmarks/convert-timestamp-benchmark.cc M be/src/common/global-types.h M be/src/common/init.cc M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/parquet-column-readers.cc M be/src/exprs/CMakeLists.txt M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/cast-functions-ir.cc M be/src/exprs/decimal-operators-ir.cc M be/src/exprs/decimal-operators.h M be/src/exprs/expr-test.cc M be/src/exprs/literal.cc M be/src/exprs/timestamp-functions-ir.cc M be/src/exprs/timestamp-functions.cc A be/src/exprs/timezone_db-test.cc M be/src/exprs/timezone_db.cc M be/src/exprs/timezone_db.h M be/src/runtime/raw-value-test.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.cc M be/src/runtime/timestamp-value.h M be/src/runtime/timestamp-value.inline.h M be/src/service/frontend.cc M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M be/src/util/CMakeLists.txt M be/src/util/filesystem-util-test.cc M be/src/util/filesystem-util.cc M be/src/util/filesystem-util.h M be/src/util/hdfs-util-test.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M be/src/util/time-test.cc M be/src/util/time.cc M be/src/util/time.h A be/src/util/zip-util-test.cc A be/src/util/zip-util.cc A be/src/util/zip-util.h M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/rat_exclude_files.txt A cmake_modules/FindCctz.cmake M common/thrift/CMakeLists.txt M common/thrift/ImpalaInternalService.thrift A common/thrift/Zip.thrift M common/thrift/metrics.json A fe/src/main/java/org/apache/impala/util/ZipUtil.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M testdata/bin/create-load-data.sh M testdata/data/timezoneverification.csv A testdata/tzdb/2017c-corrupt.zip A testdata/tzdb/2017c.zip A testdata/tzdb/alias.conf A testdata/tzdb_tiny/America/New_York A testdata/tzdb_tiny/Etc/GMT+4 A testdata/tzdb_tiny/US/Eastern A testdata/tzdb_tiny/UTC A testdata/tzdb_tiny/Zulu A testdata/tzdb_tiny/posix/UTC A testdata/tzdb_tiny/posixrules M testdata/workloads/functional-query/queries/QueryTest/exprs.test M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py A tests/custom_cluster/test_shared_tzdb.py D tests/query_test/test_timezones.py 72 files changed, 3,089 insertions(+), 1,167 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/9986/18 -- To view, visit http://gerrit.cloudera.org:8080/9986 To unsubscribe
[Impala-ASF-CR] IMPALA-3307: Add support for IANA time-zone db
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/9986 ) Change subject: IMPALA-3307: Add support for IANA time-zone db .. Patch Set 17: > Uploaded patch set 17: Commit message was updated. Added support for configurable timezone aliases (instead of just abbreviations). -- To view, visit http://gerrit.cloudera.org:8080/9986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93c1fbffe81f067919706e30db0a34d0e58e7e77 Gerrit-Change-Number: 9986 Gerrit-PatchSet: 17 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Jun 2018 17:37:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3307: Add support for IANA time-zone db
Attila Jeges has uploaded a new patch set (#17). ( http://gerrit.cloudera.org:8080/9986 ) Change subject: IMPALA-3307: Add support for IANA time-zone db .. IMPALA-3307: Add support for IANA time-zone db Impala currently uses two different libraries for timestamp manipulations: boost and glibc. Issues with boost: - Time-zone database is currently hard coded in timezone_db.cc. Impala admins cannot update it without upgrading Impala. - Time-zone database is flat, therefore can’t track year-to-year changes. - Time-zone database is not updated on a regular basis. Issues with glibc: - Uses /usr/share/zoneinfo/ database which could be out of sync on some of the nodes in the Impala cluster. - Uses the host system’s local time-zone. Different nodes in the Impala cluster might use a different local time-zone. - Conversion functions take a global lock, which causes severe performance degradation. In addition to the issues above, the fact that /usr/share/zoneinfo/ and the hard-coded boost time-zone database are both in use is a source of inconsistency in itself. This patch makes the following changes: - Instead of boost and glibc, impalad uses Google's CCTZ to implement time-zone conversions. - Introduces a new startup flag (--hdfs_zone_info_zip) to impalad to specify an HDFS/S3/ADLS path to a zip archive that contains the shared compiled IANA time-zone database. If the startup flag is set, impalad will use the specified time-zone database. Otherwise, impalad will use the default /usr/share/zoneinfo time-zone database. - Introduces a new startup flag (--hdfs_zone_alias_conf) to impalad to specify an HDFS/S3/ADLS path to a shared config file that contains definitions for non-standard time-zone aliases. - impalad reads the entire time-zone database into an in-memory map on startup for fast lookups. - The name of the coordinator node’s local time-zone is saved to the query context when preparing query execution. This time-zone is used whenever the current time-zone is referred afterwards in an execution node. - Adds a new ZipUtil class to extract files from a zip archive. The implementation is not vulnerable to Zip Slip. Cherry-picks: not for 2.x. Change-Id: I93c1fbffe81f067919706e30db0a34d0e58e7e77 --- M CMakeLists.txt M be/CMakeLists.txt M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/benchmarks/CMakeLists.txt A be/src/benchmarks/convert-timestamp-benchmark.cc M be/src/common/global-types.h M be/src/common/init.cc M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/parquet-column-readers.cc M be/src/exprs/CMakeLists.txt M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/cast-functions-ir.cc M be/src/exprs/decimal-operators-ir.cc M be/src/exprs/decimal-operators.h M be/src/exprs/expr-test.cc M be/src/exprs/literal.cc M be/src/exprs/timestamp-functions-ir.cc M be/src/exprs/timestamp-functions.cc A be/src/exprs/timezone_db-test.cc M be/src/exprs/timezone_db.cc M be/src/exprs/timezone_db.h M be/src/runtime/raw-value-test.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.cc M be/src/runtime/timestamp-value.h M be/src/runtime/timestamp-value.inline.h M be/src/service/frontend.cc M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M be/src/util/CMakeLists.txt M be/src/util/filesystem-util-test.cc M be/src/util/filesystem-util.cc M be/src/util/filesystem-util.h M be/src/util/hdfs-util-test.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M be/src/util/time-test.cc M be/src/util/time.cc M be/src/util/time.h A be/src/util/zip-util-test.cc A be/src/util/zip-util.cc A be/src/util/zip-util.h M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/rat_exclude_files.txt A cmake_modules/FindCctz.cmake M common/thrift/CMakeLists.txt M common/thrift/ImpalaInternalService.thrift A common/thrift/Zip.thrift M common/thrift/metrics.json A fe/src/main/java/org/apache/impala/util/ZipUtil.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M testdata/bin/create-load-data.sh M testdata/data/timezoneverification.csv A testdata/tzdb/2017c-corrupt.zip A testdata/tzdb/2017c.zip A testdata/tzdb/alias.conf A testdata/tzdb_tiny/America/New_York A testdata/tzdb_tiny/Etc/GMT+4 A testdata/tzdb_tiny/US/Eastern A testdata/tzdb_tiny/UTC A testdata/tzdb_tiny/Zulu A testdata/tzdb_tiny/posix/UTC A testdata/tzdb_tiny/posixrules M testdata/workloads/functional-query/queries/QueryTest/exprs.test M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py A tests/custom_cluster/test_shared_tzdb.py D tests/query_test/test_timezones.py 72 files changed, 3,089 insertions(+), 1,167 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/9986/17 -- To view, visit http://gerrit.cloudera.org:8080/9986 To unsubscribe
[Impala-ASF-CR] IMPALA-6408: [DOCS] Add a missing info about SHUFFLE
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10685 ) Change subject: IMPALA-6408: [DOCS] Add a missing info about SHUFFLE .. Patch Set 1: Sorry about the diffs caused by the formatting changes! We refactored this doc, a few weeks ago, so the missing and incorrect info are all fixed. -- To view, visit http://gerrit.cloudera.org:8080/10685 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5738557354c384aab983f64722dde5944037aed7 Gerrit-Change-Number: 10685 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Comment-Date: Mon, 11 Jun 2018 17:35:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3307: Add support for IANA time-zone db
Attila Jeges has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/9986 ) Change subject: IMPALA-3307: Add support for IANA time-zone db .. IMPALA-3307: Add support for IANA time-zone db Impala currently uses two different libraries for timestamp manipulations: boost and glibc. Issues with boost: - Time-zone database is currently hard coded in timezone_db.cc. Impala admins cannot update it without upgrading Impala. - Time-zone database is flat, therefore can’t track year-to-year changes. - Time-zone database is not updated on a regular basis. Issues with glibc: - Uses /usr/share/zoneinfo/ database which could be out of sync on some of the nodes in the Impala cluster. - Uses the host system’s local time-zone. Different nodes in the Impala cluster might use a different local time-zone. - Conversion functions take a global lock, which causes severe performance degradation. In addition to the issues above, the fact that /usr/share/zoneinfo/ and the hard-coded boost time-zone database are both in use is a source of inconsistency in itself. This patch makes the following changes: - Instead of boost and glibc, impalad uses Google's CCTZ to implement time-zone conversions. - Introduces a new startup flag (--hdfs_zone_info_zip) to impalad to specify an HDFS/S3/ADLS path to a zip archive that contains the shared compiled IANA time-zone database. If the startup flag is set, impalad will use the specified time-zone database. Otherwise, impalad will use the default /usr/share/zoneinfo time-zone database. - Introduces a new startup flag (--hdfs_zone_abbrev_conf) to impalad to specify an HDFS/S3/ADLS path to a shared config file that contains definitions for non-standard time-zone abbreviations. - impalad reads the entire time-zone database into an in-memory map on startup for fast lookups. - The name of the coordinator node’s local time-zone is saved to the query context when preparing query execution. This time-zone is used whenever the current time-zone is referred afterwards in an execution node. - Adds a new ZipUtil class to extract files from a zip archive. The implementation is not vulnerable to Zip Slip. Cherry-picks: not for 2.x. Change-Id: I93c1fbffe81f067919706e30db0a34d0e58e7e77 --- M CMakeLists.txt M be/CMakeLists.txt M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/benchmarks/CMakeLists.txt A be/src/benchmarks/convert-timestamp-benchmark.cc M be/src/common/global-types.h M be/src/common/init.cc M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/parquet-column-readers.cc M be/src/exprs/CMakeLists.txt M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/cast-functions-ir.cc M be/src/exprs/decimal-operators-ir.cc M be/src/exprs/decimal-operators.h M be/src/exprs/expr-test.cc M be/src/exprs/literal.cc M be/src/exprs/timestamp-functions-ir.cc M be/src/exprs/timestamp-functions.cc A be/src/exprs/timezone_db-test.cc M be/src/exprs/timezone_db.cc M be/src/exprs/timezone_db.h M be/src/runtime/raw-value-test.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.cc M be/src/runtime/timestamp-value.h M be/src/runtime/timestamp-value.inline.h M be/src/service/frontend.cc M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M be/src/util/CMakeLists.txt M be/src/util/filesystem-util-test.cc M be/src/util/filesystem-util.cc M be/src/util/filesystem-util.h M be/src/util/hdfs-util-test.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M be/src/util/time-test.cc M be/src/util/time.cc M be/src/util/time.h A be/src/util/zip-util-test.cc A be/src/util/zip-util.cc A be/src/util/zip-util.h M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/rat_exclude_files.txt A cmake_modules/FindCctz.cmake M common/thrift/CMakeLists.txt M common/thrift/ImpalaInternalService.thrift A common/thrift/Zip.thrift M common/thrift/metrics.json A fe/src/main/java/org/apache/impala/util/ZipUtil.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M testdata/bin/create-load-data.sh M testdata/data/timezoneverification.csv A testdata/tzdb/2017c-corrupt.zip A testdata/tzdb/2017c.zip A testdata/tzdb/alias.conf A testdata/tzdb_tiny/America/New_York A testdata/tzdb_tiny/Etc/GMT+4 A testdata/tzdb_tiny/US/Eastern A testdata/tzdb_tiny/UTC A testdata/tzdb_tiny/Zulu A testdata/tzdb_tiny/posix/UTC A testdata/tzdb_tiny/posixrules M testdata/workloads/functional-query/queries/QueryTest/exprs.test M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py A tests/custom_cluster/test_shared_tzdb.py D tests/query_test/test_timezones.py 72 files changed, 3,089 insertions(+), 1,167 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/9986/16 -- To view, visit http://gerrit.cloudera.org:8080/9986 To unsu
[Impala-ASF-CR] IMPALA-6408: [DOCS] Add a missing info about SHUFFLE
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10685 Change subject: IMPALA-6408: [DOCS] Add a missing info about SHUFFLE .. IMPALA-6408: [DOCS] Add a missing info about SHUFFLE Change-Id: I5738557354c384aab983f64722dde5944037aed7 --- M docs/topics/impala_hints.xml M docs/topics/impala_parquet.xml 2 files changed, 287 insertions(+), 294 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/10685/1 -- To view, visit http://gerrit.cloudera.org:8080/10685 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5738557354c384aab983f64722dde5944037aed7 Gerrit-Change-Number: 10685 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-6835: Improve Kudu scanner error messages to include the table name and the plan node id
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10671 ) Change subject: IMPALA-6835: Improve Kudu scanner error messages to include the table name and the plan node id .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2631/ -- To view, visit http://gerrit.cloudera.org:8080/10671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0377fc8591738dc45092d228fcf292ddbb367825 Gerrit-Change-Number: 10671 Gerrit-PatchSet: 1 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 16:53:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10669 ) Change subject: IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10669/1/be/src/runtime/fragment-instance-state.cc File be/src/runtime/fragment-instance-state.cc: http://gerrit.cloudera.org:8080/#/c/10669/1/be/src/runtime/fragment-instance-state.cc@a371 PS1, Line 371: : : > An alternative we discussed is to bump this log statement to VLOG_ROW() or >From git blame, it appears to be added in 368115cda. Probably was useful when >developing that change. Let's not worry about deleting it if we find it a >nuisance now. -- To view, visit http://gerrit.cloudera.org:8080/10669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0445950385fa6160764feaed9a993fa0e59b242 Gerrit-Change-Number: 10669 Gerrit-PatchSet: 1 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Mon, 11 Jun 2018 16:35:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3307: Add support for IANA time-zone db
Attila Jeges has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/9986 ) Change subject: IMPALA-3307: Add support for IANA time-zone db .. IMPALA-3307: Add support for IANA time-zone db Impala currently uses two different libraries for timestamp manipulations: boost and glibc. Issues with boost: - Time-zone database is currently hard coded in timezone_db.cc. Impala admins cannot update it without upgrading Impala. - Time-zone database is flat, therefore can’t track year-to-year changes. - Time-zone database is not updated on a regular basis. Issues with glibc: - Uses /usr/share/zoneinfo/ database which could be out of sync on some of the nodes in the Impala cluster. - Uses the host system’s local time-zone. Different nodes in the Impala cluster might use a different local time-zone. - Conversion functions take a global lock, which causes severe performance degradation. In addition to the issues above, the fact that /usr/share/zoneinfo/ and the hard-coded boost time-zone database are both in use is a source of inconsistency in itself. This patch makes the following changes: - Instead of boost and glibc, impalad uses Google's CCTZ to implement time-zone conversions. - Introduces a new startup flag (--hdfs_zone_info_zip) to impalad to specify an HDFS/S3/ADLS path to a zip archive that contains the shared compiled IANA time-zone database. If the startup flag is set, impalad will use the specified time-zone database. Otherwise, impalad will use the default /usr/share/zoneinfo time-zone database. - Introduces a new startup flag (--hdfs_zone_abbrev_conf) to impalad to specify an HDFS/S3/ADLS path to a shared config file that contains definitions for non-standard time-zone abbreviations. - impalad reads the entire time-zone database into an in-memory map on startup for fast lookups. - The name of the coordinator node’s local time-zone is saved to the query context when preparing query execution. This time-zone is used whenever the current time-zone is referred afterwards in an execution node. - Adds a new ZipUtil class to extract files from a zip archive. The implementation is not vulnerable to Zip Slip. Cherry-picks: not for 2.x. Change-Id: I93c1fbffe81f067919706e30db0a34d0e58e7e77 --- M CMakeLists.txt M be/CMakeLists.txt M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/benchmarks/CMakeLists.txt A be/src/benchmarks/convert-timestamp-benchmark.cc M be/src/common/global-types.h M be/src/common/init.cc M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/parquet-column-readers.cc M be/src/exprs/CMakeLists.txt M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/cast-functions-ir.cc M be/src/exprs/decimal-operators-ir.cc M be/src/exprs/decimal-operators.h M be/src/exprs/expr-test.cc M be/src/exprs/literal.cc M be/src/exprs/timestamp-functions-ir.cc M be/src/exprs/timestamp-functions.cc A be/src/exprs/timezone_db-test.cc M be/src/exprs/timezone_db.cc M be/src/exprs/timezone_db.h M be/src/runtime/raw-value-test.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.cc M be/src/runtime/timestamp-value.h M be/src/runtime/timestamp-value.inline.h M be/src/service/frontend.cc M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M be/src/util/CMakeLists.txt M be/src/util/filesystem-util-test.cc M be/src/util/filesystem-util.cc M be/src/util/filesystem-util.h M be/src/util/hdfs-util-test.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M be/src/util/time-test.cc M be/src/util/time.cc M be/src/util/time.h A be/src/util/zip-util-test.cc A be/src/util/zip-util.cc A be/src/util/zip-util.h M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/rat_exclude_files.txt A cmake_modules/FindCctz.cmake M common/thrift/CMakeLists.txt M common/thrift/ImpalaInternalService.thrift A common/thrift/Zip.thrift M common/thrift/metrics.json A fe/src/main/java/org/apache/impala/util/ZipUtil.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M testdata/bin/create-load-data.sh M testdata/data/timezoneverification.csv A testdata/tzdb/2017c-corrupt.zip A testdata/tzdb/2017c.zip A testdata/tzdb/abbrev.conf A testdata/tzdb_tiny/America/New_York A testdata/tzdb_tiny/Etc/GMT+4 A testdata/tzdb_tiny/US/Eastern A testdata/tzdb_tiny/UTC A testdata/tzdb_tiny/Zulu A testdata/tzdb_tiny/posix/UTC A testdata/tzdb_tiny/posixrules M testdata/workloads/functional-query/queries/QueryTest/exprs.test M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py A tests/custom_cluster/test_shared_tzdb.py D tests/query_test/test_timezones.py 72 files changed, 3,119 insertions(+), 1,167 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/9986/15 -- To view, visit http://gerrit.cloudera.org:8080/9986 To uns
[Impala-ASF-CR] IMPALA-5392: Added all stack frames to ThreadInfo summary.
Jim Apple has abandoned this change. ( http://gerrit.cloudera.org:8080/10145 ) Change subject: IMPALA-5392: Added all stack frames to ThreadInfo summary. .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/10145 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I80ab4aad03e0c1f01fecad6b87779531244c28b7 Gerrit-Change-Number: 10145 Gerrit-PatchSet: 10 Gerrit-Owner: Abhishek Sharma Gerrit-Reviewer: Abhishek Sharma Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Charles Agnello Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Jim Apple
[Impala-ASF-CR] IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance
Dan Hecht has posted comments on this change. (
http://gerrit.cloudera.org:8080/10669 )
Change subject: IMPALA-7157: Avoid unnecessarily pretty printing profiles per
fragment instance
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10669/1/be/src/runtime/fragment-instance-state.cc
File be/src/runtime/fragment-instance-state.cc:
http://gerrit.cloudera.org:8080/#/c/10669/1/be/src/runtime/fragment-instance-state.cc@368
PS1, Line 368: if (VLOG_FILE_IS_ON) {
you should just remove that if-stmt now. the VLOG_FILE macro will have the same
effect.
--
To view, visit http://gerrit.cloudera.org:8080/10669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0445950385fa6160764feaed9a993fa0e59b242
Gerrit-Change-Number: 10669
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Comment-Date: Mon, 11 Jun 2018 16:25:16 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7145: fix leak of OpenSSL context when spilling
Tim Armstrong has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/10666 ) Change subject: IMPALA-7145: fix leak of OpenSSL context when spilling .. IMPALA-7145: fix leak of OpenSSL context when spilling Add a RAII wrapper for the OpenSSL context that automatically frees on all exit paths from the function. Add a backend test wrapper that enables LeakSanitizer for an individual test. This is a step towards IMPALA-2746. Testing: Enable LeakSanitizer for openssl-util-test. This reliably found the bug. Ran core tests under ASAN. Change-Id: I98760ed8f31b18b489a156f945c29c95c9bf3184 --- M be/CMakeLists.txt M be/src/util/CMakeLists.txt M be/src/util/asan.h M be/src/util/openssl-util.cc 4 files changed, 36 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/10666/6 -- To view, visit http://gerrit.cloudera.org:8080/10666 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I98760ed8f31b18b489a156f945c29c95c9bf3184 Gerrit-Change-Number: 10666 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-2746: part 1: enable LSAN for many backend tests
Tim Armstrong has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/10668 ) Change subject: IMPALA-2746: part 1: enable LSAN for many backend tests .. IMPALA-2746: part 1: enable LSAN for many backend tests This turns on leak sanitizer for backend tests that required relatively small modifications to pass. We suppress a few leaks, mainly related to the embedded JVM. Testing: Ran core tests under ASAN. Change-Id: Ibdda092a4eb4bc827c75a8c121e5428ec746b7f4 --- M be/CMakeLists.txt M be/src/catalog/CMakeLists.txt M be/src/codegen/CMakeLists.txt M be/src/codegen/instruction-counter-test.cc M be/src/common/CMakeLists.txt M be/src/common/atomic-test.cc M be/src/common/atomic.h M be/src/exec/CMakeLists.txt M be/src/experiments/CMakeLists.txt M be/src/exprs/CMakeLists.txt M be/src/exprs/expr-test.cc M be/src/rpc/CMakeLists.txt M be/src/runtime/CMakeLists.txt M be/src/runtime/bufferpool/CMakeLists.txt M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/io/CMakeLists.txt M be/src/runtime/io/disk-io-mgr-stress.cc M be/src/runtime/io/disk-io-mgr-stress.h M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/mem-tracker.cc M be/src/scheduling/CMakeLists.txt M be/src/service/CMakeLists.txt M be/src/statestore/CMakeLists.txt M be/src/udf/udf.cc M be/src/util/CMakeLists.txt M be/src/util/decompress-test.cc A bin/lsan-suppressions.txt 28 files changed, 295 insertions(+), 239 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/10668/5 -- To view, visit http://gerrit.cloudera.org:8080/10668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibdda092a4eb4bc827c75a8c121e5428ec746b7f4 Gerrit-Change-Number: 10668 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7145: fix leak of OpenSSL context when spilling
Tim Armstrong has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/10666 ) Change subject: IMPALA-7145: fix leak of OpenSSL context when spilling .. IMPALA-7145: fix leak of OpenSSL context when spilling Add a RAII wrapper for the OpenSSL context that automatically frees on all exit paths from the function. Add a backend test wrapper that enables LeakSanitizer for an individual test. This is a step towards IMPALA-2746. Testing: Enable LeakSanitizer for openssl-util-test. This reliably found the bug. Ran core tests under ASAN. Change-Id: I98760ed8f31b18b489a156f945c29c95c9bf3184 --- M be/CMakeLists.txt M be/src/util/CMakeLists.txt M be/src/util/asan.h M be/src/util/openssl-util.cc 4 files changed, 35 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/10666/5 -- To view, visit http://gerrit.cloudera.org:8080/10666 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I98760ed8f31b18b489a156f945c29c95c9bf3184 Gerrit-Change-Number: 10666 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7145: fix leak of OpenSSL context when spilling
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10666 Change subject: IMPALA-7145: fix leak of OpenSSL context when spilling .. IMPALA-7145: fix leak of OpenSSL context when spilling Add a RAII wrapper for the OpenSSL context that automatically frees on all exit paths from the function. Add a backend test wrapper that enables LeakSanitizer for an individual test. This is a step towards IMPALA-2746. I enabled util/ tests that passed LeakSanitizer with minimal modifications. Any test that starts a JVM has a bunch of false positives. Testing: Enable LeakSanitizer for openssl-util-test. This reliably found the bug. Ran core tests under ASAN. Change-Id: I98760ed8f31b18b489a156f945c29c95c9bf3184 --- M be/CMakeLists.txt M be/src/util/CMakeLists.txt M be/src/util/asan.h M be/src/util/openssl-util.cc 4 files changed, 35 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/10666/4 -- To view, visit http://gerrit.cloudera.org:8080/10666 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I98760ed8f31b18b489a156f945c29c95c9bf3184 Gerrit-Change-Number: 10666 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding bit width
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10683 ) Change subject: IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding bit width .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ff3b00cbcab09dec11b3607d7d9a9c2c0025e1a Gerrit-Change-Number: 10683 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 11 Jun 2018 16:12:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2746: part 1: enable LSAN for many backend tests
Tim Armstrong has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/10668 ) Change subject: IMPALA-2746: part 1: enable LSAN for many backend tests .. IMPALA-2746: part 1: enable LSAN for many backend tests This turns on leak sanitizer for backend tests that required relatively small modifications to pass. We suppress a few leaks, mainly related to the embedded JVM. Testing: Ran core tests under ASAN. Change-Id: Ibdda092a4eb4bc827c75a8c121e5428ec746b7f4 --- M be/CMakeLists.txt M be/src/catalog/CMakeLists.txt M be/src/codegen/CMakeLists.txt M be/src/codegen/instruction-counter-test.cc M be/src/common/CMakeLists.txt M be/src/common/atomic-test.cc M be/src/common/atomic.h M be/src/exec/CMakeLists.txt M be/src/experiments/CMakeLists.txt M be/src/exprs/CMakeLists.txt M be/src/exprs/expr-test.cc M be/src/rpc/CMakeLists.txt M be/src/runtime/CMakeLists.txt M be/src/runtime/bufferpool/CMakeLists.txt M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/io/CMakeLists.txt M be/src/runtime/io/disk-io-mgr-stress.cc M be/src/runtime/io/disk-io-mgr-stress.h M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/mem-tracker.cc M be/src/scheduling/CMakeLists.txt M be/src/service/CMakeLists.txt M be/src/statestore/CMakeLists.txt M be/src/udf/udf.cc M be/src/util/CMakeLists.txt M be/src/util/decompress-test.cc A bin/lsan-suppressions.txt 28 files changed, 295 insertions(+), 238 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/10668/4 -- To view, visit http://gerrit.cloudera.org:8080/10668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibdda092a4eb4bc827c75a8c121e5428ec746b7f4 Gerrit-Change-Number: 10668 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding bit width
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10683 Change subject: IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding bit width .. IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding bit width Reading dictionary encoded Parquet data pages where the bit width is larger than the encoded type's size (e.g. coding 8 bit TINYINT with 16 bit dictionary indices) led to DCHECK error in debug builds. Impala does not create such parquet files (an N bit type can have maximum 2^N distinct values, so N bit dictionary indices are enough for a dictionary that contains every possible value), but the Parquet standard does not forbid to do so. These DCHECKs were probably introduced by a copy paste error (similar checks exist in the non-dictionary encoded bit reader functions, where they are valid). Testing: - a new test is added to check that these data pages can be decoded correctly Change-Id: I9ff3b00cbcab09dec11b3607d7d9a9c2c0025e1a --- M be/src/util/bit-packing.inline.h M be/src/util/bit-stream-utils.inline.h M testdata/data/README A testdata/data/dict_encoding_with_large_bit_width.parquet M tests/query_test/test_scanners.py 5 files changed, 21 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/10683/1 -- To view, visit http://gerrit.cloudera.org:8080/10683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9ff3b00cbcab09dec11b3607d7d9a9c2c0025e1a Gerrit-Change-Number: 10683 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer
[Impala-ASF-CR] IMPALA-7121: Clean up partitionIds from HdfsTable
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/10654 ) Change subject: IMPALA-7121: Clean up partitionIds_ from HdfsTable .. Patch Set 2: Code-Review+1 Thanks, lgtm. -- To view, visit http://gerrit.cloudera.org:8080/10654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b5a480e570aeae565fafd4f3e2b279e7a98c7da Gerrit-Change-Number: 10654 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Jun 2018 12:48:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7121: Clean up partitionIds from HdfsTable
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/10654 ) Change subject: IMPALA-7121: Clean up partitionIds_ from HdfsTable .. Patch Set 2: (1 comment) FYI, ran a core test suite and all tests are green. http://gerrit.cloudera.org:8080/#/c/10654/1/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java File fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java: http://gerrit.cloudera.org:8080/#/c/10654/1/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@455 PS1, Line 455: Preconditions.checkState( > It doesn't seem related to the changes of this commit and it might change t Thanks for pointing this out! This is no longer needed, I used this when I was experimenting with simply returning partitionMap_.keySet() that contains the default partition from getPartitionIds(). But since I remove the default partition's ID from the returned ID list this is no longer needed. -- To view, visit http://gerrit.cloudera.org:8080/10654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b5a480e570aeae565fafd4f3e2b279e7a98c7da Gerrit-Change-Number: 10654 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Jun 2018 12:16:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7121: Clean up partitionIds from HdfsTable
Gabor Kaszab has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/10654 ) Change subject: IMPALA-7121: Clean up partitionIds_ from HdfsTable .. IMPALA-7121: Clean up partitionIds_ from HdfsTable The purpose of introducing partitionIds_ member to HdfsTable was to be able to return the IDs of all the current partitions in constant time. Apparently, partitionMap_ also contains these IDs as the key of the map and this is accessible via keySet() also in constant time. It seems reasonable then to remove partitionIds_ and use partitionMap_.keySet() in getPartitionIds() to save some memory. One thing needs extra attention here is that modifying the result of keySet() would also modify partitionMap_ and we should avoid doing this. On every callsites of getPartitionIds() the first step the caller does is to copy the received items to a separate set. So as a solution getPartitionIds() internally creates a copy of the keySet(), removes the default partition and returns this copy to be sure that partitionMap_ can't be altered. The caller sites are also changed not to copy the items but to simpy use the set they received. This will guarantee that we don't regress the computing complexity of getting the partition IDs. Change-Id: I8b5a480e570aeae565fafd4f3e2b279e7a98c7da --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java 2 files changed, 15 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/10654/2 -- To view, visit http://gerrit.cloudera.org:8080/10654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8b5a480e570aeae565fafd4f3e2b279e7a98c7da Gerrit-Change-Number: 10654 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 ) Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. Patch Set 21: This change did not cherrypick successfully into branch 2.x. To resolve this, please do the cherry-pick manually and submit it to Gerrit at refs/for/2.x or add an exception to the branch 2.x copy of bin/ignored_commits.json. Thanks, your friendly bot at https://jenkins.impala.io/job/cherrypick-2.x-and-test/609/ . -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 21 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 11 Jun 2018 07:57:42 + Gerrit-HasComments: No
