[Impala-ASF-CR] IMPALA-1653: Don't close hiveserver2 session when connection is closed
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13306 ) Change subject: IMPALA-1653: Don't close hiveserver2 session when connection is closed .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG@30 PS3, Line 30: We may want to consider changing : some of this behavior. > We probably should consider removing states of expired sessions or Impala m To be fair, Impala is already vulnerable to similar kind of DoS attack today. A client can establish many connections to an Impala server, consuming all the FE service threads while leaving the sessions idle. All other clients will be blocked by this misbehaving client. See IMPALA-7802. Before this change, we could have as many sessions as number of FE service threads. After this change, the session will be kept alive even after the connection is closed so we could in theory have way more sessions than number of FE service threads. It seems there is a more pressing need for idle session timeout after this patch as it seems easier to create the situation of issuing a query without fetching all the results, holding up resources in the cluster but I could be missing something. Previously, the client disconnection will automatically close the session and unregistering all in-flight queries, freeing up the resources. -- To view, visit http://gerrit.cloudera.org:8080/13306 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4555cd9b73db5b4dde92cd4fac4f9bfa3664d78 Gerrit-Change-Number: 13306 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 15 May 2019 07:22:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: temporal query support for Kudu tables
Todd Lipcon has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13342 Change subject: WIP: temporal query support for Kudu tables .. WIP: temporal query support for Kudu tables This adds support for 'table ASOF TIMESTAMP ' for Kudu tables in queries. For example: functional_kudu> select * from tinytable; Query submitted at: 2019-05-15 00:47:57 (Coordinator: http://va1022.halxg.cloudera.com:25000) +--+-+ | a| b | +--+-+ | aaa | bbb | | c| | | | f | +--+-+ Fetched 3 row(s) in 0.17s functional_kudu> delete from tinytable; Modified 3 row(s), 0 row error(s) in 0.12s functional_kudu> insert into tinytable select * from tinytable asof timestamp utc_timestamp() - interval 1 minute; Modified 3 row(s), 0 row error(s) in 0.13s [va1022.halxg.cloudera.com:21000] functional_kudu> select * from tinytable; +--+-+ | a| b | +--+-+ | aaa | bbb | | c| | | | f | +--+-+ Fetched 3 row(s) in 0.15s WIP: needs tests, and probably should be "AS OF" instead of "ASOF", but I couldn't convince the parser to not have a shift/reduce conflict. Also seems likely we should be taking timestamps in localtime, rather than UTC time -- otherwise it's quite confusing that 'as of now()' actually returns a time in the future/past because it's interpreting a local timestamp as a UTC timestamp. Change-Id: I939898548c307810d509cf486e2c0f1f335a5ac4 --- M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scanner.cc M common/thrift/PlanNodes.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/jflex/sql-scanner.flex 11 files changed, 87 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/13342/1 -- To view, visit http://gerrit.cloudera.org:8080/13342 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I939898548c307810d509cf486e2c0f1f335a5ac4 Gerrit-Change-Number: 13342 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon
[Impala-ASF-CR] WIP: temporal query support for Kudu tables
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13342 )
Change subject: WIP: temporal query support for Kudu tables
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13342/1/fe/src/main/java/org/apache/impala/analysis/TableRef.java
File fe/src/main/java/org/apache/impala/analysis/TableRef.java:
http://gerrit.cloudera.org:8080/#/c/13342/1/fe/src/main/java/org/apache/impala/analysis/TableRef.java@147
PS1, Line 147: public TableRef(List path, String alias,
TableSampleClause tableSample, Expr asOfExpr) {
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/13342/1/fe/src/main/java/org/apache/impala/analysis/TableRef.java@392
PS1, Line 392: throw new AnalysisException("AS OF TIMESTAMP expression
must be a constant expression: " +
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/13342/1/fe/src/main/java/org/apache/impala/analysis/TableRef.java@396
PS1, Line 396: throw new AnalysisException("AS OF TIMESTAMP expression
must be a timestamp type but is '" +
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/13342/1/fe/src/main/java/org/apache/impala/analysis/TableRef.java@402
PS1, Line 402: throw new AnalysisException("Invalid AS OF TIMESTAMP
expression: " + ie.getMessage());
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13342
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I939898548c307810d509cf486e2c0f1f335a5ac4
Gerrit-Change-Number: 13342
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Wed, 15 May 2019 07:55:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: temporal query support for Kudu tables
Nong Li has posted comments on this change. ( http://gerrit.cloudera.org:8080/13342 ) Change subject: WIP: temporal query support for Kudu tables .. Patch Set 1: This is great! -- To view, visit http://gerrit.cloudera.org:8080/13342 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I939898548c307810d509cf486e2c0f1f335a5ac4 Gerrit-Change-Number: 13342 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Nong Li (20) Gerrit-Comment-Date: Wed, 15 May 2019 08:07:37 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: temporal query support for Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13342 ) Change subject: WIP: temporal query support for Kudu tables .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3235/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13342 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I939898548c307810d509cf486e2c0f1f335a5ac4 Gerrit-Change-Number: 13342 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Nong Li (20) Gerrit-Comment-Date: Wed, 15 May 2019 08:26:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8121: part 3: invalidate on memory pressure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13302 ) Change subject: IMPALA-8121: part 3: invalidate on memory pressure .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4259/ -- To view, visit http://gerrit.cloudera.org:8080/13302 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d55ef0058abcf70f75b10ae9d89a0274859969 Gerrit-Change-Number: 13302 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 10:18:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8138: Remove FAULT INJECTION RPC DELAY
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13060 ) Change subject: IMPALA-8138: Remove FAULT_INJECTION_RPC_DELAY .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I712b188e0cdf91f431c9b94052501e5411af407b Gerrit-Change-Number: 13060 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 15 May 2019 10:58:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8138: Remove FAULT INJECTION RPC DELAY
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13060 ) Change subject: IMPALA-8138: Remove FAULT_INJECTION_RPC_DELAY .. IMPALA-8138: Remove FAULT_INJECTION_RPC_DELAY This patch removes the FAULT_INJECTION_RPC_DELAY macro and replaces its uses with DebugAction which is more flexible. For example, it supports JITTER which injects random delays. Every backend rpc has a debug action of the form RPC_NAME_DELAY. DebugAction has previously always been used via query options. However, for the rpcs considered here there is not always a query with an accessible TQUeryOptions available (for example, we do not send any query info with the RemoteShutdown rpc), so this patch introduces a flag, '--debug_actions', which is used to control these rpc delay debug actions. Testing: - Updated existing tests to use the new mechanism. Change-Id: I712b188e0cdf91f431c9b94052501e5411af407b Reviewed-on: http://gerrit.cloudera.org:8080/13060 Reviewed-by: Thomas Marshall Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M be/src/service/control-service.cc M be/src/service/data-stream-service.cc M be/src/service/impala-internal-service.cc M be/src/testutil/fault-injection-util.cc M be/src/testutil/fault-injection-util.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M tests/custom_cluster/test_rpc_timeout.py 9 files changed, 54 insertions(+), 83 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I712b188e0cdf91f431c9b94052501e5411af407b Gerrit-Change-Number: 13060 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13251 ) Change subject: IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading .. IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading This fixes three issues for functional dataset loading: - works around HIVE-21675, a bug in which 'CREATE VIEW IF NOT EXISTS' does not function correctly in our current Hive build. This has been fixed already, but the workaround is pretty simple, and actually the 'drop and recreate' pattern is used more widely for data-loading than the 'create if not exists' one. - Moves the creation of the 'hive_index' table from load-dependent-tables.sql to a new load-dependent-tables-hive2.sql file which is only executed on Hive 2. - Moving from MR to Tez execution changed the behavior of data loading by disabling the auto-merging of small files. With Hive-on-MR, this behavior defaulted to true, but with Hive-on-Tez it defaults false. The change is likely motivated by the fact that Tez automatically groups small splits on the _input_ side and thus is less likely to produce lots of small files. However, that grouping functionality doesn't work properly in localhost clusters (TEZ-3310) so we aren't seeing the benefit. So, this patch enables the post-process merging of small files. Prior to this change, the 'alltypesaggmultifilesnopart' test table was getting 40+ files inside it, which broke various planner tests. With the change, it gets the expected 4 files. Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd Reviewed-on: http://gerrit.cloudera.org:8080/13251 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/resources/hive-site.xml.py M testdata/bin/create-load-data.sh A testdata/bin/load-dependent-tables-hive2.sql M testdata/bin/load-dependent-tables.sql M testdata/datasets/functional/functional_schema_template.sql 6 files changed, 60 insertions(+), 11 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd Gerrit-Change-Number: 13251 Gerrit-PatchSet: 8 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13251 ) Change subject: IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd Gerrit-Change-Number: 13251 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 May 2019 11:00:44 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13329 ) Change subject: WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4263/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13329 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12 Gerrit-Change-Number: 13329 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 May 2019 11:35:51 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13329 ) Change subject: WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader .. Patch Set 3: I do not understand the reason behind the failed build. Rebased and started a new build. I will address the comments later. -- To view, visit http://gerrit.cloudera.org:8080/13329 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12 Gerrit-Change-Number: 13329 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 May 2019 11:36:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13215 ) Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and partition .. Patch Set 15: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4262/ -- To view, visit http://gerrit.cloudera.org:8080/13215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 Gerrit-Change-Number: 13215 Gerrit-PatchSet: 15 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 May 2019 11:52:26 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13329 ) Change subject: WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4264/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13329 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12 Gerrit-Change-Number: 13329 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 May 2019 13:03:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8504: Support CREATE TABLE statement with Kudu/HMS integration
Grant Henke has posted comments on this change. (
http://gerrit.cloudera.org:8080/13318 )
Change subject: IMPALA-8504: Support CREATE TABLE statement with Kudu/HMS
integration
..
Patch Set 1:
(15 comments)
http://gerrit.cloudera.org:8080/#/c/13318/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13318/1//COMMIT_MSG@7
PS1, Line 7: IMPALA-8504: Support CREATE TABLE statement with Kudu/HMS
integration
I was under the impression there would need to be special coordination between
calls to create the table in HMS and Kudu based on the design doc.
In the legacy world, Impala creates an HMS entry and a Kudu table.
Now, I think we just create the Kudu table and that auto-populates the HMS. But
I didn't see the calls to the HMS removed in this patch.
For legacy tables, calls to HMS likely still need to occur. Though those are
not likely create table calls, but alter and drop calls.
http://gerrit.cloudera.org:8080/#/c/13318/1//COMMIT_MSG@9
PS1, Line 9: CATS
CTAS - Create Table As Select?
http://gerrit.cloudera.org:8080/#/c/13318/1//COMMIT_MSG@9
PS1, Line 9: statments
statements
http://gerrit.cloudera.org:8080/#/c/13318/1//COMMIT_MSG@13
PS1, Line 13: CREATE EXTERNAL TABLE t STORED AS KUDU TBLPROPERTIES
How is this different then it was before? Why does it need to be different?
http://gerrit.cloudera.org:8080/#/c/13318/1//COMMIT_MSG@21
PS1, Line 21: 2) When Kudu/HMS integration is enabled, the external table is no
longer
I think this discussion is still in flight with Kudu devs.
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@329
PS1, Line 329: // When Kudu/HMS integration is not enabled, remove hidden
table property
What about legacy tables?
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/main/java/org/apache/impala/catalog/KuduTable.java
File fe/src/main/java/org/apache/impala/catalog/KuduTable.java:
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/main/java/org/apache/impala/catalog/KuduTable.java@130
PS1, Line 130: public String getStorageHandlerClassName() {
Should this return the handler from the HMS instead of generating one based on
HMS integration being enabled?
I suspect there are two cases, the first is when creating a table, and the
second is when using an already created table. When creating a table I suspect
KUDU_STORAGE_HANDLER would always be used. When using an already created table,
it could be KUDU_LEGACY_STORAGE_HANDLER, even if the HMS integration was turned
on.
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/main/java/org/apache/impala/catalog/KuduTable.java@183
PS1, Line 183: hmsConfig = kuduClient.getHiveMetastoreConfig();
Do we care if Kudu is configured to use a different HMS than Impala? They need
to be the same right?
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/main/java/org/apache/impala/catalog/KuduTable.java@245
PS1, Line 245: // When Kudu/HMS integration is enabled, generates the
Kudu table name from
Does it matter if it's a legacy table here? In that case it will be
`impala::db_name.table_name` right?
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java:
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java@212
PS1, Line 212: // When Kudu/HMS integration is enabled, generates the
Kudu table name from
Can this logic be shared with the table name logic in KuduTable.load? Perhaps
factored next to the table name function in KuduUtil.
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2578
PS1, Line 2578: String newKuduTableName =
KuduUtil.getLegacyDefaultKuduTableName(
What about when HMS integration is enabled?
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/main/java/org/apache/impala/util/KuduUtil.java
File fe/src/main/java/org/apache/impala/util/KuduUtil.java:
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/main/java/org/apache/impala/util/KuduUtil.java@397
PS1, Line 397: public static String getKuduTableName(String metastoreDbName,
Maybe this should take a boolean for HMS enabled, and return the correct format
based on that boolean.
http://gerrit.cloudera.org:8080/#/c/13318/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java:
PS1:
Because this doesn't d
[Impala-ASF-CR] IMPALA-8545: Test Ldap authentication
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13337 )
Change subject: IMPALA-8545: Test Ldap authentication
..
Patch Set 2:
(7 comments)
Thanks for fixing this. I think this fills a major test gap. I think we need to
figure out some issues with test sequencing in bin/run-all-tests.sh and
starting/stopping clusters. It looks like this runs as part of the FE tests, so
will actually affect the cluster that the e2e tests run against.
http://gerrit.cloudera.org:8080/#/c/13337/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13337/2//COMMIT_MSG@8
PS2, Line 8:
Can you mention that this changes JdbcTests to start custom clusters.
http://gerrit.cloudera.org:8080/#/c/13337/2/fe/src/test/java/org/apache/impala/service/JdbcTestBase.java
File fe/src/test/java/org/apache/impala/service/JdbcTestBase.java:
http://gerrit.cloudera.org:8080/#/c/13337/2/fe/src/test/java/org/apache/impala/service/JdbcTestBase.java@44
PS2, Line 44: public static void cleanUp() throws Exception {
Brief comment? I think the important thing is that child classes should call
this from their cleanUp() method.
http://gerrit.cloudera.org:8080/#/c/13337/2/fe/src/test/java/org/apache/impala/service/JdbcTestBase.java@72
PS2, Line 72: protected void addTestTable(String createTableSql) throws
Exception {
Brief method comment?
http://gerrit.cloudera.org:8080/#/c/13337/2/fe/src/test/java/org/apache/impala/service/LdapJdbcTest.java
File fe/src/test/java/org/apache/impala/service/LdapJdbcTest.java:
http://gerrit.cloudera.org:8080/#/c/13337/2/fe/src/test/java/org/apache/impala/service/LdapJdbcTest.java@54
PS2, Line 54: user
testUser_? When reading the below code I was looking for a local variable or
argument called 'user'
http://gerrit.cloudera.org:8080/#/c/13337/2/fe/src/test/java/org/apache/impala/service/LdapJdbcTest.java@55
PS2, Line 55: password
testPassword_?
http://gerrit.cloudera.org:8080/#/c/13337/2/fe/src/test/java/org/apache/impala/service/LdapJdbcTest.java@73
PS2, Line 73: CustomClusterRunner.StartImpalaCluster();
I think it would be better if we changed it so either:
* We have a set of JDBC end-to-end tests that don't start/stop clusters. These
get run against the regular e2e cluster. Then we can run JDBC custom cluster
tests with the regular custom cluster tests.
* All JDBC tests start their own cluster and are treated the same as custom
cluster tests.
I think I prefer 1.
Actually... I think as-is this introduces a major problem because this cluster
will be left running after the FE tests for the e2e tests and
TEST_START_CLUSTER_ARGS won't have an effect (i.e. we won't test what we think
we're testing).
http://gerrit.cloudera.org:8080/#/c/13337/2/fe/src/test/java/org/apache/impala/testutil/CustomClusterRunner.java
File fe/src/test/java/org/apache/impala/testutil/CustomClusterRunner.java:
http://gerrit.cloudera.org:8080/#/c/13337/2/fe/src/test/java/org/apache/impala/testutil/CustomClusterRunner.java@22
PS2, Line 22: public class CustomClusterRunner {
Can you add a note to bin/run-all-tests.sh that JdbcTests can start custom
clusters.
--
To view, visit http://gerrit.cloudera.org:8080/13337
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I92b5e60860c60209c1bd8afe5b3ea201fb7a7513
Gerrit-Change-Number: 13337
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 15 May 2019 16:56:09 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Add USE CDP HIVE=true case to build-all-flag-combinations.sh
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13335 ) Change subject: Add USE_CDP_HIVE=true case to build-all-flag-combinations.sh .. Patch Set 1: The job that calls build-all-flag-combinations.sh passed, not sure why the other job failed. -- To view, visit http://gerrit.cloudera.org:8080/13335 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f8e689242e20efb37fbadf7c04764ea8ffb9a9f Gerrit-Change-Number: 13335 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 15 May 2019 17:07:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1653: Don't close hiveserver2 session when connection is closed
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13306 ) Change subject: IMPALA-1653: Don't close hiveserver2 session when connection is closed .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG@30 PS3, Line 30: We may want to consider changing : some of this behavior. > To be fair, Impala is already vulnerable to similar kind of DoS attack toda We had started discussing this a bit offline. Agree that we rely on the session timeout to clean up idle queries since the current disconnection behaviour may be bailing us out in some cases. One idea that Thomas suggested was having a disconnected session timeout that could be enabled by default. It's easy to convince ourselves that this doesn't break existing workflows. I think this is a minor extension to ImpalaServer::ExpireSessions() - we're sweeping over the list of sessions periodically anyway. One nice thing about that approach is that you could have a --disconnected_session_timeout or similar that you could set to a low value to approximate the old behaviour. So I'm pretty positive on that one. We'd also discussed having different default behaviour for HTTP and TCP and/or hiding it behind a flag, but that's a little undesirable because of the added complexity (and it's not clear that immediately closing sessions was the right behaviour). We could also try to mitigate the cost of the queries by enabled --idle_query_timeout by default. That could disrupt workloads but is less risky since all well-behaved clients will fetch results eager. One data point is that, when I tried a while back, our tests ran fine with an idle query timeout but broke with an idle session timeout (since the tend to open connections and keep them around while the tests are running). I'm not sure how I feel about that idea. -- To view, visit http://gerrit.cloudera.org:8080/13306 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4555cd9b73db5b4dde92cd4fac4f9bfa3664d78 Gerrit-Change-Number: 13306 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 15 May 2019 17:07:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Add USE CDP HIVE=true case to build-all-flag-combinations.sh
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13335 Change subject: Add USE_CDP_HIVE=true case to build-all-flag-combinations.sh .. Add USE_CDP_HIVE=true case to build-all-flag-combinations.sh This does a basic debug build with USE_CDP_HIVE=true to verify we don't break anything. Change-Id: I3f8e689242e20efb37fbadf7c04764ea8ffb9a9f --- M bin/jenkins/build-all-flag-combinations.sh 1 file changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/13335/1 -- To view, visit http://gerrit.cloudera.org:8080/13335 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3f8e689242e20efb37fbadf7c04764ea8ffb9a9f Gerrit-Change-Number: 13335 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12797 ) Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4265/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I839d7a3a2f5e1309c33a1f66013ef11628c5dc11 Gerrit-Change-Number: 12797 Gerrit-PatchSet: 17 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 15 May 2019 17:17:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12797 ) Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I839d7a3a2f5e1309c33a1f66013ef11628c5dc11 Gerrit-Change-Number: 12797 Gerrit-PatchSet: 17 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 15 May 2019 17:17:31 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix condition for starting YARN on USE CDP HIVE=true
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13343 Change subject: Fix condition for starting YARN on USE_CDP_HIVE=true .. Fix condition for starting YARN on USE_CDP_HIVE=true Dataload on Hive 3 uses YARN, but YARN is not needed for Hive 2 dataload. This fixes the condition so that YARN does not start for USE_CDP_HIVE=false (Hive 2). Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 --- M testdata/cluster/admin 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/13343/1 -- To view, visit http://gerrit.cloudera.org:8080/13343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Gerrit-Change-Number: 13343 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-8369 : Skip test owner privileges test when running against Hive-3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13339 ) Change subject: IMPALA-8369 : Skip test_owner_privileges test when running against Hive-3 .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4266/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f904446f50b5095443bf27b3092a2e3665b76d3 Gerrit-Change-Number: 13339 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 17:26:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Skip test owner privileges test when running against Hive-3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13339 ) Change subject: IMPALA-8369 : Skip test_owner_privileges test when running against Hive-3 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f904446f50b5095443bf27b3092a2e3665b76d3 Gerrit-Change-Number: 13339 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 17:26:37 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix condition for starting YARN on USE CDP HIVE=true
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13343 to look at the new patch set (#2). Change subject: Fix condition for starting YARN on USE_CDP_HIVE=true .. Fix condition for starting YARN on USE_CDP_HIVE=true Dataload on Hive 3 uses YARN, but YARN is not needed for Hive 2 dataload. This fixes the condition so that YARN does not start for USE_CDP_HIVE=false (Hive 2). Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 --- M testdata/cluster/admin 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/13343/2 -- To view, visit http://gerrit.cloudera.org:8080/13343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Gerrit-Change-Number: 13343 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Fix condition for starting YARN on USE CDP HIVE=true
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13343 ) Change subject: Fix condition for starting YARN on USE_CDP_HIVE=true .. Patch Set 2: Do we have the same bug in run-hive-server.sh? if [[ $USE_CDP_HIVE && -n "$SENTRY_HOME" ]]; then (I'll never understand bash) -- To view, visit http://gerrit.cloudera.org:8080/13343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Gerrit-Change-Number: 13343 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 17:45:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13310 ) Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/13310/1/docs/topics/impala_grant.xml File docs/topics/impala_grant.xml: http://gerrit.cloudera.org:8080/#/c/13310/1/docs/topics/impala_grant.xml@62 PS1, Line 62: | VIEW > This is incorrect, the syntax is still "TABLE" even for view. Done http://gerrit.cloudera.org:8080/#/c/13310/1/docs/topics/impala_grant.xml@71 PS1, Line 71: The object_type of VIEW is supported : for the SELECT(column_name) privilege. : > Remove this. This is incorrect. Done http://gerrit.cloudera.org:8080/#/c/13310/1/docs/topics/impala_revoke.xml File docs/topics/impala_revoke.xml: http://gerrit.cloudera.org:8080/#/c/13310/1/docs/topics/impala_revoke.xml@63 PS1, Line 63: | VIEW > similarly, remove this. Done http://gerrit.cloudera.org:8080/#/c/13310/1/docs/topics/impala_revoke.xml@65 PS1, Line 65: : The object_type of VIEW is supported : for the SELECT(column_name) privilege. : : > remove this as well. Done -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 17:48:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13310 ) Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/326/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 17:49:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Hello Fredy Wijaya, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13310 to look at the new patch set (#2). Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. IMPALA-8424: [DOCS] Support for column-level permissions on views Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 --- M docs/shared/impala_common.xml M docs/topics/impala_grant.xml M docs/topics/impala_revoke.xml 3 files changed, 11 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/13310/2 -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13310 ) Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. Patch Set 2: > (4 comments) > > Do we have a doc somewhere that says column-level permissions/privileges > on views aren't supported? If yes, I think we should update that > doc by removing that info. If not, I guess the users can assume > column-level permissions are supported on both tables and views. No mention of column-level permission on views in docs. So this will end up being a very minor edit patch -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 17:51:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13310 ) Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. Patch Set 2: (1 comment) > Patch Set 2: > > > (4 comments) > > > > Do we have a doc somewhere that says column-level permissions/privileges > > on views aren't supported? If yes, I think we should update that > > doc by removing that info. If not, I guess the users can assume > > column-level permissions are supported on both tables and views. > > No mention of column-level permission on views in docs. So this will end up > being a very minor edit patch I see. http://gerrit.cloudera.org:8080/#/c/13310/2/docs/shared/impala_common.xml File docs/shared/impala_common.xml: http://gerrit.cloudera.org:8080/#/c/13310/2/docs/shared/impala_common.xml@176 PS2, Line 176: , VIEW we should also remove VIEW. -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 17:55:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Hello Fredy Wijaya, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13310 to look at the new patch set (#3). Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. IMPALA-8424: [DOCS] Support for column-level permissions on views Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 --- M docs/shared/impala_common.xml M docs/topics/impala_grant.xml M docs/topics/impala_revoke.xml 3 files changed, 11 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/13310/3 -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13310 ) Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. Patch Set 3: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/327/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 17:58:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13310 ) Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13310/2/docs/shared/impala_common.xml File docs/shared/impala_common.xml: http://gerrit.cloudera.org:8080/#/c/13310/2/docs/shared/impala_common.xml@176 PS2, Line 176: , VIEW > we should also remove VIEW. Done -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 17:58:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13310 ) Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 18:01:50 + Gerrit-HasComments: No
[Impala-ASF-CR] Add USE CDP HIVE=true case to build-all-flag-combinations.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13335 ) Change subject: Add USE_CDP_HIVE=true case to build-all-flag-combinations.sh .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3236/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13335 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f8e689242e20efb37fbadf7c04764ea8ffb9a9f Gerrit-Change-Number: 13335 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 15 May 2019 18:05:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13310 ) Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/326/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 18:05:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13310 ) Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. Patch Set 3: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/327/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 18:07:09 + Gerrit-HasComments: No
[Impala-ASF-CR] wip
Sudhanshu Arora has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13334 Change subject: wip .. wip To show the approach that I am following. This needs more work based on AcidUtils#getAcidState Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java 2 files changed, 53 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/13334/5 -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 5 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] Fix condition for starting YARN on USE CDP HIVE=true
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13343 ) Change subject: Fix condition for starting YARN on USE_CDP_HIVE=true .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3237/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Gerrit-Change-Number: 13343 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 18:23:25 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix condition for starting YARN on USE CDP HIVE=true
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13343 ) Change subject: Fix condition for starting YARN on USE_CDP_HIVE=true .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3238/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Gerrit-Change-Number: 13343 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 18:26:44 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13329 ) Change subject: WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13329 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12 Gerrit-Change-Number: 13329 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 May 2019 18:24:31 + Gerrit-HasComments: No
[Impala-ASF-CR] wip
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: wip .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3239/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 5 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 15 May 2019 18:42:36 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix condition for starting YARN on USE CDP HIVE=true
Hello Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13343 to look at the new patch set (#3). Change subject: Fix condition for starting YARN on USE_CDP_HIVE=true .. Fix condition for starting YARN on USE_CDP_HIVE=true Dataload on Hive 3 uses YARN, but YARN is not needed for Hive 2 dataload. This fixes the condition so that YARN does not start for USE_CDP_HIVE=false (Hive 2). This fixes a similar condition for manipulating the classpath in testdata/bin/run-hive-server.sh. Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 --- M testdata/bin/run-hive-server.sh M testdata/cluster/admin 2 files changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/13343/3 -- To view, visit http://gerrit.cloudera.org:8080/13343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Gerrit-Change-Number: 13343 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] Fix condition for starting YARN on USE CDP HIVE=true
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13343 ) Change subject: Fix condition for starting YARN on USE_CDP_HIVE=true .. Patch Set 3: > Do we have the same bug in run-hive-server.sh? > > if [[ $USE_CDP_HIVE && -n "$SENTRY_HOME" ]]; then > > > (I'll never understand bash) Changed the condition in run-hive-server.sh. -- To view, visit http://gerrit.cloudera.org:8080/13343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Gerrit-Change-Number: 13343 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 18:51:56 + Gerrit-HasComments: No
[Impala-ASF-CR] Add USE CDP HIVE=true case to build-all-flag-combinations.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13335 ) Change subject: Add USE_CDP_HIVE=true case to build-all-flag-combinations.sh .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4267/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13335 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f8e689242e20efb37fbadf7c04764ea8ffb9a9f Gerrit-Change-Number: 13335 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 15 May 2019 18:53:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1653: Don't close hiveserver2 session when connection is closed
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13306 ) Change subject: IMPALA-1653: Don't close hiveserver2 session when connection is closed .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG@30 PS3, Line 30: We may want to consider changing : some of this behavior. > We had started discussing this a bit offline. Agree that we rely on the ses Enabling --idle_query_timeout by default sounds like a reasonable mitigation against clients which don't fetch results. As Thomas said, the session state isn't that big so we can bound it at some large numbers and do LRU reclamation or set a reasonable large limit (e.g. 24 hrs) on how long a session will be maintained by default. What was the idle_session-timeout set to when you ran tests with it enabled ? -- To view, visit http://gerrit.cloudera.org:8080/13306 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4555cd9b73db5b4dde92cd4fac4f9bfa3664d78 Gerrit-Change-Number: 13306 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 15 May 2019 19:03:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1653: Don't close hiveserver2 session when connection is closed
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13306 ) Change subject: IMPALA-1653: Don't close hiveserver2 session when connection is closed .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG@30 PS3, Line 30: We may want to consider changing : some of this behavior. > Enabling --idle_query_timeout by default sounds like a reasonable mitigatio It failed even if set to a very high value, IIRC even 1h. The problem with keeping the session state around isn't the size of the session state itself, it's the queries dangling off it. The idle query timeout only cancels queries, doesn't unregister them. -- To view, visit http://gerrit.cloudera.org:8080/13306 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4555cd9b73db5b4dde92cd4fac4f9bfa3664d78 Gerrit-Change-Number: 13306 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 15 May 2019 19:12:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1653: Don't close hiveserver2 session when connection is closed
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13306 ) Change subject: IMPALA-1653: Don't close hiveserver2 session when connection is closed .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG@30 PS3, Line 30: We may want to consider changing : some of this behavior. > It failed even if set to a very high value, IIRC even 1h. I believe the advantage of that is that the client can still check the status, get the profile, etc. We talked in the past about extending the idle query timeout to unregister queries at some point, e.g. after a timeout specified by yet another config option or some multiple of or delta on top of the original timeout. -- To view, visit http://gerrit.cloudera.org:8080/13306 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4555cd9b73db5b4dde92cd4fac4f9bfa3664d78 Gerrit-Change-Number: 13306 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 15 May 2019 19:15:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Fix condition for starting YARN on USE CDP HIVE=true
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13343 ) Change subject: Fix condition for starting YARN on USE_CDP_HIVE=true .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Gerrit-Change-Number: 13343 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 19:15:10 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix condition for starting YARN on USE CDP HIVE=true
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13343 ) Change subject: Fix condition for starting YARN on USE_CDP_HIVE=true .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4268/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Gerrit-Change-Number: 13343 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 19:28:02 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix condition for starting YARN on USE CDP HIVE=true
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13343 ) Change subject: Fix condition for starting YARN on USE_CDP_HIVE=true .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3240/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Gerrit-Change-Number: 13343 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 19:36:32 + Gerrit-HasComments: No
[Impala-ASF-CR] DWX-124: Prometheus metrics support in Impala
hars...@cloudera.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13345 Change subject: DWX-124: Prometheus metrics support in Impala .. DWX-124: Prometheus metrics support in Impala -- This change adds Prometheus text explosion format metric generation in impala, more details about text explostion can be found here: https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format -- Added unit test to test this change -- Modified distcc file support ubuntu 18.04 -- Doc that talk about this approach is here: https://docs.google.com/document/d/1KeOuFowH83q9l7iGHFMXL2mAejt4EZCjn3XfKDu7sho/edit?usp=sharing Change-Id: I5349085a2007b568cb97f9b8130804ea64d7bb08 --- M be/src/util/collection-metrics.h M be/src/util/histogram-metric.h M be/src/util/metrics-test.cc M be/src/util/metrics.cc M be/src/util/metrics.h M bin/distcc/distcc_server_setup.sh 6 files changed, 248 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/13345/1 -- To view, visit http://gerrit.cloudera.org:8080/13345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5349085a2007b568cb97f9b8130804ea64d7bb08 Gerrit-Change-Number: 13345 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8503: allow the Hive Metastore to start with kudu-hive plugin
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13319 ) Change subject: IMPALA-8503: allow the Hive Metastore to start with kudu-hive plugin .. Patch Set 1: While this is of course fine for you to use for local testing of your other changes for now, I don't think this is something that we want to review and commit. Seems better to just wait until kudu-hive.jar is available through the normal mechanisms, which shouldn't take very long (unless there's an issue I don't know about) -- To view, visit http://gerrit.cloudera.org:8080/13319 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4bd1488ced51840ec986d29ed371e26168abcc76 Gerrit-Change-Number: 13319 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 15 May 2019 19:41:06 + Gerrit-HasComments: No
[Impala-ASF-CR] DWX-124: Prometheus metrics support in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13345 ) Change subject: DWX-124: Prometheus metrics support in Impala .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3241/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5349085a2007b568cb97f9b8130804ea64d7bb08 Gerrit-Change-Number: 13345 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 15 May 2019 20:26:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13346 Change subject: IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API .. IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API Impala uses a workaround to detect if a user is a Sentry admin by by asking the Sentry API to list privileges associated with the user, since previously Sentry did not provide an suitable API to peform this check. SENTRY-2440 provides a new API (https://issues.apache.org/jira/browse/SENTRY-2440), and thus it would be good to update the Impala to make the code easier to understand. Testing: 1. Passed the tests in AuthorizationTest.java. 2. Passed the Sentry related tests in AuthorizationStmtTest.java. 2. Passed the tests in test_sentry.py. Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82 --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java 5 files changed, 25 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/13346/1 -- To view, visit http://gerrit.cloudera.org:8080/13346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82 Gerrit-Change-Number: 13346 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya
[Impala-ASF-CR] WIP try to build thrift with no default operators
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13312 ) Change subject: WIP try to build thrift with no_default_operators .. Patch Set 1: (1 comment) Thanks for having a look. I addressed your comment and will fold this into the ongoing review for IMPALA-8460 (Simplify cluster membership management) http://gerrit.cloudera.org:8080/#/c/13312/1/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/13312/1/fe/src/main/java/org/apache/impala/service/Frontend.java@1397 PS1, Line 1397: List access_events = Lists.newArrayList(); > You can just pass in any iterable to this function: https://google.github.i Done -- To view, visit http://gerrit.cloudera.org:8080/13312 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic956af99b4862151088cefd67066f4215c6c42d9 Gerrit-Change-Number: 13312 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 15 May 2019 20:37:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13310 ) Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. IMPALA-8424: [DOCS] Support for column-level permissions on views Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Reviewed-on: http://gerrit.cloudera.org:8080/13310 Reviewed-by: Fredy Wijaya Tested-by: Impala Public Jenkins --- M docs/shared/impala_common.xml M docs/topics/impala_grant.xml M docs/topics/impala_revoke.xml 3 files changed, 11 insertions(+), 15 deletions(-) Approvals: Fredy Wijaya: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13346 ) Change subject: IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3242/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82 Gerrit-Change-Number: 13346 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 20:47:55 + Gerrit-HasComments: No
[Impala-ASF-CR] DWX-124: Prometheus metrics support in Impala
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13345 )
Change subject: DWX-124: Prometheus metrics support in Impala
..
Patch Set 1:
(17 comments)
Did a pass over it. Mainly style and best practice comments at this stage.
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/collection-metrics.h
File be/src/util/collection-metrics.h:
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/collection-metrics.h@170
PS1, Line 170: string name, std::stringstream* val, std::stringstream*
metric_kind) {
It's annoying that the original author of this file didn't move the ToJson(),
etc methods to a .cc file. Putting non-perf-critical stuff in headers mainly
blows up compile times. Anyway, you don't need to fix this, I just wanted to
flag the bad practice.
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/collection-metrics.h@170
PS1, Line 170: string
Use std::string in header files (we have a policy not to import std::string
into the global namespace in headers, but some other header is probably
violating that)
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/collection-metrics.h@172
PS1, Line 172: string collect_data;
We should just append to val instead of adding an intermediate string.
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/collection-metrics.h@175
PS1, Line 175: tatic_cast(
Why are the casts needed? this seems suspicious to me - like it might not work
correctly for some types.
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/collection-metrics.h@178
PS1, Line 178: collect_data += name + "_last " + std::to_string(value_) +
"\n";
For string concatenation we either prefer using << to append to a stringstream,
or strings::Substitute() to fill out a template. Since we are already building
a stringstream, we should go with the first approach. That may also avoid the
need for some of this to_string() calls.
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/collection-metrics.h@200
PS1, Line 200: sqrt
std::sqrt
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/collection-metrics.h@204
PS1, Line 204: *metric_kind << "# TYPE " + name + " counter";
use << to append to the stream instead of string concatenation.
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/histogram-metric.h
File be/src/util/histogram-metric.h:
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/histogram-metric.h@81
PS1, Line 81: histogram_data +=
Same comments about using << to append directly to value
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/histogram-metric.h@99
PS1, Line 99: *metric_kind << "# TYPE " + name + " histogram";
Same comment about <<
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/metrics.h
File be/src/util/metrics.h:
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/metrics.h@171
PS1, Line 171: string
std::string in headers
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/metrics.h@173
PS1, Line 173: *metric_kind << "# TYPE " + name + " " +
PrintThriftEnum(kind()).c_str();
Use << instead of +
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/metrics.cc
File be/src/util/metrics.cc:
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/metrics.cc@190
PS1, Line 190: return;
return not needed
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/metrics.cc@206
PS1, Line 206:
Don't need to add vertical whitespace
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/metrics.cc@220
PS1, Line 220: void MetricGroup::ToPrometheus(bool include_children,
std::stringstream* out_val) {
Same comments apply about using << instead of strings.
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/metrics.cc@220
PS1, Line 220: std::
shouldn't need to use std:: prefix in .cc files.
http://gerrit.cloudera.org:8080/#/c/13345/1/be/src/util/metrics.cc@245
PS1, Line 245:
unnecessary vertical whitespace and return
http://gerrit.cloudera.org:8080/#/c/13345/1/bin/distcc/distcc_server_setup.sh
File bin/distcc/distcc_server_setup.sh:
http://gerrit.cloudera.org:8080/#/c/13345/1/bin/distcc/distcc_server_setup.sh@58
PS1, Line 58: if ! [[ $LSB_VERSION == 14.04 || $LSB_VERSION == 16.04 ||
$LSB_VERSION == 18.04 ]]; then
Great if this works - but can you do this in a separate change? I also wanted
to know how you tested it.
--
To view, visit http://gerrit.cloudera.org:8080/13345
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5349085a2007b568cb97f9b8130804ea64d7bb08
Gerrit-Change-Number: 13345
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 15 May 2019 20:49:52 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] DWX-124: Prometheus metrics support in Impala
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13345 ) Change subject: DWX-124: Prometheus metrics support in Impala .. Patch Set 1: (3 comments) Didn't look at code yet but a couple of administrative comments http://gerrit.cloudera.org:8080/#/c/13345/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13345/1//COMMIT_MSG@7 PS1, Line 7: DWX-124: Prometheus metrics support in Impala DWX isn't an Apache project. Please reference an IMPALA JIRA or file one if one isn't present. http://gerrit.cloudera.org:8080/#/c/13345/1//COMMIT_MSG@9 PS1, Line 9: -- This change adds Prometheus text explosion format metric generation in impala, more details Can you wrap lines to 72 chars in commit messages. http://gerrit.cloudera.org:8080/#/c/13345/1//COMMIT_MSG@13 PS1, Line 13: -- Doc that talk about this approach is here: https://docs.google.com/document/d/1KeOuFowH83q9l7iGHFMXL2mAejt4EZCjn3XfKDu7sho/edit?usp=sharing This doc isn't publicly accesssible. -- To view, visit http://gerrit.cloudera.org:8080/13345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5349085a2007b568cb97f9b8130804ea64d7bb08 Gerrit-Change-Number: 13345 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 15 May 2019 20:37:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Todd Lipcon, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13215 to look at the new patch set (#16). Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and partition .. IMPALA-8438: Store WriteId and ValidWriteId list for table and partition This happens when tables load metadata from HMS. Add MetastoreShim functions to support HMS3 only functions. Add validwriteIdlists to query profile through timeline. Tests: Manually tests HMS2 and HMS3, using log files to check Unit tests against HMS3 ToDo: WriteId and valid writeIds can be fetched in other time, need more study on that. Profile example: Query Compilation: 5s057ms - Metadata load started: 63.006ms (63.006ms) - Metadata load finished. loaded-tables=2/2...: 4s801ms (4s738ms) - Loaded ValidWriteIdLists: acid.insert_only_no_partitions:6:9223372036854775807:: acid.insert_only_with_partitions:3:9223372036854775807:: : 4s921ms (120.580ms) - Analysis finished: 4s929ms (8.013ms) Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 --- M common/thrift/CatalogObjects.thrift A fe/src/compat-hive-2/java/org/apache/hadoop/hive/common/ValidWriteIdList.java M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java M fe/src/main/java/org/apache/impala/catalog/FeTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java 15 files changed, 348 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/13215/16 -- To view, visit http://gerrit.cloudera.org:8080/13215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 Gerrit-Change-Number: 13215 Gerrit-PatchSet: 16 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13215 ) Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and partition .. Patch Set 16: submit patch-16, run core tests, no faiures. https://master-02.jenkins.cloudera.com/job/impala-private-parameterized/5051/ -- To view, visit http://gerrit.cloudera.org:8080/13215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 Gerrit-Change-Number: 13215 Gerrit-PatchSet: 16 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 May 2019 21:42:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7290: part 2: Add HS2 support to Impala shell
Hello Fredy Wijaya, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12884 to look at the new patch set (#16). Change subject: IMPALA-7290: part 2: Add HS2 support to Impala shell .. IMPALA-7290: part 2: Add HS2 support to Impala shell HS2 is the new default and the user-visible differences are minimal. Beeswax is still supported via --protocol=beeswax but will be deprecated. Differences are abstracted into ImpalaClient subclasses. This support requires Impala-specific extensions to the HS2 interface, similar to the existing extensions to Beeswax. Thus the HS2 shell is only forwards-compatible with newer Impala versions. I considered trying to gracefully degrade when the new extensions weren't present, but it didn't seem to be worth the ongoing testing effort. Here are the changes required to make it work: * Switch to TBinaryProtocolAccelerated to avoid perf regression. The HS2 protocol requires decoding more primitive values (because its not a string-per-row), which was slow with the pure python implementation of TBinaryProtocol. * Added bitarray module to efficiently unpack null indicators * Minimise invasiveness of changes by transposing and stringifying the columnar results into rows in impala_client.py. The transposition needs to happen before display anyway. * Add PingImpalaHS2Service() to get back version string and webserver address. * Add CloseImpalaOperation() extension to return DML row counts. * Add is_closed member to query handles to avoid shell independently tracking whether the query handle was closed or not. * Include query status in HS2 log to match beeswax. * HS2 GetLog() command now includes query status error message for consistency with beeswax. * "set"/"set all" uses the client requests options, not the session default. This captures the effective value of TIMEZONE, which was previously missing. This also requires test changes where the tests set non-default values, e.g. for ABORT_ON_ERROR. * "set all" on the server side returns REMOVED query options - the shell needs to know these so it can correctly ignore them. * Clean up self.orig_cmd/self.last_leading comment argument passing to avoid implicit parameter passing through multiple function calls. * Clean up argument handling in shell tests to consistently pass around lists of arguments instead of strings that are subject to shell tokenisation rules. Testing: * Shell tests can run with both protocols * Add tests for formatting of all types and NULL values * Added testing for floating point output formatting, which does change as a result of switching to server-side vs client-side formatting. * Verified that newly-added tests were actually going through HS2 by disabling hs2 on the minicluster and running tests. Performance: Baseline from beeswax shell for large extract is as follows: $ time impala-shell.sh -B -q 'select * from tpch_parquet.orders' > /dev/null real0m6.708s user0m5.132s sys 0m0.204s After this change it is somewhat slower, but we generally don't consider bulk extract performance through the shell to be perf-critical: real0m7.625s user0m6.436s sys 0m0.256s Change-Id: I6d5cc83d545aacc659523f29b1d6feed672e2a12 --- M LICENSE.txt M be/src/runtime/dml-exec-state.cc M be/src/runtime/dml-exec-state.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M bin/rat_exclude_files.txt M common/thrift/ImpalaService.thrift M infra/python/deps/compiled-requirements.txt A shell/ext-py/bitarray-0.9.0/PKG-INFO A shell/ext-py/bitarray-0.9.0/bitarray/__init__.py A shell/ext-py/bitarray-0.9.0/bitarray/_bitarray.c A shell/ext-py/bitarray-0.9.0/bitarray/test_bitarray.py A shell/ext-py/bitarray-0.9.0/setup.py M shell/impala_client.py M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/make_shell_tarball.sh M shell/option_parser.py M shell/thrift_sasl.py M testdata/workloads/functional-query/queries/QueryTest/set.test M tests/custom_cluster/test_shell_interactive.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/hs2/test_hs2.py M tests/query_test/test_observability.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 31 files changed, 7,233 insertions(+), 462 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/12884/16 -- To view, visit http://gerrit.cloudera.org:8080/12884 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6d5cc83d545aacc659523f29b1d6feed672e2a12 Gerrit-Change-Number: 12884 Gerrit-PatchSet: 16 Gerrit-Owner:
[Impala-ASF-CR] IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13215 ) Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and partition .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3243/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 Gerrit-Change-Number: 13215 Gerrit-PatchSet: 16 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 May 2019 22:15:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13215 ) Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and partition .. Patch Set 16: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 Gerrit-Change-Number: 13215 Gerrit-PatchSet: 16 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 May 2019 22:22:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13215 ) Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and partition .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4269/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 Gerrit-Change-Number: 13215 Gerrit-PatchSet: 17 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 May 2019 22:22:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13215 ) Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and partition .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 Gerrit-Change-Number: 13215 Gerrit-PatchSet: 17 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 May 2019 22:22:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12797 ) Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs .. Patch Set 17: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I839d7a3a2f5e1309c33a1f66013ef11628c5dc11 Gerrit-Change-Number: 12797 Gerrit-PatchSet: 17 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 15 May 2019 22:34:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12797 ) Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs .. IMPALA-4356,IMPALA-7331: codegen all ScalarExprs Based on initial draft patch by Pooja Nilangekar. Codegen'd expressions can be executed in two ways - either by being called directly from a fully codegend function, or from interpreted code via a function pointer (previously ScalarFnCall::scalar_fn_wrapper_). This change moves the function pointer from ScalarFnCall to its base class ScalarExpr, so the full expr tree can be codegen'd, not just the ScalarFnCall subtrees. The key refactoring and improvements are: * ScalarExpr::Get*Val() switches between interpreted and the codegen'd function pointer code paths in an inline function, avoiding a virtual function call to ScalarFnCal::Get*Val(). * Boilerplate logic is moved to ScalarExpr::GetCodegendComputeFn(), which calls a virtual function GetCodegenComputeFnImpl(). * ScalarFnCall's logic for deciding whether to interpret or codegen is better abstracted and exposed to ScalarExpr as IsInterpretable() and ShouldCodegen() methods. * The ScalarExpr::codegend_compute_fn_ function pointer is only populated for expressions that are "codegen entry points". These include the roots of expr trees and non-root expressions where the parent expression calls Get*Val() from the pseudo-codegend GetCodegendComputeFnWrapper(). * ScalarFnCall is always initialised for interpreted execution. Otherwise the function pointer is needed for non-root expressions, e.g. to support ScalarExprEvaluator::GetConstantVal(). * Latent bugs/gaps for codegen of CollectionVal are fixed. CollectionVal is modified to use the StringVal memory layout to allow code sharing with StringVal. These fixes allowed simplification of IsNotEmptyPredicate codegen (from IMPALA-7657). I chose to tackle two problems in one change - adding support for generating codegen'd function pointers for all ScalarExprs, and adding the "entry point" concept - to avoid a blow-up in the number of codegen'd entry points that could lead to longer codegen times and/or worse code because of inlining changes. IMPALA-7331 (CHAR codegen support functions) is also fixed because it was simpler to enable CHAR codegen within ScalarExpr than to carry forward the exiting CHAR workarounds from ScalarFnCall. The CHAR-specific codegen support required in the scalar expr subsystem is very limited. StringVal intermediates are used everywhere. Only SlotRef actually operates on the different tuple layout, and the required codegen support for SlotRef already exists for UDA intermediates anyway. Testing: * Ran exhaustive tests. Perf: * Ran a basic insert benchmark, which went from 10.1s to 7.6s create table foo stored as parquet as select case when l_orderkey % 2 = 0 then 'aaa' else 'bbb' end from tpch30_parquet.lineitem; * Ran a basic CHAR expr test: set num_nodes=1; set mt_dop=1; select count(*) from lineitem where cast(l_linestatus as CHAR(2)) = 'O ' and cast(l_returnflag as CHAR(2)) = 'N ' The time spent in the scan went from 520ms to 220ms. * Added perf regression test to tpcds-insert, similar to the manual benchmark. * Ran single-node TPC-H with large and small scale factors, to estimate impact on execution perf and query startup time, respectively. +--+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--+---+-++++ | TPCH(30) | parquet / none / none | 6.84| -0.18% | 4.49 | -0.31% | +--+---+-++++ +--+--+---++-++---++---++-++ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval | +--+--+---++-++---++---++-++ | TPCH(30) | TPCH-Q20 | parquet / none / none | 2.58 | 2.47| +4.18% | 1.29% | 0.88%| 5 | +4.12% | 2.31| 5.81 | | TPCH(30) | TPCH-Q17 | parquet / none / none | 4.81 | 4.61| +4.33% | 2.18% | 2.15%| 5 | +3.91% | 1.73| 3.09 | | TPCH(30) | TPCH-Q21 | parquet / none / none | 26.45 | 26.16 | +1.09% | 0.37% | 0.50%| 5 | +1.36% | 2.02| 3.94 | | TPCH(30) | TPCH-Q9 | parquet / none / none | 15.92 | 15.75 | +1.09% | 2.87% | 1.65%| 5
[Impala-ASF-CR] IMPALA-7290: part 2: Add HS2 support to Impala shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12884 ) Change subject: IMPALA-7290: part 2: Add HS2 support to Impala shell .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3244/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12884 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d5cc83d545aacc659523f29b1d6feed672e2a12 Gerrit-Change-Number: 12884 Gerrit-PatchSet: 16 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 15 May 2019 22:35:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Skip test owner privileges test when running against Hive-3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13339 ) Change subject: IMPALA-8369 : Skip test_owner_privileges test when running against Hive-3 .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4266/ -- To view, visit http://gerrit.cloudera.org:8080/13339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f904446f50b5095443bf27b3092a2e3665b76d3 Gerrit-Change-Number: 13339 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 22:52:18 + Gerrit-HasComments: No
[Impala-ASF-CR] Enable data cache by default for all S3 builds
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/13330 )
Change subject: Enable data cache by default for all S3 builds
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13330/2/bin/run-all-tests.sh
File bin/run-all-tests.sh:
http://gerrit.cloudera.org:8080/#/c/13330/2/bin/run-all-tests.sh@64
PS2, Line 64: /tmp
Do we know if the choice of filesystem matters for the cache? For example, /tmp
might be on tmpfs. Does it matter?
http://gerrit.cloudera.org:8080/#/c/13330/2/bin/run-all-tests.sh@78
PS2, Line 78: ${TARGET_FILESYSTEM}" == "s3"
I think the data caching is orthogonal to the filesystem, and we might want to
test with the filesystem as HDFS (e.g. on containers). So, maybe have an
DATA_CACHE_ENABLED flag that when true enables the cache. Then, the
customizable things would the directory and size. For automated tests, this can
be passed in as part of the test configuration.
--
To view, visit http://gerrit.cloudera.org:8080/13330
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I09117ab289c2355408212a5fc6493ab751f4853b
Gerrit-Change-Number: 13330
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Ho
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Lars Volker
Gerrit-Comment-Date: Wed, 15 May 2019 23:34:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Add USE CDP HIVE=true case to build-all-flag-combinations.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13335 ) Change subject: Add USE_CDP_HIVE=true case to build-all-flag-combinations.sh .. Patch Set 1: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4267/ -- To view, visit http://gerrit.cloudera.org:8080/13335 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f8e689242e20efb37fbadf7c04764ea8ffb9a9f Gerrit-Change-Number: 13335 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 16 May 2019 00:22:10 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix condition for starting YARN on USE CDP HIVE=true
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13343 ) Change subject: Fix condition for starting YARN on USE_CDP_HIVE=true .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Gerrit-Change-Number: 13343 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 16 May 2019 00:23:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8541: Remove existing roles in order to not interfere with a test.
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13349 Change subject: IMPALA-8541: Remove existing roles in order to not interfere with a test. .. IMPALA-8541: Remove existing roles in order to not interfere with a test. The test TestShortUsernameUsed() in AuthorizationTest.java would fail if the current user is associated with some role granted the privilege to access some databases, e.g., the database "functional". Specifically, for the SQL statement "select * from alltypes", if the role is already granted the privilege to access the database "functional", only AnalysisException will be thrown. To address this issue, we remove all the lingering roles before running this test. Testing: Passed all the tests in AuthorizationTest.java. Change-Id: I7122ee0cc90eaf7d45c5637c28c9cdb59502967b --- M fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java 1 file changed, 15 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/13349/1 -- To view, visit http://gerrit.cloudera.org:8080/13349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7122ee0cc90eaf7d45c5637c28c9cdb59502967b Gerrit-Change-Number: 13349 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya
[Impala-ASF-CR] [DOCS] Added back the note about Deflate not supported for text files.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13350 ) Change subject: [DOCS] Added back the note about Deflate not supported for text files. .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/328/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If44d3ad0653d73de030e8928077760a15ea18877 Gerrit-Change-Number: 13350 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 16 May 2019 00:24:20 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Added back the note about Deflate not supported for text files.
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13350 Change subject: [DOCS] Added back the note about Deflate not supported for text files. .. [DOCS] Added back the note about Deflate not supported for text files. - The text was removed in impala-7107. Putting it back per Sahil's request. Change-Id: If44d3ad0653d73de030e8928077760a15ea18877 --- M docs/topics/impala_file_formats.xml 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/13350/1 -- To view, visit http://gerrit.cloudera.org:8080/13350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If44d3ad0653d73de030e8928077760a15ea18877 Gerrit-Change-Number: 13350 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] Fix condition for starting YARN on USE CDP HIVE=true
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13343 ) Change subject: Fix condition for starting YARN on USE_CDP_HIVE=true .. Fix condition for starting YARN on USE_CDP_HIVE=true Dataload on Hive 3 uses YARN, but YARN is not needed for Hive 2 dataload. This fixes the condition so that YARN does not start for USE_CDP_HIVE=false (Hive 2). This fixes a similar condition for manipulating the classpath in testdata/bin/run-hive-server.sh. Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Reviewed-on: http://gerrit.cloudera.org:8080/13343 Reviewed-by: Todd Lipcon Tested-by: Impala Public Jenkins --- M testdata/bin/run-hive-server.sh M testdata/cluster/admin 2 files changed, 2 insertions(+), 2 deletions(-) Approvals: Todd Lipcon: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If2b0a529eadd5a436f0318229600180f72a27207 Gerrit-Change-Number: 13343 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-6042: Allow Impala shell to use a global impalarc config
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13313 )
Change subject: IMPALA-6042: Allow Impala shell to use a global impalarc config
..
Patch Set 3:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13313/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13313/3//COMMIT_MSG@9
PS3, Line 9: Currently, impalarc files can be specified on a per-user basis
(stored in ~/.impalarc), and they aren't created by default. The Impala shell
should pick up /etc/impalarc as well, in addition to the user-specific
configurations.
:
: The intent here is to allow a "global" configuration of the shell
by a system administrator.
nit: we should try to keep it within 72-char width
http://gerrit.cloudera.org:8080/#/c/13313/1/shell/impala_shell.py
File shell/impala_shell.py:
http://gerrit.cloudera.org:8080/#/c/13313/1/shell/impala_shell.py@1628
PS1, Line 1628: test_global_config = os.path.expanduser(options.global_config)
: if os.path.isfile(test_global_config):
: # Always output the source of the global config if verbose
: if options.verbose:
: print_to_stderr(
: "Loading in options from global config file: %s \n" %
test_global_config)
: if test_global_config != default_global_config:
: default_global_config = test_global_config
: elif test_global_config != default_global_config:
: print_to_stderr('%s not found.\n' % test_global_config)
: sys.exit(1)
:
: # use path to custom file specified by user in config_file
option
: input_config = os.path.expanduser(options.config_file)
: # verify input_config, if found
: if os.path.isfile(input_config) and input_config !=
default_user_config:
: if options.verbose:
: print_to_stderr("Loading in options from config file: %s
\n" % input_config)
: # command line overrides loading ~/.impalarc
: default_use
> Thanks for the heads up. I tried my best to make it more readable. It does
One confusion I had earlier was particularly this line: input_config !=
user_config: to mean file not found. I think this can be made more readable
like below:
if input_config != user_config:
if os.path.isfile(input_config):
if options.verbose:
print_to_stderr("Loading in options from config file: %s \n" %
input_config)
# Command line overrides loading ~/.impalarc
configs_to_load[1] = input_config
else:
print_to_stderr('%s not found.\n' % input_config)
sys.exit(1)
Now it's fairly easy to tell that the "else" means file not found :)
http://gerrit.cloudera.org:8080/#/c/13313/1/shell/impala_shell.py@1656
PS1, Line 1656:
nit: formatting is a bit odd. In Python, we usually indent parser.option_list
to be in the same col as config_file.
s_options, q_options = get_config_from_file(config_file,
parser.option_list)
http://gerrit.cloudera.org:8080/#/c/13313/1/shell/option_parser.py
File shell/option_parser.py:
http://gerrit.cloudera.org:8080/#/c/13313/1/shell/option_parser.py@260
PS1, Line 260: help=SUPPRESS_HELP
> The global config option is only used for testing purposes, so users should
One potential use case that maybe useful is is to specify a different location
of global config than /etc/impalarc. And currently there's no way to do this
with this CR other than using --global_config that's supposed to be a hidden
flag. Furthermore, having --global_config as an impala shell is kinda redundant
especially since we already have --config_file.
I think a better way is to use an environment variable, e.g.
IMPALA_CONFIG_FILE. With this, if we want to use a different global config file
than the default /etc/impalarc, users can specify the IMPALA_CONFIG_FILE in
/etc/profile or something of that sort. If IMPALA_CONFIG_FILE is not present,
it will simply use the /etc/impalarc. This has benefits of being able to
specify a different global config file as well as making testing easier.
What do you think?
--
To view, visit http://gerrit.cloudera.org:8080/13313
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a3179b6d9c9e3b2b01d6d3c5847cadb68782816
Gerrit-Change-Number: 13313
Gerrit-PatchSet: 3
Gerrit-Owner: Ethan Xue
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Ethan Xue
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Thu, 16 May 2019 00:32:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Added back the note about Deflate not supported for text files.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13350 ) Change subject: [DOCS] Added back the note about Deflate not supported for text files. .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/328/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If44d3ad0653d73de030e8928077760a15ea18877 Gerrit-Change-Number: 13350 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 16 May 2019 00:44:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8447: [DOCS] INSERT event is supported in automatic invalidation
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13300 ) Change subject: IMPALA-8447: [DOCS] INSERT event is supported in automatic invalidation .. Patch Set 1: (10 comments) http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml File docs/topics/impala_metadata.xml: http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@135 PS1, Line 135: notifications > should it be "notification events"? Done http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@146 PS1, Line 146: ADD, or DROP their : partitions > After IMPALA-7973, the add, drop and alter partition events trigger a refre Done http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@161 PS1, Line 161: Refreshes > I think may be we should also mention that if the table is not loaded, the Done. Do you mean if the table is not loaded because INSERT failed or no row was inserted? http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@198 PS1, Line 198: non-zero > may be say positive value to be accurate. Non-zero implies negative values Changed to "positive integer" http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@208 PS1, Line 208: such as adding new data to existing : tables/partitions from Spark > I think this is a very generic statement which may cause users to avoid usi Ok, will expand in a separate patch. Leaving this here for now, http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@221 PS1, Line 221: dump data directly > I think a better way to say this would be ".. and add or remove data into t Done http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@233 PS1, Line 233: Inserts into tables from Hive are ignored. > This can be misinterpreted as insert from Hive is not supported. I think in Removed the line http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@237 PS1, Line 237: invalidate > invalidated. Also, I think this is a implementation detail which do not nee Removed. http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@242 PS1, Line 242: Unlike other types of events, inserts by Impala triggers table refresh > This can be removed. I think its implementation detail and users may not ge Removed http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@267 PS1, Line 267: org.apache.hive.hcatalog.listener.DbNotificationListener > After IMPALA-7971, we also need hive.metastore.dml.events set to true Done -- To view, visit http://gerrit.cloudera.org:8080/13300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I68133b0beeb15cacc73829b8a8b0838fc7f4b7d8 Gerrit-Change-Number: 13300 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 16 May 2019 01:07:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8447: [DOCS] INSERT event is supported in automatic invalidation
Hello Vihang Karajgaonkar, Bharath Krishna, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13300 to look at the new patch set (#2). Change subject: IMPALA-8447: [DOCS] INSERT event is supported in automatic invalidation .. IMPALA-8447: [DOCS] INSERT event is supported in automatic invalidation - Added the INSERT events to the supported events. - Noted the limitation with inserts from SparkSQL. Change-Id: I68133b0beeb15cacc73829b8a8b0838fc7f4b7d8 --- M docs/topics/impala_metadata.xml 1 file changed, 34 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/13300/2 -- To view, visit http://gerrit.cloudera.org:8080/13300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I68133b0beeb15cacc73829b8a8b0838fc7f4b7d8 Gerrit-Change-Number: 13300 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API
Fang-Yu Rao has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13346 ) Change subject: IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API .. IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API Impala uses a workaround to detect if a user is a Sentry admin by by asking the Sentry API to list privileges associated with the user, since previously Sentry did not provide an suitable API to peform this check. SENTRY-2440 provides a new API (https://issues.apache.org/jira/browse/SENTRY-2440), and thus it would be good to update the Impala to make the code easier to understand. Testing: 1. Passed the tests in AuthorizationTest.java. 2. Passed the Sentry related tests in AuthorizationStmtTest.java. 3. Passed the tests in test_sentry.py. Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82 --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java 2 files changed, 20 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/13346/3 -- To view, visit http://gerrit.cloudera.org:8080/13346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82 Gerrit-Change-Number: 13346 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8447: [DOCS] INSERT event is supported in automatic invalidation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13300 ) Change subject: IMPALA-8447: [DOCS] INSERT event is supported in automatic invalidation .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/329/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I68133b0beeb15cacc73829b8a8b0838fc7f4b7d8 Gerrit-Change-Number: 13300 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 16 May 2019 01:15:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8447: [DOCS] INSERT event is supported in automatic invalidation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13300 ) Change subject: IMPALA-8447: [DOCS] INSERT event is supported in automatic invalidation .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/329/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I68133b0beeb15cacc73829b8a8b0838fc7f4b7d8 Gerrit-Change-Number: 13300 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 16 May 2019 01:08:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Fang-Yu Rao has uploaded a new patch set (#8). (
http://gerrit.cloudera.org:8080/12974 )
Change subject: IMPALA-7608: Estimate row count from file size when no stats
available
..
IMPALA-7608: Estimate row count from file size when no stats available
Added the feature that computes an estimated number of rows in the current
hdfs table if the statistics for the cardinality of the current hdfs table is
not
available.
Also added an additional query option to revert the change in case of
regression.
Testing:
(1) In CardinalityTest.java, replaced the original statement
"verifyCardinality("SELECT a FROM functional.tinytable", -1);"
with "verifyCardinality("SELECT a FROM functional.tinytable", 10);".
(2) In CardinalityTest.java, added a new test to ensure that
the returned cardinality is still -1 when this feature is disabled.
(3) In CarginalityTest.java, added more tests to check the cardinality
of most PlanNode implementations.
(4) In set.test, modified three related test cases to make sure that
the added query option is included after executing "set all" in various
scenarios.
(5) There are 8 JUnit tests in PlannerTest.java that would produce different
distributed query plans when this feature is enabled. Added an additional
JUnit test for each of those 8 affected JUnit tests when this feature is
enabled.
Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
R
testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-shuffle-hdfs-num-rows-est-disabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-shuffle-hdfs-num-rows-est-enabled.test
R
testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection-hdfs-num-rows-est-disabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection-hdfs-num-rows-est-enabled.test
R
testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-disabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
R
testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters-hdfs-num-rows-est-disabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters-hdfs-num-rows-est-enabled.test
R
testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation-hdfs-num-rows-est-disabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation-hdfs-num-rows-est-enabled.test
R
testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements-hdfs-num-rows-est-disabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements-hdfs-num-rows-est-enabled.test
R
testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing-hdfs-num-rows-est-disabled.test
C
testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing-hdfs-num-rows-est-enabled.test
R
testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite-hdfs-num-rows-est-disabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
26 files changed, 14,074 insertions(+), 75 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/12974/8
--
To view, visit http://gerrit.cloudera.org:8080/12974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a
Gerrit-Change-Number: 12974
Gerrit-PatchSet: 8
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8541: Remove existing roles in order to not interfere with a test.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13349 ) Change subject: IMPALA-8541: Remove existing roles in order to not interfere with a test. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3245/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7122ee0cc90eaf7d45c5637c28c9cdb59502967b Gerrit-Change-Number: 13349 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 16 May 2019 01:20:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13346 ) Change subject: IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3246/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82 Gerrit-Change-Number: 13346 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 16 May 2019 01:56:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12974 ) Change subject: IMPALA-7608: Estimate row count from file size when no stats available .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3247/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a Gerrit-Change-Number: 12974 Gerrit-PatchSet: 8 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 16 May 2019 02:05:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13215 ) Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and partition .. Patch Set 17: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4269/ -- To view, visit http://gerrit.cloudera.org:8080/13215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 Gerrit-Change-Number: 13215 Gerrit-PatchSet: 17 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 16 May 2019 03:42:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13215 )
Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and
partition
..
Patch Set 17:
> Patch Set 17: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4269/
The two failures are not related:
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/261/
The two test failures have the similar error message:
Exception: ("DB {0} didn't show up after {1}s", 'hms_sanity_db', 30)
Exception: ("DB {0} didn't show up after {1}s", 'hive_test_desc_db', 30)
Looks like the testing cluster has some performance issue
--
To view, visit http://gerrit.cloudera.org:8080/13215
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54
Gerrit-Change-Number: 13215
Gerrit-PatchSet: 17
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 16 May 2019 04:13:01 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13215 )
Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and
partition
..
Patch Set 17:
> Patch Set 17:
>
> > Patch Set 17: Verified-1
> >
> > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4269/
>
> The two failures are not related:
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/261/
> The two test failures have the similar error message:
> Exception: ("DB {0} didn't show up after {1}s", 'hms_sanity_db', 30)
> Exception: ("DB {0} didn't show up after {1}s", 'hive_test_desc_db', 30)
> Looks like the testing cluster has some performance issue
The failures are exact the same as the tests failures for a different
patch(which only has tests changes):
https://gerrit.cloudera.org/c/13339
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/258/
--
To view, visit http://gerrit.cloudera.org:8080/13215
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54
Gerrit-Change-Number: 13215
Gerrit-PatchSet: 17
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 16 May 2019 04:27:12 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API
Fang-Yu Rao has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13346 ) Change subject: IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API .. IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API Impala uses a workaround to detect if a user is a Sentry admin by calling the Sentry API to list privileges associated with the user since previously Sentry did not provide a suitable API to peform this check. This patch updates CDH_BUILD_NUMBER to 1075921 that contains a new API in SENTRY-2440 to perform a Sentry admin check. Testing: 1. Passed the tests in AuthorizationTest.java. 2. Passed the Sentry related tests in AuthorizationStmtTest.java. 3. Passed the tests in test_sentry.py. Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82 --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java 4 files changed, 26 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/13346/4 -- To view, visit http://gerrit.cloudera.org:8080/13346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82 Gerrit-Change-Number: 13346 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13346 ) Change subject: IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3248/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82 Gerrit-Change-Number: 13346 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 16 May 2019 05:40:03 + Gerrit-HasComments: No
