Review Request 64490: HIVE-14498

2017-12-10 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64490/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-14498
https://issues.apache.org/jira/browse/HIVE-14498


Repository: hive-git


Description
---

HIVE-14498


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
a0b163d19c381a20df28188ca678a19785e4e112 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 62c9172ef5d9ae74b158d1b4f1b8c5c0eca4e375 
  metastore/scripts/upgrade/derby/047-HIVE-14498.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 
f93d0d1d127156732a523fb45c3c2635d3ee530e 
  metastore/scripts/upgrade/derby/hive-txn-schema-3.0.0.derby.sql 
52713df30c66b34a3d5b815ca2814e7dca1e45a6 
  metastore/scripts/upgrade/derby/upgrade-2.3.0-to-3.0.0.derby.sql 
1f2647dfbf3263feda9afd98ab6767d7ea3d0557 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 
75891017584f93f94c55abcae10c512ad6bcb525 
  metastore/scripts/upgrade/mssql/032-HIVE-14498.mssql.sql PRE-CREATION 
  metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 
26c82af74c58cb097df8fa4d36a3b641602e1047 
  metastore/scripts/upgrade/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql 
864a5e5bd5c06810cab9d2f09d3b968845059a7a 
  metastore/scripts/upgrade/mysql/047-HIVE-14498.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 
915af8bf4bb7864371f5b9ba8bfb8a71a064ec36 
  metastore/scripts/upgrade/mysql/hive-txn-schema-3.0.0.mysql.sql 
1df32c4b3548d385e8861b9312042cc25bdf84d7 
  metastore/scripts/upgrade/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql 
caa059d893635e6afadab867760cb035b3a111d4 
  metastore/scripts/upgrade/oracle/047-HIVE-14498.oracle.sql PRE-CREATION 
  metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 
65c72af87343d84fe6eb894c503bebfe54fe4618 
  metastore/scripts/upgrade/oracle/hive-txn-schema-3.0.0.oracle.sql 
12c24a5863e0b8382c643f0404700e0243585db0 
  metastore/scripts/upgrade/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql 
33174c8a9a9bc768919ddda406324eda05aaf313 
  metastore/scripts/upgrade/postgres/046-HIVE-14498.postgres.sql PRE-CREATION 
  metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
415b5e0189bee95a104e3e3d9cd5f25187260a5b 
  metastore/scripts/upgrade/postgres/hive-txn-schema-3.0.0.postgres.sql 
1fa99aff5fcbbb96ed51b9c02850dfabcd6d3d76 
  metastore/scripts/upgrade/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql 
01d359e5f4c632793ac8e3fb67aa6ea5492dac54 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java 
6d48783d48581fb96ea1b5ded23ce0d549dc80a6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
55ef8de9a5c7144931d0a6ff13224765ee737fea 
  
ql/src/java/org/apache/hadoop/hive/ql/hooks/MaterializedViewRegistryUpdateHook.java
 a57e4c888b204388b393bb173e2eac91c867137a 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
50bdce89a44a8dc87a97e394d00e5dadebbbd351 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
 77a7b0f26bd21f5e36a2814a929513b3bdc6c9ce 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
ba64f97105eaaa0a5d3f2ad1883db42608c95ae8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
a09b7961c2dbc26b4d2fa912d0be7037885f63e4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
28e3621d3264f4f704da0d775b396f7b7764fdb6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 
09aa82f1f0c1a90b08669b91615f26fb1f7cd649 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java 
69b076a08a70fcea4f262ccbf9e063733ddd25f2 
  
ql/src/test/queries/clientnegative/materialized_view_no_transactional_rewrite.q 
PRE-CREATION 
  
ql/src/test/queries/clientnegative/materialized_view_no_transactional_rewrite_2.q
 PRE-CREATION 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q 
761903fd583cbe8fce003990b8d2e9534e126f2f 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_2.q 
9983bae7a1f4a478b10e85b18163d34be3ded3b8 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_3.q 
6462d9a677e9662c60455a79aa8ef522cba5fe65 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_4.q 
PRE-CREATION 
  
ql/src/test/queries/clientpositive/materialized_view_create_rewrite_multi_db.q 
e4cdc22e3b287bdfbfd05713ba9c6b851dcc3097 
  
ql/src/test/results/clientnegative/materialized_view_no_transactional_rewrite.q.out
 PRE-CREATION 
  
ql/src/test/results/clientnegative/materialized_view_no_transactional_rewrite_2.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite.q.out 
6db992128a4d2ff5c86f9e23b18e1d611a9bcc0d 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite_2.q.out 
3d3e05d489cd872cfe9cba65eab4f803c95f3286 
  ql/src/test/resul

[jira] [Created] (HIVE-18259) Automatic cleanup of invalidation cache for materialized views

2017-12-10 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-18259:
--

 Summary: Automatic cleanup of invalidation cache for materialized 
views
 Key: HIVE-18259
 URL: https://issues.apache.org/jira/browse/HIVE-18259
 Project: Hive
  Issue Type: Improvement
  Components: Materialized views
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


HIVE-14498 introduces the invalidation cache for materialized views, which 
keeps track of the transactions executed on a given table to infer whether 
materialized view contents are outdated or not.

Currently, the cache keeps information of transactions in memory to guarantee 
quick response time, i.e., quick resolution about the view freshness, at query 
rewriting time. This information can grow large, thus we would like to run a 
thread that cleans useless transactions from the cache, i.e., transactions that 
do invalidate any materialized view in the system, at an interval defined by a 
property.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18260) Add test case scenarios for materialized views invalidation cache and registry

2017-12-10 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-18260:
--

 Summary: Add test case scenarios for materialized views 
invalidation cache and registry
 Key: HIVE-18260
 URL: https://issues.apache.org/jira/browse/HIVE-18260
 Project: Hive
  Issue Type: Test
  Components: Materialized views
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


HIVE-14498 introduces the invalidation cache for materialized views at the 
metastore server and modifies the registry for materialized views in HS2 to 
work properly with it.

Beyond the ptests included with the patch, we need to add unit tests to 
simulate all different order of operations / concurrent operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18261) Load data failed as moving acls from source data to the target dir

2017-12-10 Thread Zac Zhou (JIRA) 
Zac Zhou created HIVE-18261:
---

 Summary: Load data failed as moving acls from source data to the 
target dir
 Key: HIVE-18261
 URL: https://issues.apache.org/jira/browse/HIVE-18261
 Project: Hive
  Issue Type: Bug
Reporter: Zac Zhou
Assignee: Zac Zhou


Running load data command, I got the following error:

[ERROR]2017-12-07 15:14:38,986, [Class]1512630876274, executeHqlSync method 
failed
java.sql.SQLException: Error while processing statement: FAILED: Execution 
Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. 
org.apache.hadoop.security.AccessControlException: Permission denied. user=*** 
is not the owner of inode=data
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:250)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:227)
at 
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:306)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1722)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1706)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:1675)
at org.apache.hadoop.hdfs.server.namenode.FSDirAclOp.setAcl(FSDirAclOp.java:145)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setAcl(FSNamesystem.java:7951)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setAcl(NameNodeRpcServer.java:1835)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setAcl(ClientNamenodeProtocolServerSideTranslatorPB.java:1330)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
at 
org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:349)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:251)
at 
com.netease.bdms.executor.executor.HQLResultManager.fetchResultsByJdbc(HQLResultManager.java:231)
at 
com.netease.bdms.executor.executor.HQLResultManager.saveFirstResultSet(HQLResultManager.java:341)
at 
com.netease.bdms.executor.executor.HQLJdbcJobRunnable.executeHqlSync(HQLJdbcJobRunnable.java:265)
at 
com.netease.bdms.executor.executor.HQLJdbcJobRunnable.runInternal(HQLJdbcJobRunnable.java:170)
at 
com.netease.bdms.executor.executor.HQLJdbcJobRunnable.run(HQLJdbcJobRunnable.java:67)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:748)

The job was failed, but the data had been moved into the target dir.

The root cause is Namenode check the path owner when seting acl. it will failed 
if the user is not the owner.

I think the job should not fail if failing to set acl, just like 
[HIVE-13716|https://issues.apache.org/jira/browse/HIVE-13716] 





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 64433: HIVE-18203 change the way WM is enabled and allow dropping the last resource plan

2017-12-10 Thread Harish Jaiprakash 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64433/#review193371
---




ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
Lines 939 (patched)


Yup, seem unintuitive to use enable in one and disable in another command. 
Is it better to add an extension to ALTER RESOURCE PLAN `plan_name` 
FORCE_DEACTIVATE;



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Line 9781 (original), 9802 (patched)


If alter is a rename then, the name displayed here would be new name and 
hence the old name was being passed to ensure errors are logged with current 
name.


- Harish Jaiprakash


On Dec. 8, 2017, 6:41 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64433/
> ---
> 
> (Updated Dec. 8, 2017, 6:41 a.m.)
> 
> 
> Review request for hive, Harish Jaiprakash and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  62c9172ef5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 55ef8de9a5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 8417ebb7d5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> dbdbbf25db 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 50bdce89a4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> a09b7961c2 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 1dcfe9dd28 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ResourcePlanParser.g 95c87259c5 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
> d70d52054d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> c58e4507f2 
>   ql/src/test/queries/clientpositive/resourceplan.q 002b21c1b9 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 093e5d58b6 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 223be6aa32 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6e0da5781e 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  fc254c6f53 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  573ac0173d 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  2e80c9d3b1 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  75fbfa23d2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  da518ab6e3 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 1085ce566a 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  24c59f2f1b 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  1e4fe5d973 
> 
> 
> Diff: https://reviews.apache.org/r/64433/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>