Re: Review Request 68351: HIVE-20388: Move common classes out of metastore-server

2018-08-23 Thread Alexander Kolbasov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68351/
---

(Updated Aug. 24, 2018, 1:12 a.m.)


Review request for hive, Alan Gates, Peter Vary, and Vihang Karajgaonkar.


Changes
---

Merged with upstream


Bugs: HIVE-20388
https://issues.apache.org/jira/browse/HIVE-20388


Repository: hive-git


Description
---

HIVE-20388: Move common classes out of metastore-server


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 
f746d2dff32573470ee104311d014893205894d5 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java 
a50ec18b8aee0e24b632147f02c43526e0fe3071 
  ql/src/java/org/apache/hadoop/hive/ql/util/HiveStrictManagedMigration.java 
54df541c6646f1f6ca6cbedcd13cd31e56193332 
  service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 
d23e4389a770d11008d309d41780a8b692fc551a 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnCommonUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ColumnType.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 c551b80d8a57bdf11fb3cf7ea4e004b0e9f3995d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 e971d0f6e0d18696c06dc7233a886b462a94791b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 faf6810420ccf6088a3398711c0c216af722f062 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MaterializationsRebuildLockCleanerTask.java
 9ce7d6db497079ef874eef387d86d11d893d4470 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFS.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/TableIterable.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/conf/TimeValidator.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/CompositePartitionSpecProxy.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionListComposingSpecProxy.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecProxy.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecWithSharedSDProxy.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenIdentifier.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenSecretManager.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenSelector.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/HadoopThriftAuthBridge.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/HadoopThriftAuthBridge23.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/MetastoreDelegationTokenManager.java
 8d84fd7f323af1de4ac994863d81e5b06be5bcce 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 015dfd9ec971adcdece7b6b55c1962bc09f6edee 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
 bd202edb91adf8a4e5bcfa71d1f5dbcaba3d45c2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java
  
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 23ac7da98f08b63ec8e8a2f582e459c0da0059cb 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java

[jira] [Created] (HIVE-20454) CLONE - extend inheritPerms to ACID in Hive 1.X

2018-08-23 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-20454:
-

 Summary: CLONE - extend inheritPerms to ACID in Hive 1.X
 Key: HIVE-20454
 URL: https://issues.apache.org/jira/browse/HIVE-20454
 Project: Hive
  Issue Type: Bug
Reporter: Eugene Koifman
Assignee: Sergey Shelukhin
 Fix For: 2.4.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20453) backport HIVE-16233 (LLAP cache defragmentation) to branch-2

2018-08-23 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-20453:
---

 Summary: backport HIVE-16233 (LLAP cache defragmentation) to 
branch-2
 Key: HIVE-20453
 URL: https://issues.apache.org/jira/browse/HIVE-20453
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-23 Thread Vihang Karajgaonkar via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/
---

(Updated Aug. 23, 2018, 8:06 p.m.)


Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and Sergey 
Shelukhin.


Changes
---

added catName in the thrift definition for filterspec as suggested.


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306 : Implement projection spec for fetching only requested fields from 
partitions


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 7ab64eadac7948a7f5077260694926cc5b6e4e4b 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 cc19f2389e7b595722dcc1f3296877a02b20e0a4 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 a595732f04af4304974186178377192227bb80fb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 d098dba100bdfe56aa6ecafb31880098a2d7c6cb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 7fc1e43de03eac1cfe802439ba38f83988299169 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 92424a43feefc8c0db7c91302045437f3afbf274 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
8a4bdd8ed827572f5fd9d291c5454630d84284bd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 324035a8096a001d3fb170b8640805258d5e2cdd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 5ae00af564b05ec2720a03072f94c8f8579378a7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 91405b9a334a4b031a5dc7f4a1757a3895bfb386 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d27224b23580b4662a85c874b657847ed068c9a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 b61ee81533930c889f23d2551041055cbdd1a6b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
 7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
 68f07e2569b6531cf3e18919209aed1a17e88bf7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 5233bee59220244e89f05b6c4dbf86a2cc6dc9fe 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 0934aeb3a7d5413cacde500a5575e4f676306bd0 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 70a17f51b9b5a9fb0b5640988318fd39a82b895d

Re: Review Request 68476: HIVE-20433

2018-08-23 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68476/
---

(Updated Aug. 23, 2018, 8:05 p.m.)


Review request for hive, Ashutosh Chauhan and Gopal V.


Changes
---

Fixed ptest failures.


Bugs: HIVE-20433
https://issues.apache.org/jira/browse/HIVE-20433


Repository: hive-git


Description
---

Implicit String to Timestamp conversion is slow


Diffs (updated)
-

  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java
 8a057d1dab 


Diff: https://reviews.apache.org/r/68476/diff/2/

Changes: https://reviews.apache.org/r/68476/diff/1-2/


Testing
---


Thanks,

Deepak Jaiswal

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-23 Thread Vihang Karajgaonkar via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/
---

(Updated Aug. 23, 2018, 6:44 p.m.)


Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and Sergey 
Shelukhin.


Changes
---

Addressed some of the review comments. More to come.


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306 : Implement projection spec for fetching only requested fields from 
partitions


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 7ab64eadac7948a7f5077260694926cc5b6e4e4b 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 cc19f2389e7b595722dcc1f3296877a02b20e0a4 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 a595732f04af4304974186178377192227bb80fb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 d098dba100bdfe56aa6ecafb31880098a2d7c6cb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 7fc1e43de03eac1cfe802439ba38f83988299169 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 92424a43feefc8c0db7c91302045437f3afbf274 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
8a4bdd8ed827572f5fd9d291c5454630d84284bd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 324035a8096a001d3fb170b8640805258d5e2cdd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 5ae00af564b05ec2720a03072f94c8f8579378a7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 91405b9a334a4b031a5dc7f4a1757a3895bfb386 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d27224b23580b4662a85c874b657847ed068c9a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 b61ee81533930c889f23d2551041055cbdd1a6b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
 7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
 68f07e2569b6531cf3e18919209aed1a17e88bf7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 5233bee59220244e89f05b6c4dbf86a2cc6dc9fe 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 0934aeb3a7d5413cacde500a5575e4f676306bd0 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 70a17f51b9b5a9fb0b5640988318fd39a82b895d

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-23 Thread Vihang Karajgaonkar via Review Board 


> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
> > Lines 1646 (patched)
> > 
> >
> > does this allow for x.y.z?
> > Does this work for sd columns and other list fields?

yes, this works for multi-valued fields, so something like sd.cols.name will 
work. Only name will be set for all the sd.cols for each partition. I will add 
more documentation here to make it clearer.


> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
> > Lines 1647 (patched)
> > 
> >
> > Should this be optional?
> > Is this a regexp or something else? What is the syntax here? What are _ 
> > or % wildcards?

This does not need to be optional, the default requiredness of the field makes 
sure that thrift will always check if this field is set and on the writers will 
serialize the field only when its possible (null value is not serialized). Will 
update the documentation


> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
> > Lines 1648 (patched)
> > 
> >
> > Why not just have optional include and exclude patterns instead of a 
> > pattern and a boolean?

Thats an interesting thought. Do you think it would be useful for clients to 
have something like PARAM_KEY LIKE ('includepattern') AND NOT LIKE 
('excludepattern')? It should not be too difficult to add to have two separate 
include and exclude paramKeyPattern.


> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
> > Lines 1658 (patched)
> > 
> >
> > Do we need catalog name?

Good point. I will add it.


> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 56 (patched)
> > 
> >
> > This shows as an unused import

removed it


> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 65 (patched)
> > 
> >
> > Unused import?

removed it.


> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 852 (patched)
> > 
> >
> > Does this and other new methods belong here or in MetastoreServerUtils?

Its unclear what is the difference between MetaStoreUtils and 
MetastoreServerUtils? Is MetaStoreUtils going to move to common in the near 
future? If yes, I can move it to MetastoreServerUtils.


> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 853 (patched)
> > 
> >
> > It would be good to explain what this method actually does converts 
> > list of objects to a comma-separated string of stringified objects. Then 
> > you can explain the intended use case.

This method is used only in MetastoreDirectSQL class. Moved it back to that 
class. It was probably a left over from the many iterations of my code changes. 
Updated the documentations as suggested.


> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 950 (patched)
> > 
> >
> > What is this class?

This class was originally in HMSHandler and used to grouping Partitions based 
on a common StorageDescriptor. In order to do that, this class is used to 
generate the hashKey of partitions so that all the partitions having the same 
hashcode can be grouped together. The hashCode is overridden such that if the 
sd.cols, input/outputformat, serializationLib and base location is same, it 
returns the same hashCode so that all such partitions can be grouped together.

I had to move this class from HMSHandler to MetaStoreUtils so that this code 
could be reused by the new API as well as the previously existing APIs using 
this grouping logic. I think I can move

[jira] [Created] (HIVE-20452) DPP: Simple n-tuple expressions support

2018-08-23 Thread Gopal V (JIRA) 
Gopal V created HIVE-20452:
--

 Summary: DPP: Simple n-tuple expressions support
 Key: HIVE-20452
 URL: https://issues.apache.org/jira/browse/HIVE-20452
 Project: Hive
  Issue Type: Improvement
  Components: Tez
Affects Versions: 4.0.0
Reporter: Gopal V


{code}
create table sales (s_qty bigint, s_item_sk bigint) partitioned by(s_region 
string, s_date date);

insert into sales partition(s_region='USA', s_date='2018-01-01') values (1,1), 
(2,2);
insert into sales partition(s_region='EU', s_date='2018-02-01') values (1,1), 
(2,2);
{code}

{code}

select count(*) from sales s, agg_sales a where concat(a.s_region, '^', 
a.s_date) = concat(s.s_region, '^', s.s_date) and s.sold_qty > a.median_s_qty 
and a.s_item_sk = s.s_item_sk;

select count(*) from sales s, agg_sales a where (a.s_region, a.s_date) = 
(s.s_region, s.s_date) and s.sold_qty > a.median_s_qty and a.s_item_sk = 
s.s_item_sk; 

select count(*) from sales s, agg_sales a where a.s_region = s.s_region and  
a.s_date = s.s_date and s.sold_qty > a.median_s_qty and a.s_item_sk = 
s.s_item_sk;

{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20451) Metastore client and server tarball issues

2018-08-23 Thread Alan Gates (JIRA) 
Alan Gates created HIVE-20451:
-

 Summary: Metastore client and server tarball issues
 Key: HIVE-20451
 URL: https://issues.apache.org/jira/browse/HIVE-20451
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Affects Versions: 4.0.0
Reporter: Alan Gates


With the split of the metastore into common and server there are now two sets 
of tarballs.  There are a couple of issues here.
 # It doesn't make sense to have separate source tarballs for each.  The source 
release should still be done from the standalone-metastore directory and 
include all code for the metastore.
 # The binary tarballs should have separate names.  At the moment both are 
named apache-hive-metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20450) Add replication test for LOAD command on ACID table.

2018-08-23 Thread Sankar Hariappan (JIRA) 
Sankar Hariappan created HIVE-20450:
---

 Summary: Add replication test for LOAD command on ACID table.
 Key: HIVE-20450
 URL: https://issues.apache.org/jira/browse/HIVE-20450
 Project: Hive
  Issue Type: Bug
  Components: repl
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan


Add replication test for LOAD command on ACID/MM table.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20449) DruidMiniTests - Move creation of druid table from allTypesOrc to test setup phase

2018-08-23 Thread Nishant Bangarwa (JIRA) 
Nishant Bangarwa created HIVE-20449:
---

 Summary: DruidMiniTests - Move creation of druid table from 
allTypesOrc to test setup phase
 Key: HIVE-20449
 URL: https://issues.apache.org/jira/browse/HIVE-20449
 Project: Hive
  Issue Type: Improvement
Reporter: Nishant Bangarwa
Assignee: Nishant Bangarwa


Multiple druid tests end up creating a Druid table from allTypesOrc table. 
Moving this table creation to a pre-test setup phase would avoid redundant work 
in tests and possibly help in reducing test runtimes. 

Thanks, [~jcamachorodriguez] for suggesting this improvement. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20448) get_json_object() unnecessarily unnests array values

2018-08-23 Thread JIRA 
Zoltán Borók-Nagy created HIVE-20448:


 Summary: get_json_object() unnecessarily unnests array values
 Key: HIVE-20448
 URL: https://issues.apache.org/jira/browse/HIVE-20448
 Project: Hive
  Issue Type: Bug
Reporter: Zoltán Borók-Nagy


Hive's implementation of get_json_object() unnecessarily unnests array values, 
e.g.:
{noformat}
hive> select get_json_object('[{"key": [1,2,3]}, {"key": [4,5]}]', '$[*].key');
OK
[1,2,3,4,5]{noformat}
I think for the above query the following output is more reasonable:
{noformat}
[[1,2,3],[4,5]]{noformat}
This is also the output that one gets from
 * [https://jsonpath.curiousconcept.com/] 
 * [http://jsonpath.com/]
 * [https://jsonpath.herokuapp.com/]

Python's jsonpath package also produces this output.

I tested it with Hive 2.1.1, but it is likely that it affects other versions as 
well.

If we'd really want to unnest the arrays, then '$[*].key[*]' is the way to go. 
It produces the same output everywhere.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20447) Add JSON Outputformat support

2018-08-23 Thread Max (JIRA) 
Max created HIVE-20447:
--

 Summary: Add JSON Outputformat support
 Key: HIVE-20447
 URL: https://issues.apache.org/jira/browse/HIVE-20447
 Project: Hive
  Issue Type: Task
  Components: Beeline
Reporter: Max
Assignee: Max


This function is present in SQLLine. We need add it to beeline too.

https://github.com/julianhyde/sqlline/pull/84



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20446) CachedStore: bug fixes for q file tests: TestMiniLlapCliDriver, TestMiniTezCliDriver, TestMinimrCliDriver when CachedStore is enabled

2018-08-23 Thread Vaibhav Gumashta (JIRA) 
Vaibhav Gumashta created HIVE-20446:
---

 Summary: CachedStore: bug fixes for q file tests: 
TestMiniLlapCliDriver, TestMiniTezCliDriver, TestMinimrCliDriver when 
CachedStore is enabled
 Key: HIVE-20446
 URL: https://issues.apache.org/jira/browse/HIVE-20446
 Project: Hive
  Issue Type: Sub-task
  Components: Standalone Metastore
Affects Versions: 3.1.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20445) Add randomized tests to TestArrowColumnarBatchSerDe

2018-08-23 Thread Teddy Choi (JIRA) 
Teddy Choi created HIVE-20445:
-

 Summary: Add randomized tests to TestArrowColumnarBatchSerDe
 Key: HIVE-20445
 URL: https://issues.apache.org/jira/browse/HIVE-20445
 Project: Hive
  Issue Type: Test
Reporter: Teddy Choi
Assignee: Teddy Choi


Use random schemas/data for TestArrowColumnarBatchSerDe to improve coverage of 
arrow conversion/serialization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-23 Thread Alexander Kolbasov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207802
---



Some initial comments.


standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 1645 (patched)


Please add comment explaining what this is.



standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 1646 (patched)


does this allow for x.y.z?
Does this work for sd columns and other list fields?



standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 1647 (patched)


Should this be optional?
Is this a regexp or something else? What is the syntax here? What are _ or 
% wildcards?



standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 1648 (patched)


Why not just have optional include and exclude patterns instead of a 
pattern and a boolean?



standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 1658 (patched)


Do we need catalog name?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
Lines 56 (patched)


This shows as an unused import



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
Lines 65 (patched)


Unused import?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
Lines 852 (patched)


Does this and other new methods belong here or in MetastoreServerUtils?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
Lines 853 (patched)


It would be good to explain what this method actually does converts list of 
objects to a comma-separated string of stringified objects. Then you can 
explain the intended use case.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
Lines 950 (patched)


What is this class?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
Lines 953 (patched)


Do we need this and other fields? Why can't we just use 
this.sd.outputFormat, etc?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
Lines 978 (patched)


Why do we need an explicit empty constructor?


- Alexander Kolbasov


On Aug. 21, 2018, 6:50 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 21, 2018, 6:50 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>