[jira] [Created] (HIVE-20461) Metastore unit tests do not run with directSQL enabled

2018-08-24 Thread Vihang Karajgaonkar (JIRA) 
Vihang Karajgaonkar created HIVE-20461:
--

 Summary: Metastore unit tests do not run with directSQL enabled
 Key: HIVE-20461
 URL: https://issues.apache.org/jira/browse/HIVE-20461
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore, Test
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


Some of the Metastore tests do not use directSQL by default. We should have a 
mechanism to force usage of directSQL so that we have them covered atleast.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-24 Thread Vihang Karajgaonkar via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/
---

(Updated Aug. 25, 2018, 12:23 a.m.)


Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and Sergey 
Shelukhin.


Changes
---

Made changes suggested by Aihua


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306 : Implement projection spec for fetching only requested fields from 
partitions


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 47f96f323ade4e15753d6fd98709b9a882de624d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 a595732f04af4304974186178377192227bb80fb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 3c0d0a55b1dcca96dca77676873b68e52703715d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 7fc1e43de03eac1cfe802439ba38f83988299169 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 e54a7321e2e1e6069a3e598627bc6f6eaed93449 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
feb44d5159f131fae932739923b1a41f5e72e74b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 6b660f55aca5b99517447f164339ed92261c9370 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 faf6810420ccf6088a3398711c0c216af722f062 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d27224b23580b4662a85c874b657847ed068c9a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 b61ee81533930c889f23d2551041055cbdd1a6b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
 7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
 68f07e2569b6531cf3e18919209aed1a17e88bf7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 506d9ab5590a245049f3c4e71324823465a91586 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 5233bee59220244e89f05b6c4dbf86a2cc6dc9fe 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 0934aeb3a7d5413cacde500a5575e4f676306bd0

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-24 Thread Vihang Karajgaonkar via Review Board 


> On Aug. 24, 2018, 11:19 p.m., Aihua Xu wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Line 376 (original), 416 (patched)
> > 
> >
> > Why some functions like extractSqlString() not moving into 
> > MetastoreDirectSqlUtils class, but extractSqlLong() gets moved? Can we make 
> > it consistent?

I moved only the functions which were needed in PartitionProjectionEvaluator 
class. Makes sense to move these as well to keep it consistent.


> On Aug. 24, 2018, 11:19 p.m., Aihua Xu wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 547 (patched)
> > 
> >
> > Better add comments for this public function.

Added a javadoc


> On Aug. 24, 2018, 11:19 p.m., Aihua Xu wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 886 (patched)
> > 
> >
> > This function is a little confusing actually. Since we are not used 
> > anywhere else, probably we don't need to create such function seperately.

this was a left over from some of my code changes while development. Reverted 
this part.


> On Aug. 24, 2018, 11:19 p.m., Aihua Xu wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
> > Lines 200 (patched)
> > 
> >
> > This function can be static and probably gives the name find() as well? 
> > findUtil is a little confusing to me.

This method cannot be changed to static since it operates on the roots field 
which is non-static. renamed the findUtil to find as suggested.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207920
---


On Aug. 25, 2018, 12:23 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 25, 2018, 12:23 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>

[jira] [Created] (HIVE-20460) AcidUtils.Directory.getAbortedDirectories() may be missed for full CRUD tables

2018-08-24 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-20460:
-

 Summary: AcidUtils.Directory.getAbortedDirectories() may be missed 
for full CRUD tables
 Key: HIVE-20460
 URL: https://issues.apache.org/jira/browse/HIVE-20460
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


{{Directory.getAbortedDirectories()}} lists deltas where all txns in the range 
are aborted.

These are then purged by {{Worker}} (\{{CompactorMR}} but only for insert-only 
tables.

Full CRUD tables currently rely on {{FileSystem.rename()}} in {{MoveTask}} and 
so no reader (or {{Cleaner}} should every see a delta where all data is 
aborted.  

 

Once rename() is eliminated for full CRUD (just like insert-only) transactional 
tables, Cleaner (or Worker) should take care of these.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68509: HIVE-20451: Metastore client and server tarball issues

2018-08-24 Thread Alexander Kolbasov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68509/
---

Review request for hive, Alan Gates, Peter Vary, and Vihang Karajgaonkar.


Bugs: HIVE-20451
https://issues.apache.org/jira/browse/HIVE-20451


Repository: hive-git


Description
---

HIVE-20451: Metastore client and server tarball issues


Diffs
-

  standalone-metastore/metastore-common/pom.xml 
b0d9eee927e77728c27da95196d25f6ba1dfcb0c 
  standalone-metastore/metastore-common/src/assembly/bin.xml 
5992a484cf649d8028eb91db61a8a44e8f71cb8e 
  standalone-metastore/metastore-server/pom.xml 
f2c10b08cf08d194947f8d7737d745c9582dd837 
  standalone-metastore/pom.xml 99bf437e4c54721f76ecfc147e839498453ad60d 


Diff: https://reviews.apache.org/r/68509/diff/1/


Testing
---


Thanks,

Alexander Kolbasov

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-24 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207920
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Line 376 (original), 416 (patched)


Why some functions like extractSqlString() not moving into 
MetastoreDirectSqlUtils class, but extractSqlLong() gets moved? Can we make it 
consistent?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Lines 547 (patched)


Better add comments for this public function.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Lines 886 (patched)


This function is a little confusing actually. Since we are not used 
anywhere else, probably we don't need to create such function seperately.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
Lines 200 (patched)


This function can be static and probably gives the name find() as well? 
findUtil is a little confusing to me.


- Aihua Xu


On Aug. 24, 2018, 6:15 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 24, 2018, 6:15 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>

Review Request 68506: HIVE-20187

2018-08-24 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68506/
---

Review request for hive and Gunther Hagleitner.


Bugs: HIVE-20187
https://issues.apache.org/jira/browse/HIVE-20187


Repository: hive-git


Description
---

Incorrect query results in hive when hive.convert.join.bucket.mapjoin.tez is 
set to true
In some cases, Bucket mapjoin is incorrectly selected which leads to wrong 
results.


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java
 89db530f54 
  ql/src/test/queries/clientpositive/bucket_map_join_tez2.q adcf6962ab 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out 4f042cee50 
  ql/src/test/results/clientpositive/llap/tez_smb_main.q.out 9929989f0e 


Diff: https://reviews.apache.org/r/68506/diff/1/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-20459) add ThriftHiveMetastore.get_open_txns(long txnid)

2018-08-24 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-20459:
-

 Summary: add ThriftHiveMetastore.get_open_txns(long txnid)
 Key: HIVE-20459
 URL: https://issues.apache.org/jira/browse/HIVE-20459
 Project: Hive
  Issue Type: Improvement
  Components: Metastore, Transactions
Reporter: Eugene Koifman


we currently have {{ThriftHiveMetastore.get_open_txns()}} which maps to 
{{TxnHandler.getOpenTxns()}}.  The usual usage is 
{{TxnUtils.createValidReadTxnList(GetOpenTxnsResponse txns, long currentTxn)}} 
where the complete list transactions is obtained from Metastore and then 
anything above currentTxn is thrown away.  
Would be useful to add {{ThriftHiveMetastore.get_open_txns(long txnid)}} and 
{{TxnHandler.getOpenTxns(long)}} to not retrieve things that will be thrown 
away.  Especially when there are a lot of running transactions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20458) hive-schema-3.1.0.postgres.sql - some tables are not quoted

2018-08-24 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-20458:
-

 Summary: hive-schema-3.1.0.postgres.sql - some tables are not 
quoted
 Key: HIVE-20458
 URL: https://issues.apache.org/jira/browse/HIVE-20458
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore, Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman


a number of tables related to transactional metadata are not quoted in this 
script:

COMPACTION_QUEUE, HIVE_LOCKS, etc

this causes Postgres to create the tables in lower case.  The table creation 
scripts should follow the same convention as other tables.

hive-schema...mysql.sql also doesn't quote Create Table for acid meta tables




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-24 Thread Vihang Karajgaonkar via Review Board 


> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 852 (patched)
> > 
> >
> > Does this and other new methods belong here or in MetastoreServerUtils?
> 
> Vihang Karajgaonkar wrote:
> Its unclear what is the difference between MetaStoreUtils and 
> MetastoreServerUtils? Is MetaStoreUtils going to move to common in the near 
> future? If yes, I can move it to MetastoreServerUtils.

Moved them to MetastoreServerUtils.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207802
---


On Aug. 24, 2018, 6:15 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 24, 2018, 6:15 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d27224b23580b4662a85c874b657847ed068c9a3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-24 Thread Vihang Karajgaonkar via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/
---

(Updated Aug. 24, 2018, 6:15 p.m.)


Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and Sergey 
Shelukhin.


Changes
---

Added some of the suggested changes by Alexander


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306 : Implement projection spec for fetching only requested fields from 
partitions


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 7ab64eadac7948a7f5077260694926cc5b6e4e4b 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 cc19f2389e7b595722dcc1f3296877a02b20e0a4 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 a595732f04af4304974186178377192227bb80fb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 d098dba100bdfe56aa6ecafb31880098a2d7c6cb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 7fc1e43de03eac1cfe802439ba38f83988299169 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 92424a43feefc8c0db7c91302045437f3afbf274 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
8a4bdd8ed827572f5fd9d291c5454630d84284bd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 324035a8096a001d3fb170b8640805258d5e2cdd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 5ae00af564b05ec2720a03072f94c8f8579378a7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 91405b9a334a4b031a5dc7f4a1757a3895bfb386 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d27224b23580b4662a85c874b657847ed068c9a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 b61ee81533930c889f23d2551041055cbdd1a6b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
 7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
 68f07e2569b6531cf3e18919209aed1a17e88bf7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 5233bee59220244e89f05b6c4dbf86a2cc6dc9fe 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 0934aeb3a7d5413cacde500a5575e4f676306bd0 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 70a17f51b9b5a9fb0b5640988318fd39a82b895d

Re: Review Request 68496: Optimized & cleaned up HBaseQTest runner

2018-08-24 Thread denys kuzmenko via Review Board 


> On Aug. 24, 2018, 8:17 a.m., Peter Vary wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseCliDriver.java
> > Line 82 (original), 82 (patched)
> > 
> >
> > Since we removed clearTestSideEffects here, are we sure that 
> > clearPostTestEffects does the same removal?

This is how it was initially before the session lifecycle change (that is 
unrelated to cleanup, same way it's done downstream):
HIVE-19882: Fix QTestUtil session lifecycle (Zoltan Haindrich reviewed by Jason 
Dere) Zoltan Haindrich 2018. 06. 21. 6:15


> On Aug. 24, 2018, 8:17 a.m., Peter Vary wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseNegativeCliDriver.java
> > Line 78 (original), 77 (patched)
> > 
> >
> > Same as above. Does clearPostEffects is enough?

This is how it was initially before the session lifecycle change (that is 
unrelated to cleanup, same way it's done downstream):
HIVE-19882: Fix QTestUtil session lifecycle (Zoltan Haindrich reviewed by Jason 
Dere) Zoltan Haindrich 2018. 06. 21. 6:15


- denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68496/#review207858
---


On Aug. 24, 2018, 7:28 a.m., denys kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68496/
> ---
> 
> (Updated Aug. 24, 2018, 7:28 a.m.)
> 
> 
> Review request for hive, Marta Kuczora and Peter Vary.
> 
> 
> Bugs: HIVE-20394
> https://issues.apache.org/jira/browse/HIVE-20394
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 1. Set proper cluster destroy order
> 2. Propagated proper HBaseTestContext
> 3. Ported downstream fixes (CDH-63695)
> 4. General clean up
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/queries/negative/cascade_dbdrop.q 
> 48be8cd07018d005a28ad8070e13c51a46cf6d06 
>   hbase-handler/src/test/queries/positive/hbase_handler_snapshot.q 
> e4290717a85a462a45bc14e6295fa7eccad8a51d 
>   hbase-handler/src/test/results/negative/cascade_dbdrop.q.out 
> 803e35e40681df017a2f4e6d017b72749417d461 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java
>  31195c4523a35caf473eeecd8c8142cd1265ca18 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/accumulo/AccumuloQTestUtil.java
>  956478d778be03070b242ae45d340b5a99ac9316 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCoreBlobstoreCliDriver.java
>  3cf5ebb3df3455f9ebe701b79f5814e40f34ccc5 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreCliDriver.java
>  1ead1448d1e1a981068c4bdf5484346babb72ec0 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreCompareCliDriver.java
>  6b4c6c6a794da754bcaf8c4374fd9f85d51f318f 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseCliDriver.java
>  70cbf04823bb3d7208df245074eb9b732eadd18e 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseNegativeCliDriver.java
>  c76a70e72c13c2b42c32526398c639f041e3 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreNegativeCliDriver.java
>  07ae6ac206926c450c86b9519da469fc51f1d55d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CorePerfCliDriver.java
>  55e744e0f3d7b266593f4c8fb82eb8539ac0a563 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
> 07df0c9d1ed7fa3bf28325a1c37acf043d8ca848 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
> 7b203a928110101ba0de695acacadef1f1484b44 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestArguments.java 
> PRE-CREATION 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 5adbb63693b1f2fafa47939b2833310f5dd96bf2 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/ql/parse/CoreParseNegative.java
>  8f5744d2f1c8c059b4148aafde0558d76b723b65 
> 
> 
> Diff: https://reviews.apache.org/r/68496/diff/1/
> 
> 
> Testing
> ---
> 
> fixed existing tests
> 
> 
> Thanks,
> 
> denys kuzmenko
> 
>

[GitHub] hive pull request #421: HIVE-20447 Add JSON Outputformat support

2018-08-24 Thread Tmin10 
GitHub user Tmin10 opened a pull request:

https://github.com/apache/hive/pull/421

HIVE-20447 Add JSON Outputformat support

Based on https://github.com/julianhyde/sqlline/pull/84
Issue: https://issues.apache.org/jira/browse/HIVE-20447

1. JSON output format implementation including special symbols escaping 
based on [1] and [2];
2. Test for new output format;
3. Little refactoring in AbstractOutputFormat, XMLAttributeOutputFormat and 
XMLElementOutputFormat.

I also send request to take edit rights on Hive wiki to change 
documentation about beeline output formats.

[1] http://json.org
[2] 
https://docs.microsoft.com/en-us/sql/relational-databases/json/how-for-json-escapes-special-characters-and-control-characters-sql-server?view=sql-server-2017

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Tmin10/hive json-output

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/421.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #421


commit c37acb7bdab47e2c5d2dc73a04c0e13e6b2f8902
Author: Tmin10 
Date:   2018-08-23T13:07:19Z

HIVE-20447 Add JSON Outputformat support




---

[jira] [Created] (HIVE-20457) Create authorization mechanism for granting/revoking privileges to change Hive properties

2018-08-24 Thread Oleksiy Sayankin (JIRA) 
Oleksiy Sayankin created HIVE-20457:
---

 Summary: Create authorization mechanism for granting/revoking 
privileges to change Hive properties
 Key: HIVE-20457
 URL: https://issues.apache.org/jira/browse/HIVE-20457
 Project: Hive
  Issue Type: Improvement
  Components: Security
Reporter: Oleksiy Sayankin
Assignee: Oleksiy Sayankin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68496: Optimized & cleaned up HBaseQTest runner

2018-08-24 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68496/#review207858
---



Thank you Denis for the patch!
Nice work, just 1 question and several nit below.
Thanks,
Peter


itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseCliDriver.java
Line 63 (original)


nit: Unnecessary change. If we decide for any reason to create a new patch. 
Please remove.



itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseCliDriver.java
Lines 77 (patched)


nit: Unnecessary change. If we decide for any reason to create a new patch. 
Please remove.



itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseCliDriver.java
Line 82 (original), 82 (patched)


Since we removed clearTestSideEffects here, are we sure that 
clearPostTestEffects does the same removal?



itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseCliDriver.java
Line 111 (original)


nit: Unnecessary change. If we decide for any reason to create a new patch. 
Please remove.



itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseCliDriver.java
Line 113 (original)


nit: Unnecessary change. If we decide for any reason to create a new patch. 
Please remove.



itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseCliDriver.java
Lines 130 (patched)


nit: Unnecessary change. If we decide for any reason to create a new patch. 
Please remove.



itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseNegativeCliDriver.java
Line 78 (original), 77 (patched)


Same as above. Does clearPostEffects is enough?



itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseNegativeCliDriver.java
Lines 106 (patched)


nit: Unnecessary change. If we decide for any reason to create a new patch. 
Please remove.



itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseNegativeCliDriver.java
Line 129 (original)


nit: Unnecessary change. If we decide for any reason to create a new patch. 
Please remove.


- Peter Vary


On aug. 24, 2018, 7:28 de, denys kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68496/
> ---
> 
> (Updated aug. 24, 2018, 7:28 de)
> 
> 
> Review request for hive, Marta Kuczora and Peter Vary.
> 
> 
> Bugs: HIVE-20394
> https://issues.apache.org/jira/browse/HIVE-20394
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 1. Set proper cluster destroy order
> 2. Propagated proper HBaseTestContext
> 3. Ported downstream fixes (CDH-63695)
> 4. General clean up
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/queries/negative/cascade_dbdrop.q 
> 48be8cd07018d005a28ad8070e13c51a46cf6d06 
>   hbase-handler/src/test/queries/positive/hbase_handler_snapshot.q 
> e4290717a85a462a45bc14e6295fa7eccad8a51d 
>   hbase-handler/src/test/results/negative/cascade_dbdrop.q.out 
> 803e35e40681df017a2f4e6d017b72749417d461 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java
>  31195c4523a35caf473eeecd8c8142cd1265ca18 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/accumulo/AccumuloQTestUtil.java
>  956478d778be03070b242ae45d340b5a99ac9316 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCoreBlobstoreCliDriver.java
>  3cf5ebb3df3455f9ebe701b79f5814e40f34ccc5 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreCliDriver.java
>  1ead1448d1e1a981068c4bdf5484346babb72ec0 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreCompareCliDriver.java
>  6b4c6c6a794da754bcaf8c4374fd9f85d51f318f 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseCliDriver.java
>  70cbf04823bb3d7208df245074eb9b732eadd18e 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseNegativeCliDriver.java
>  c76a70e72c13c2b42c32526398c639f041e3 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreNegativeCliDriver.java
>  07ae6ac206926c450c86b9519da469fc51f1d55d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CorePerfCliDriver.java
>  55e744e0f3d7b266593f4c8fb82eb8539ac0a563 
>

[jira] [Created] (HIVE-20456) Query fails with FNFException using MR with skewjoin enabled and auto convert join disabled

2018-08-24 Thread Aditya Shah (JIRA) 
Aditya Shah created HIVE-20456:
--

 Summary: Query fails with FNFException using MR with skewjoin 
enabled and auto convert join disabled
 Key: HIVE-20456
 URL: https://issues.apache.org/jira/browse/HIVE-20456
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.1.0, 2.1.1, 1.2.0
Reporter: Aditya Shah
Assignee: Aditya Shah


When skew join is enabled and auto convert join is disabled the query fails 
with file not found exception. The following query reproduces the error:



set hive.optimize.skewjoin = true;
set hive.auto.convert.join = false;
set hive.groupby.orderby.position.alias = true;
set hive.on.master=true;
set hive.execution.engine=mr;

drop database if exists test cascade;
create database if not exists test;
use test;

CREATE EXTERNAL TABLE test_table1
( `a` int , `b` int, `c` int)
PARTITIONED BY (
`d` int)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

CREATE EXTERNAL TABLE test_table2
( `a` int , `b` int, `c` int)
PARTITIONED BY (
`d` int)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';


CREATE EXTERNAL TABLE test_table3
( `a` int , `b` int, `c` int)
PARTITIONED BY (
`e` int)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES (
'field.delim'='\u0001',
'serialization.format'='\u0001')
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';


CREATE EXTERNAL TABLE test_table4 (`a` int , `b` int, `c` int)
PARTITIONED BY (
`e` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES (
'field.delim'='\u0001',
'serialization.format'='\u0001')
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';


with
temp1 as (
select
g.a,
n.b,
u.c
from
test_table2 g
inner join test_table4 u on g.a = u.a
inner join test_table3 n on u.b = n.b
),
temp2 as (
select * from test_table4 where a = 2
),
temp21 as (
select
g.b,
n.c,
u.a
from
temp2 g
inner join test_table3 u on g.b = u.b
inner join test_table2 n on u.c = n.c
group by g.b, n.c, u.a
),
stack as (
select * from temp1
union all
select * from temp21
)
select * from stack;



The query runs perfectly fine when tez is used or other combinations of skew 
join and auto convert join are set. On diagnosing the issue, the problem was 
when a conditional task resolves tasks it puts the resolved task directly in 
the runnable state without checking the parental dependencies as well as 
whether the task is already queued.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68496: Optimized & cleaned up HBaseQTest runner

2018-08-24 Thread denys kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68496/
---

Review request for hive, Marta Kuczora and Peter Vary.


Bugs: HIVE-20394
https://issues.apache.org/jira/browse/HIVE-20394


Repository: hive-git


Description
---

1. Set proper cluster destroy order
2. Propagated proper HBaseTestContext
3. Ported downstream fixes (CDH-63695)
4. General clean up


Diffs
-

  hbase-handler/src/test/queries/negative/cascade_dbdrop.q 
48be8cd07018d005a28ad8070e13c51a46cf6d06 
  hbase-handler/src/test/queries/positive/hbase_handler_snapshot.q 
e4290717a85a462a45bc14e6295fa7eccad8a51d 
  hbase-handler/src/test/results/negative/cascade_dbdrop.q.out 
803e35e40681df017a2f4e6d017b72749417d461 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java
 31195c4523a35caf473eeecd8c8142cd1265ca18 
  
itests/util/src/main/java/org/apache/hadoop/hive/accumulo/AccumuloQTestUtil.java
 956478d778be03070b242ae45d340b5a99ac9316 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCoreBlobstoreCliDriver.java
 3cf5ebb3df3455f9ebe701b79f5814e40f34ccc5 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreCliDriver.java 
1ead1448d1e1a981068c4bdf5484346babb72ec0 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreCompareCliDriver.java
 6b4c6c6a794da754bcaf8c4374fd9f85d51f318f 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseCliDriver.java
 70cbf04823bb3d7208df245074eb9b732eadd18e 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreHBaseNegativeCliDriver.java
 c76a70e72c13c2b42c32526398c639f041e3 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreNegativeCliDriver.java
 07ae6ac206926c450c86b9519da469fc51f1d55d 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CorePerfCliDriver.java
 55e744e0f3d7b266593f4c8fb82eb8539ac0a563 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
07df0c9d1ed7fa3bf28325a1c37acf043d8ca848 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
7b203a928110101ba0de695acacadef1f1484b44 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestArguments.java 
PRE-CREATION 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
5adbb63693b1f2fafa47939b2833310f5dd96bf2 
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/parse/CoreParseNegative.java
 8f5744d2f1c8c059b4148aafde0558d76b723b65 


Diff: https://reviews.apache.org/r/68496/diff/1/


Testing
---

fixed existing tests


Thanks,

denys kuzmenko

[jira] [Created] (HIVE-20455) Log spew from security.authorization.PrivilegeSynchonizer.run

2018-08-24 Thread Gopal V (JIRA) 
Gopal V created HIVE-20455:
--

 Summary: Log spew from 
security.authorization.PrivilegeSynchonizer.run 
 Key: HIVE-20455
 URL: https://issues.apache.org/jira/browse/HIVE-20455
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


Typo in the class name aside, this keeps running and thrashing through every 
single DB in the cluster if not all directories are actually accessible to the 
user who runs HS2.

{code}
2018-08-24T03:14:41,932  INFO [PrivilegeSynchonizer] metastore.HiveMetaStore: 
1: get_table : tbl=hive.airline_ontime.flights_raw
2018-08-24T03:14:41,932  INFO [PrivilegeSynchonizer] HiveMetaStore.audit: 
ugi=gopal ip=unknown-ip-addr  cmd=get_table : 
tbl=hive.airline_ontime.flights_raw
{code}

The minimum fix needed is to move the Thread.sleep() outside of the try/catch 
block, so that the sleep is triggered even if the attempt throws an exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)