[jira] [Created] (HIVE-20399) CTAS w/a custom table location that is not fully qualified fails for MM tables

2018-08-15 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-20399:
---

 Summary: CTAS w/a custom table location that is not fully 
qualified fails for MM tables
 Key: HIVE-20399
 URL: https://issues.apache.org/jira/browse/HIVE-20399
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20400) create table should always use a fully qualified path to avoid potential FS ambiguity

2018-08-15 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-20400:
---

 Summary: create table should always use a fully qualified path to 
avoid potential FS ambiguity
 Key: HIVE-20400
 URL: https://issues.apache.org/jira/browse/HIVE-20400
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20398) [Hive Metastore] Add a Configuration Item to Skip the HDFS Data Modification

2018-08-15 Thread wuchang (JIRA) 
wuchang created HIVE-20398:
--

 Summary: [Hive Metastore] Add a Configuration Item to Skip the 
HDFS Data Modification
 Key: HIVE-20398
 URL: https://issues.apache.org/jira/browse/HIVE-20398
 Project: Hive
  Issue Type: Task
  Components: Metastore
Affects Versions: 2.3.2
Reporter: wuchang
Assignee: wuchang


When we are conducting the hive upgrading,  we have following use case:

We want to sync the operations between two metastore server (A and B) by thrift 
api, but both them are based on the same HDFS. So, for operations like 
*drop_partitions*, *drop_table*,  *insert_overwrite, create_table* which will 
cause the data modification in HDFS, we want it to be executed by only 
Metastore Server A. For metastore Server B, he will only change his metadata, 
but didn't do corresponding HDFS files operation.

So, we need a switch to control this.

like 

 
{code:java}
hive.metastore.skip.hdfs{code}
whose default value is *false* just like what is happening now.

When its value is true, the metastore server will only conduct the metadata 
modification, but skip the HDFS data modification.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68365: HIVE-19253: HMS ignores tableType property for external tables

2018-08-15 Thread Andrew Sherman via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68365/#review207383
---


Ship it!




Ship It!

- Andrew Sherman


On Aug. 15, 2018, 11:13 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68365/
> ---
> 
> (Updated Aug. 15, 2018, 11:13 p.m.)
> 
> 
> Review request for hive, Andrew Sherman, Peter Vary, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19253
> https://issues.apache.org/jira/browse/HIVE-19253
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19253: HMS ignores tableType property for external tables
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  8e2f94eb691a72cc4e6410aa76b176032d4af673 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  5233bee59220244e89f05b6c4dbf86a2cc6dc9fe 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  b74c3048fa2e18adc7f0d7cc813a180d4466fa36 
> 
> 
> Diff: https://reviews.apache.org/r/68365/diff/2/
> 
> 
> Testing
> ---
> 
> Added unit test to verify the expected behavior.
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 68365: HIVE-19253: HMS ignores tableType property for external tables

2018-08-15 Thread Alexander Kolbasov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68365/
---

(Updated Aug. 15, 2018, 11:13 p.m.)


Review request for hive, Andrew Sherman, Peter Vary, and Vihang Karajgaonkar.


Changes
---

Added extre check in MetastoreUtils to check for external table.


Bugs: HIVE-19253
https://issues.apache.org/jira/browse/HIVE-19253


Repository: hive-git


Description
---

HIVE-19253: HMS ignores tableType property for external tables


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 8e2f94eb691a72cc4e6410aa76b176032d4af673 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 5233bee59220244e89f05b6c4dbf86a2cc6dc9fe 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 b74c3048fa2e18adc7f0d7cc813a180d4466fa36 


Diff: https://reviews.apache.org/r/68365/diff/2/

Changes: https://reviews.apache.org/r/68365/diff/1-2/


Testing
---

Added unit test to verify the expected behavior.


Thanks,

Alexander Kolbasov

Re: Review Request 68359: HIVE-20393

2018-08-15 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68359/
---

(Updated Aug. 15, 2018, 10:19 p.m.)


Review request for hive, Gopal V, Jesús Camacho Rodríguez, and Jason Dere.


Changes
---

Less reliannt on NPE handling.


Bugs: HIVE-20393
https://issues.apache.org/jira/browse/HIVE-20393


Repository: hive-git


Description
---

See Jira.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java f316f09953 


Diff: https://reviews.apache.org/r/68359/diff/2/

Changes: https://reviews.apache.org/r/68359/diff/1-2/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-20397) HiveStrictManagedMigration updates

2018-08-15 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-20397:
-

 Summary: HiveStrictManagedMigration updates
 Key: HIVE-20397
 URL: https://issues.apache.org/jira/browse/HIVE-20397
 Project: Hive
  Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere


- Switch from using Driver instance to using metastore calls via 
Hive.alterDatabase/Hive.alterTable
- For tables converted from ORC to ACID tables, handle renaming of the files 
- Fix error handling so utility does not terminate after the first error 
encountered




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68365: HIVE-19253: HMS ignores tableType property for external tables

2018-08-15 Thread Alexander Kolbasov 


> On Aug. 15, 2018, 8:59 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
> > Lines 839 (patched)
> > 
> >
> > It seems weird that there isn't a constant for "EXTERNAL", but I can't 
> > find one.

Will add a constant as well.


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68365/#review207354
---


On Aug. 15, 2018, 8:18 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68365/
> ---
> 
> (Updated Aug. 15, 2018, 8:18 p.m.)
> 
> 
> Review request for hive, Andrew Sherman, Peter Vary, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19253
> https://issues.apache.org/jira/browse/HIVE-19253
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19253: HMS ignores tableType property for external tables
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  8e2f94eb691a72cc4e6410aa76b176032d4af673 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  b74c3048fa2e18adc7f0d7cc813a180d4466fa36 
> 
> 
> Diff: https://reviews.apache.org/r/68365/diff/1/
> 
> 
> Testing
> ---
> 
> Added unit test to verify the expected behavior.
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 68365: HIVE-19253: HMS ignores tableType property for external tables

2018-08-15 Thread Alexander Kolbasov 


> On Aug. 15, 2018, 8:59 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
> > Lines 854 (patched)
> > 
> >
> > So you test 
> > params(EXTERNAL=true),TableType.MANAGED_TABLE
> > params(EXTERNAL=false),TableType.EXTERNAL_TABLE
> > having done all the work you may as well test
> > params(EXTERNAL=false),TableType.MANAGED_TABLE
> > params(EXTERNAL=true),TableType.EXTERNAL_TABLE
> > as well?

Will add tests.


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68365/#review207354
---


On Aug. 15, 2018, 8:18 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68365/
> ---
> 
> (Updated Aug. 15, 2018, 8:18 p.m.)
> 
> 
> Review request for hive, Andrew Sherman, Peter Vary, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19253
> https://issues.apache.org/jira/browse/HIVE-19253
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19253: HMS ignores tableType property for external tables
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  8e2f94eb691a72cc4e6410aa76b176032d4af673 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  b74c3048fa2e18adc7f0d7cc813a180d4466fa36 
> 
> 
> Diff: https://reviews.apache.org/r/68365/diff/1/
> 
> 
> Testing
> ---
> 
> Added unit test to verify the expected behavior.
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 68365: HIVE-19253: HMS ignores tableType property for external tables

2018-08-15 Thread Andrew Sherman via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68365/#review207354
---




standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
Lines 839 (patched)


It seems weird that there isn't a constant for "EXTERNAL", but I can't find 
one.



standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
Lines 854 (patched)


So you test 
params(EXTERNAL=true),TableType.MANAGED_TABLE
params(EXTERNAL=false),TableType.EXTERNAL_TABLE
having done all the work you may as well test
params(EXTERNAL=false),TableType.MANAGED_TABLE
params(EXTERNAL=true),TableType.EXTERNAL_TABLE
as well?


- Andrew Sherman


On Aug. 15, 2018, 8:18 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68365/
> ---
> 
> (Updated Aug. 15, 2018, 8:18 p.m.)
> 
> 
> Review request for hive, Andrew Sherman, Peter Vary, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19253
> https://issues.apache.org/jira/browse/HIVE-19253
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19253: HMS ignores tableType property for external tables
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  8e2f94eb691a72cc4e6410aa76b176032d4af673 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  b74c3048fa2e18adc7f0d7cc813a180d4466fa36 
> 
> 
> Diff: https://reviews.apache.org/r/68365/diff/1/
> 
> 
> Testing
> ---
> 
> Added unit test to verify the expected behavior.
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Review Request 68365: HIVE-19253: HMS ignores tableType property for external tables

2018-08-15 Thread Alexander Kolbasov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68365/
---

Review request for hive, Andrew Sherman, Peter Vary, and Vihang Karajgaonkar.


Bugs: HIVE-19253
https://issues.apache.org/jira/browse/HIVE-19253


Repository: hive-git


Description
---

HIVE-19253: HMS ignores tableType property for external tables


Diffs
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 8e2f94eb691a72cc4e6410aa76b176032d4af673 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 b74c3048fa2e18adc7f0d7cc813a180d4466fa36 


Diff: https://reviews.apache.org/r/68365/diff/1/


Testing
---

Added unit test to verify the expected behavior.


Thanks,

Alexander Kolbasov

[jira] [Created] (HIVE-20396) Test HS2 open_connection metrics

2018-08-15 Thread Laszlo Pinter (JIRA) 
Laszlo Pinter created HIVE-20396:


 Summary: Test HS2 open_connection metrics
 Key: HIVE-20396
 URL: https://issues.apache.org/jira/browse/HIVE-20396
 Project: Hive
  Issue Type: Test
  Components: HiveServer2
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
 Fix For: 4.0.0


HiveServer2 is emitting metrics _default.General.open_connections_ in both 
binary and http mode. These metrics should be tested.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles

2018-08-15 Thread Igor Dvorzhak (JIRA) 
Igor Dvorzhak created HIVE-20395:


 Summary: Parallelize files move in the 
ql.metadata.Hive#replaceFiles
 Key: HIVE-20395
 URL: https://issues.apache.org/jira/browse/HIVE-20395
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.1.0
Reporter: Igor Dvorzhak
Assignee: Igor Dvorzhak


Files move is not parallelized in `replaceFiles` method and could be very slow 
on Cloud Object Stores.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20394) Cleaned up HBaseQTest runner

2018-08-15 Thread denys kuzmenko (JIRA) 
denys kuzmenko created HIVE-20394:
-

 Summary: Cleaned up HBaseQTest runner
 Key: HIVE-20394
 URL: https://issues.apache.org/jira/browse/HIVE-20394
 Project: Hive
  Issue Type: Bug
  Components: Tests
Affects Versions: 4.0.0
Reporter: denys kuzmenko
Assignee: denys kuzmenko


1. Set proper HBase cluster destroy order

2. General clean up



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68359: HIVE-20393

2018-08-15 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68359/
---

Review request for hive, Gopal V and Jesús Camacho Rodríguez.


Bugs: HIVE-20393
https://issues.apache.org/jira/browse/HIVE-20393


Repository: hive-git


Description
---

See Jira.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java f316f09953 


Diff: https://reviews.apache.org/r/68359/diff/1/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-20393) Semijoin Reduction : markSemiJoinForDPP behaves inconsistently

2018-08-15 Thread Deepak Jaiswal (JIRA) 
Deepak Jaiswal created HIVE-20393:
-

 Summary: Semijoin Reduction : markSemiJoinForDPP behaves 
inconsistently
 Key: HIVE-20393
 URL: https://issues.apache.org/jira/browse/HIVE-20393
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


markSemiJoinForDPP has multiple issues,

 
 * Uses map tsOps which is wrong as it disallows going thru same TS which may 
have filters from more than 1 semijoin edges. This results in inconsistent 
plans for same query as semijoin edges may be processed in different order each 
time.
 * Uses getColumnExpr() which is not as robust as extractColumn() thus 
resulting in NPEs.
 * The logic to mark an edge useful when NPE is hit may end up having bad edge.

cc [~gopalv]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)