Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-05 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/
---

Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 5966740f88 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b1c2288d01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 


Diff: https://reviews.apache.org/r/66485/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-05 Thread Gopal V 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review200626
---




ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 334 (patched)


That's a REPL event & the trouble with IOW is that it also destroys commits 
in progress with the new base_n files, where n > all previous open txns.


- Gopal V


On April 6, 2018, 1:54 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 6, 2018, 1:54 a.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  5966740f88 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-06 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/
---

(Updated April 6, 2018, 9:31 p.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 5966740f88 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b1c2288d01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 22765b8e63 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 


Diff: https://reviews.apache.org/r/66485/diff/2/

Changes: https://reviews.apache.org/r/66485/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-16 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/
---

(Updated April 16, 2018, 10:35 p.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775286 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java a88453c978 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b1c2288d01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 22765b8e63 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
 7b02865e18 
  
storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java 
107ea9028a 


Diff: https://reviews.apache.org/r/66485/diff/3/

Changes: https://reviews.apache.org/r/66485/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-16 Thread Gopal V 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review201290
---




ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 352 (patched)


Table type?



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 1119 (patched)


add comment about this (As the "file that adds this commit").



storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
Lines 255 (patched)


Should probably return a new Object here (for sane debugging).


- Gopal V


On April 16, 2018, 10:35 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 16, 2018, 10:35 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java a88453c978 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 22765b8e63 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  7b02865e18 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
>  107ea9028a 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-17 Thread Sergey Shelukhin 


> On April 17, 2018, 4:44 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
> > Lines 1119 (patched)
> > 
> >
> > add comment about this (As the "file that adds this commit").

Hmm?


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review201290
---


On April 16, 2018, 10:35 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 16, 2018, 10:35 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java a88453c978 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 22765b8e63 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  7b02865e18 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
>  107ea9028a 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-17 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/
---

(Updated April 18, 2018, 1:37 a.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b8d948d3d3 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775286 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4acdd9b49b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b1c2288d01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 22765b8e63 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
 cb1d40a4a8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
 7b02865e18 
  
storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java 
107ea9028a 


Diff: https://reviews.apache.org/r/66485/diff/4/

Changes: https://reviews.apache.org/r/66485/diff/3-4/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-17 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review201371
---




standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
Lines 42 (patched)


this would need to be moved, it will be overwritten by thrift


- Sergey Shelukhin


On April 18, 2018, 1:37 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 18, 2018, 1:37 a.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b8d948d3d3 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4acdd9b49b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 22765b8e63 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
>  cb1d40a4a8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  7b02865e18 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
>  107ea9028a 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-17 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/
---

(Updated April 18, 2018, 1:46 a.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b8d948d3d3 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775286 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4acdd9b49b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b1c2288d01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 22765b8e63 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
 cb1d40a4a8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
 7b02865e18 
  
storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java 
107ea9028a 


Diff: https://reviews.apache.org/r/66485/diff/5/

Changes: https://reviews.apache.org/r/66485/diff/4-5/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-19 Thread Eugene Koifman 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review201574
---




ql/src/java/org/apache/hadoop/hive/ql/Driver.java
Lines 1324 (patched)


why not use TxnUtils.createValidCompactWriteIdList() and 
ValidCompactorWriteIdList.  It sets HWM to minOpen-1 - that is what it's for



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 1248 (patched)


should 'newBaseDir'?



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 1250 (patched)


Why is this OK?  The query produced no output?


- Eugene Koifman


On April 17, 2018, 6:46 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 17, 2018, 6:46 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b8d948d3d3 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4acdd9b49b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 22765b8e63 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
>  cb1d40a4a8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  7b02865e18 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
>  107ea9028a 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-19 Thread Sergey Shelukhin 


> On April 19, 2018, 10:48 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java
> > Lines 1324 (patched)
> > 
> >
> > why not use TxnUtils.createValidCompactWriteIdList() and 
> > ValidCompactorWriteIdList.  It sets HWM to minOpen-1 - that is what it's for

that seems to require different types that this code doesn't have... as far as 
I see it would need to call metastore or rebuild another object from scratch. 
There are too many (Table|Valid|WriteIds|*){3,} classes :)


> On April 19, 2018, 10:48 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
> > Lines 1250 (patched)
> > 
> >
> > Why is this OK?  The query produced no output?

This is the same commitJob in the compactor. I'm not sure if it can happen... 
perhaps from a number of empty inserts/ctas?


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review201574
---


On April 18, 2018, 1:46 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 18, 2018, 1:46 a.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b8d948d3d3 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4acdd9b49b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 22765b8e63 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
>  cb1d40a4a8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  7b02865e18 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
>  107ea9028a 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-19 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/
---

(Updated April 20, 2018, 1:42 a.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 536c7b427f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775286 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9cb2ff1015 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b1c2288d01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 22765b8e63 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
 cb1d40a4a8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
 7b02865e18 
  
storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java 
107ea9028a 


Diff: https://reviews.apache.org/r/66485/diff/6/

Changes: https://reviews.apache.org/r/66485/diff/5-6/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-20 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/
---

(Updated April 20, 2018, 11:15 p.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 536c7b427f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775286 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9cb2ff1015 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b1c2288d01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 22765b8e63 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
c053860b36 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
 cb1d40a4a8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
 7b02865e18 
  
storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java 
107ea9028a 


Diff: https://reviews.apache.org/r/66485/diff/7/

Changes: https://reviews.apache.org/r/66485/diff/6-7/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-22 Thread Gopal V 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review201717
---




ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 353 (patched)


Add a timestamp to the tmp-table and fail-retry if it already exists.

Dropping it might make it harder to debug this.



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 1236 (patched)


Add comment about not needing locks because these are insert-only tables 
and the base writer doesn't need locks anyway.



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 1257 (patched)


This + the next look looks a bit odd


- Gopal V


On April 20, 2018, 11:15 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 20, 2018, 11:15 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 536c7b427f 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9cb2ff1015 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 22765b8e63 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> c053860b36 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
>  cb1d40a4a8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  7b02865e18 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
>  107ea9028a 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/7/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-23 Thread Sergey Shelukhin 


> On April 23, 2018, 5:04 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
> > Lines 353 (patched)
> > 
> >
> > Add a timestamp to the tmp-table and fail-retry if it already exists.
> > 
> > Dropping it might make it harder to debug this.

This is a temporary table... it may be gone anyway. If not, this might 
necessitate a follow-up for cleanup of these failed tables.


> On April 23, 2018, 5:04 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
> > Lines 1236 (patched)
> > 
> >
> > Add comment about not needing locks because these are insert-only 
> > tables and the base writer doesn't need locks anyway.

Hmm? Can you elaborate? The tmp table is session scoped and not insert only, so 
we don't need any locks.
The original query locks are taken care of by driver. It's anyway a read query.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review201717
---


On April 20, 2018, 11:15 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 20, 2018, 11:15 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 536c7b427f 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9cb2ff1015 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 22765b8e63 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> c053860b36 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
>  cb1d40a4a8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  7b02865e18 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
>  107ea9028a 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/7/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-23 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/
---

(Updated April 23, 2018, 7:03 p.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2403d7ac6c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775286 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java a35a215bfc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 4e10649136 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b1c2288d01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 22765b8e63 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
c053860b36 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
 cb1d40a4a8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
 7b02865e18 
  
storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java 
107ea9028a 


Diff: https://reviews.apache.org/r/66485/diff/8/

Changes: https://reviews.apache.org/r/66485/diff/7-8/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-23 Thread Gopal V 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review201756
---




ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 356 (patched)


Is that supposed to be a "," or a +?


- Gopal V


On April 23, 2018, 7:03 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 23, 2018, 7:03 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2403d7ac6c 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java a35a215bfc 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 4e10649136 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 22765b8e63 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> c053860b36 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
>  cb1d40a4a8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  7b02865e18 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
>  107ea9028a 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/8/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-23 Thread Sergey Shelukhin 


> On April 23, 2018, 7:24 p.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
> > Lines 356 (patched)
> > 
> >
> > Is that supposed to be a "," or a +?

,


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review201756
---


On April 23, 2018, 7:03 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 23, 2018, 7:03 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2403d7ac6c 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java a35a215bfc 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 4e10649136 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 22765b8e63 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> c053860b36 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
>  cb1d40a4a8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  7b02865e18 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
>  107ea9028a 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/8/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-24 Thread Gopal V 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review201849
---


Ship it!




Ship It!

- Gopal V


On April 23, 2018, 7:03 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 23, 2018, 7:03 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2403d7ac6c 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java a35a215bfc 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 4e10649136 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 22765b8e63 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> c053860b36 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
>  cb1d40a4a8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  7b02865e18 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
>  107ea9028a 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/8/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-24 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/
---

(Updated April 25, 2018, 12:43 a.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java f40c60606c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775286 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java a35a215bfc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 4e10649136 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 7b7fd5d198 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 78eedd34f3 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java ec11fecba7 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b1c2288d01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java c95daafef5 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
c053860b36 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
 cb1d40a4a8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 1c8d22353f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 aee416da34 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
 7b02865e18 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 bf87cfcf7c 
  
storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java 
107ea9028a 


Diff: https://reviews.apache.org/r/66485/diff/9/

Changes: https://reviews.apache.org/r/66485/diff/8-9/


Testing
---


Thanks,

Sergey Shelukhin