subject:"Review Request 69462\: HIVE\-20936"

Re: Review Request 69462: HIVE-20936

2018-12-21 Thread Eugene Koifman


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/#review211519
---


Ship it!




Ship It!

- Eugene Koifman


On Dec. 21, 2018, 4:30 p.m., Jaume Marhuenda wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69462/
> ---
> 
> (Updated Dec. 21, 2018, 4:30 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Allow the Worker thread in the metastore to run outside of it
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  b290a40734 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  d3800cdf2a 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java 852942e6a2 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 42ce1746fd 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
> f5b901d6e8 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> cdcc0e9548 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 49662cd68b 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java a3034fb195 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> 287aeaecb0 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 0c55654475 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AlterPartitionsRequest.java
>  d85dda5acd 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClearFileMetadataRequest.java
>  3eb55b1b59 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClientCapabilities.java
>  17f8b7730a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FindSchemasByColsResp.java
>  f2f8fb475e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventRequest.java
>  f7e188dfda 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse.java
>  bd38bbe45d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataByExprRequest.java
>  fb591dcec5 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataByExprResult.java
>  e8dfba523d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataRequest.java
>  3d32f372d6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataResult.java
>  2b176efee4 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  c0fe726f8a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectionSpec.java
>  db91e0bf89 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  d26cde23fc 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  3db9095b5c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetTablesRequest.java
>  c3f71fe13e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetTablesResult.java
>  5716922bd3 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InsertEventRequestData.java
>  3ef24310b2 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NotificationEventRequest.java
>  77e57718ef 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NotificationEventResponse.java
>  f559a03b92 
>   
> s

Re: Review Request 69462: HIVE-20936

2018-12-21 Thread Jaume Marhuenda



> On Dec. 21, 2018, 8:57 p.m., Eugene Koifman wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
> > Line 34 (original), 42 (patched)
> > 
> >
> > Why does this have more state fields than CompactorInfoStruct?  Perhaps 
> > it can done in HIVE-21056

Not all fields are serialized because not all are needed but I agree it's a 
good idea to so.


- Jaume


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/#review211508
---


On Dec. 22, 2018, 12:30 a.m., Jaume Marhuenda wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69462/
> ---
> 
> (Updated Dec. 22, 2018, 12:30 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Allow the Worker thread in the metastore to run outside of it
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  b290a40734 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  d3800cdf2a 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java 852942e6a2 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 42ce1746fd 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
> f5b901d6e8 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> cdcc0e9548 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 49662cd68b 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java a3034fb195 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> 287aeaecb0 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 0c55654475 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AlterPartitionsRequest.java
>  d85dda5acd 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClearFileMetadataRequest.java
>  3eb55b1b59 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClientCapabilities.java
>  17f8b7730a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FindSchemasByColsResp.java
>  f2f8fb475e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventRequest.java
>  f7e188dfda 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse.java
>  bd38bbe45d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataByExprRequest.java
>  fb591dcec5 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataByExprResult.java
>  e8dfba523d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataRequest.java
>  3d32f372d6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataResult.java
>  2b176efee4 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  c0fe726f8a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectionSpec.java
>  db91e0bf89 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  d26cde23fc 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  3db9095b5c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetTablesRequest.java
>  c3f71fe13e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetTablesResult.java
>  5716922bd3 
>

Re: Review Request 69462: HIVE-20936

2018-12-21 Thread Jaume Marhuenda


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/
---

(Updated Dec. 22, 2018, 12:30 a.m.)


Review request for hive.


Repository: hive-git


Description
---

Allow the Worker thread in the metastore to run outside of it


Diffs (updated)
-

  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 b290a40734 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 d3800cdf2a 
  jdbc/src/java/org/apache/hive/jdbc/Utils.java 852942e6a2 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
42ce1746fd 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
f5b901d6e8 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java cdcc0e9548 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 49662cd68b 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java a3034fb195 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 287aeaecb0 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 0c55654475 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AlterPartitionsRequest.java
 d85dda5acd 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClearFileMetadataRequest.java
 3eb55b1b59 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClientCapabilities.java
 17f8b7730a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FindSchemasByColsResp.java
 f2f8fb475e 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventRequest.java
 f7e188dfda 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse.java
 bd38bbe45d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataByExprRequest.java
 fb591dcec5 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataByExprResult.java
 e8dfba523d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataRequest.java
 3d32f372d6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataResult.java
 2b176efee4 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 c0fe726f8a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectionSpec.java
 db91e0bf89 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 d26cde23fc 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 3db9095b5c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetTablesRequest.java
 c3f71fe13e 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetTablesResult.java
 5716922bd3 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InsertEventRequestData.java
 3ef24310b2 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NotificationEventRequest.java
 77e57718ef 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NotificationEventResponse.java
 f559a03b92 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/OptionalCompactionInfoStruct.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PutFileMetadataRequest.java
 0bbb9be468 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RenamePartitionRequest.java
 0a0393ff14 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/me

Re: Review Request 69462: HIVE-20936

2018-12-21 Thread Eugene Koifman


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/#review211508
---



it looks like it has merge conflits


standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
Line 34 (original), 42 (patched)


Why does this have more state fields than CompactorInfoStruct?  Perhaps it 
can done in HIVE-21056



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 586 (patched)


"rj.getID().toString()" shouldn't be inside the quotes



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java
Lines 216 (patched)


this seems unused anywhere


- Eugene Koifman


On Dec. 20, 2018, 5:02 p.m., Jaume Marhuenda wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69462/
> ---
> 
> (Updated Dec. 20, 2018, 5:02 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Allow the Worker thread in the metastore to run outside of it
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  b290a40734 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  5af047f465 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 42ce1746fd 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
> f5b901d6e8 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> cdcc0e9548 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 49662cd68b 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java a3034fb195 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> 287aeaecb0 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 0c55654475 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AlterPartitionsRequest.java
>  d85dda5acd 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClearFileMetadataRequest.java
>  3eb55b1b59 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClientCapabilities.java
>  17f8b7730a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FindSchemasByColsResp.java
>  f2f8fb475e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventRequest.java
>  f7e188dfda 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse.java
>  bd38bbe45d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataByExprRequest.java
>  fb591dcec5 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataByExprResult.java
>  e8dfba523d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataRequest.java
>  3d32f372d6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataResult.java
>  2b176efee4 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  c0fe726f8a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectionSpec.java
>  db91e0bf89 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  d26cde23fc 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  3db9095b5c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetTablesRequest.

Re: Review Request 69462: HIVE-20936

2018-12-20 Thread Jaume Marhuenda


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/
---

(Updated Dec. 21, 2018, 1:02 a.m.)


Review request for hive.


Repository: hive-git


Description
---

Allow the Worker thread in the metastore to run outside of it


Diffs (updated)
-

  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 b290a40734 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 5af047f465 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
42ce1746fd 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
f5b901d6e8 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java cdcc0e9548 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 49662cd68b 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java a3034fb195 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 287aeaecb0 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 0c55654475 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AlterPartitionsRequest.java
 d85dda5acd 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClearFileMetadataRequest.java
 3eb55b1b59 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClientCapabilities.java
 17f8b7730a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FindSchemasByColsResp.java
 f2f8fb475e 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventRequest.java
 f7e188dfda 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse.java
 bd38bbe45d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataByExprRequest.java
 fb591dcec5 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataByExprResult.java
 e8dfba523d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataRequest.java
 3d32f372d6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataResult.java
 2b176efee4 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 c0fe726f8a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectionSpec.java
 db91e0bf89 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 d26cde23fc 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 3db9095b5c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetTablesRequest.java
 c3f71fe13e 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetTablesResult.java
 5716922bd3 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InsertEventRequestData.java
 3ef24310b2 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NotificationEventRequest.java
 77e57718ef 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/NotificationEventResponse.java
 f559a03b92 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/OptionalCompactionInfoStruct.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PutFileMetadataRequest.java
 0bbb9be468 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RenamePartitionRequest.java
 0a0393ff14 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SchemaVersion.java
 dd169327cf 
  
standalone-met

Re: Review Request 69462: HIVE-20936

2018-12-17 Thread Eugene Koifman


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/#review211381
---




ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
Line 184 (original), 184 (patched)


Just realized this needs a new metastore connection.  Thrift connections 
are not thread safe - when you mulitplex calls on a single connection, response 
messages sometimes get lost or matched to the wrong request. 

If you look at how Heartbeating is done in DbTxnHandler, it does something 
similar except that it relies on ThreadLocal in Hive.get(conf).getMSC().



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
Line 187 (original), 187 (patched)


why is this added here?  The CompactionHeartbeater should do this


- Eugene Koifman


On Dec. 17, 2018, 9:59 a.m., Jaume Marhuenda wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69462/
> ---
> 
> (Updated Dec. 17, 2018, 9:59 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Allow the Worker thread in the metastore to run outside of it
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  b290a40734 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  5af047f465 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java 852942e6a2 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 42ce1746fd 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
> f5b901d6e8 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> cdcc0e9548 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 21043415d3 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 546ff955b7 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> 52453a2ec4 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 0c55654475 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/OptionalCompactionInfoStruct.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  b6a0893524 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  3170798663 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  39f8b1f05a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  d57de353c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  a896849989 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  4ef4aadfee 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  97dc0696b7 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  a5f976bc5c 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  9eb1193a27 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  fa19440ba2 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  e25a8cf9a1 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> cb899d791f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  598847df03 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreThread.java
>  6ef2e3560d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
>  b34b7d70de 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
>  3f611d6270 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache

Re: Review Request 69462: HIVE-20936

2018-12-17 Thread Eugene Koifman


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/#review211379
---




standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
Lines 1019 (patched)


you can have Conf validate the values for you, for exaple 
MATERIALIZATIONS_INVALIDATION_CACHE_IMPL



standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 1079 (patched)


nit: it seems like a returning a (possibly) list of CompactionInfoStruct is 
simpler/easier to understand.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
Lines 172 (patched)


could you add a comment at the top next to the member variables to indicate 
that these methods should be modfied to be in sync


- Eugene Koifman


On Dec. 17, 2018, 9:59 a.m., Jaume Marhuenda wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69462/
> ---
> 
> (Updated Dec. 17, 2018, 9:59 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Allow the Worker thread in the metastore to run outside of it
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  b290a40734 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  5af047f465 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java 852942e6a2 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 42ce1746fd 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
> f5b901d6e8 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> cdcc0e9548 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 21043415d3 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 546ff955b7 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> 52453a2ec4 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 0c55654475 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/OptionalCompactionInfoStruct.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  b6a0893524 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  3170798663 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  39f8b1f05a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  d57de353c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  a896849989 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  4ef4aadfee 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  97dc0696b7 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  a5f976bc5c 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  9eb1193a27 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  fa19440ba2 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  e25a8cf9a1 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> cb899d791f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  598847df03 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreThread.java
>  6ef2e3560d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
>  b34b7d70de 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/util

Re: Review Request 69462: HIVE-20936

2018-12-17 Thread Jaume Marhuenda



> On Dec. 13, 2018, 12:04 a.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
> > Line 97 (original), 94 (patched)
> > 
> >
> > this is also intialzied in init() - should it throw here instead?  It 
> > seems that the contract is that init() must be called first

init() should be called first. This is part of the retry logic in case the msc 
is created again.


> On Dec. 13, 2018, 12:04 a.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
> > Lines 241 (patched)
> > 
> >
> > What is the purpose of this?  Why doesn't the existing catch(Throwable) 
> > with the same log msg work?

The logic there is to try restart the msc if we think something is wrong with 
it. I think that only (TException | IOException) would indicate that so we 
don't need to do it for every exception.


> On Dec. 13, 2018, 12:04 a.m., Eugene Koifman wrote:
> > service/src/java/org/apache/hive/service/server/HiveServer2.java
> > Lines 1016 (patched)
> > 
> >
> > Are there any tests that actually enable HIVE_MAPREDUCE_AVAILABLE ?

Hmm, the problem with this is that the Worker thread seems to already be tested 
without actually running in the metastore. It's just created with w = new 
Worker() and w.run()


> On Dec. 13, 2018, 12:04 a.m., Eugene Koifman wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
> > Lines 1476 (patched)
> > 
> >
> > I would put both of these in CompactionInfo.  If someone adds fields to 
> > CompactionInfo, they are unlikely to ever find these methods and so some 
> > info will be lost in the marshalling back and forth. 
> > 
> > Alternatively, could CompactionInfo be a subclass of 
> > CompactionInfoStruct?

I think it could be a subclass and it would be a clean approach, but I'd rather 
do this in a different patch. I'll move the methods to CompactionInfo.


- Jaume


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/#review211256
---


On Dec. 17, 2018, 5:59 p.m., Jaume Marhuenda wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69462/
> ---
> 
> (Updated Dec. 17, 2018, 5:59 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Allow the Worker thread in the metastore to run outside of it
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  b290a40734 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  5af047f465 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java 852942e6a2 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 42ce1746fd 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
> f5b901d6e8 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> cdcc0e9548 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 21043415d3 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 546ff955b7 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> 52453a2ec4 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 0c55654475 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/OptionalCompactionInfoStruct.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  b6a0893524 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  3170798663 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  39f8b1f05a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  d57de353c6 
>   
> standa

Re: Review Request 69462: HIVE-20936

2018-12-17 Thread Jaume Marhuenda


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/
---

(Updated Dec. 17, 2018, 5:59 p.m.)


Review request for hive.


Repository: hive-git


Description
---

Allow the Worker thread in the metastore to run outside of it


Diffs (updated)
-

  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 b290a40734 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 5af047f465 
  jdbc/src/java/org/apache/hive/jdbc/Utils.java 852942e6a2 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
42ce1746fd 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
f5b901d6e8 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java cdcc0e9548 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 21043415d3 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 546ff955b7 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 52453a2ec4 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 0c55654475 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/OptionalCompactionInfoStruct.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 b6a0893524 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 3170798663 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 39f8b1f05a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 d57de353c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 a896849989 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 4ef4aadfee 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 97dc0696b7 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 a5f976bc5c 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 9eb1193a27 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 fa19440ba2 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 e25a8cf9a1 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
cb899d791f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 598847df03 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreThread.java
 6ef2e3560d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 b34b7d70de 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 3f611d6270 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 9fe9a65677 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 2170178a81 


Diff: https://reviews.apache.org/r/69462/diff/3/

Changes: https://reviews.apache.org/r/69462/diff/2-3/


Testing
---


Thanks,

Jaume Marhuenda

Re: Review Request 69462: HIVE-20936

2018-12-12 Thread Eugene Koifman


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/#review211256
---




ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java
Lines 60 (patched)


@Override



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java
Line 80 (original), 79 (patched)


@Override



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java
Lines 35 (patched)


Perhaps add that this can run inside HMS as well.



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
Line 97 (original), 94 (patched)


this is also intialzied in init() - should it throw here instead?  It seems 
that the contract is that init() must be called first



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
Lines 241 (patched)


What is the purpose of this?  Why doesn't the existing catch(Throwable) 
with the same log msg work?



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 1016 (patched)


Are there any tests that actually enable HIVE_MAPREDUCE_AVAILABLE ?



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 1019 (patched)


Why do you need reflection for this?  Why not just do
Worker w = new Worker();, etc?



standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
Lines 1020 (patched)


Perhaps this should be called hive.metastore.runworker.remotely - you can 
run 'remote' worker even with MR or 'hive.metastore.runworker.in" and support 
values "metastore" and "hs2" - this is probable more extensivble



standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 2448 (patched)


nit: could you move these to around line 2312 where the rest of TxnStore 
methods are



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
Lines 1476 (patched)


I would put both of these in CompactionInfo.  If someone adds fields to 
CompactionInfo, they are unlikely to ever find these methods and so some info 
will be lost in the marshalling back and forth. 

Alternatively, could CompactionInfo be a subclass of CompactionInfoStruct?


- Eugene Koifman


On Dec. 11, 2018, 3:45 p.m., Jaume Marhuenda wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69462/
> ---
> 
> (Updated Dec. 11, 2018, 3:45 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Allow the Worker thread in the metastore to run outside of it
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  b290a40734 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  beb36d7674 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> c6cb7c5254 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
> f5b901d6e8 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> cdcc0e9548 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 4a1cac123c 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java dc39f5ef61 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> 52453a2ec4 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 0c55654475 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  b6a0893524 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  3170798663 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-ph

Re: Review Request 69462: HIVE-20936

2018-12-11 Thread Jaume Marhuenda


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/
---

(Updated Dec. 11, 2018, 11:45 p.m.)


Review request for hive.


Repository: hive-git


Description
---

Allow the Worker thread in the metastore to run outside of it


Diffs (updated)
-

  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 b290a40734 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 beb36d7674 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
c6cb7c5254 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
f5b901d6e8 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java cdcc0e9548 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 4a1cac123c 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java dc39f5ef61 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 52453a2ec4 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 0c55654475 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 b6a0893524 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 3170798663 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 39f8b1f05a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 d57de353c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 a896849989 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 4ef4aadfee 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 97dc0696b7 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 a5f976bc5c 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 9eb1193a27 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 fa19440ba2 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 e25a8cf9a1 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
cb899d791f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 598847df03 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreThread.java
 6ef2e3560d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 b34b7d70de 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 3f611d6270 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 9fe9a65677 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 2170178a81 


Diff: https://reviews.apache.org/r/69462/diff/2/

Changes: https://reviews.apache.org/r/69462/diff/1-2/


Testing
---


Thanks,

Jaume Marhuenda

Review Request 69462: HIVE-20936

2018-12-04 Thread Jaume Marhuenda


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/
---

Review request for hive.


Repository: hive-git


Description
---

Allow the Worker thread in the metastore to run outside of it


Diffs
-

  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 b290a40734 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 beb36d7674 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
c6cb7c5254 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
f5b901d6e8 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java cdcc0e9548 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 939f8c7538 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java dc39f5ef61 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 52453a2ec4 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 9376e87531 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 52a2b71cc4 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 efeaec7d6d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 bf6dfba0cc 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 d57de353c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 9ec122180d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 bdfb480723 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 46973bf39b 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 a5f976bc5c 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 a9f11fc073 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 fa19440ba2 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 f3a78bf5ec 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
48b633e5fa 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 23a78ca9c6 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreThread.java
 6ef2e3560d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 b34b7d70de 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 3f611d6270 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 9fe9a65677 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 2170178a81 


Diff: https://reviews.apache.org/r/69462/diff/1/


Testing
---


Thanks,

Jaume Marhuenda

Re: Review Request 69462: HIVE-20936

Re: Review Request 69462: HIVE-20936

Re: Review Request 69462: HIVE-20936

Re: Review Request 69462: HIVE-20936

Re: Review Request 69462: HIVE-20936

Re: Review Request 69462: HIVE-20936

Re: Review Request 69462: HIVE-20936

Re: Review Request 69462: HIVE-20936

Re: Review Request 69462: HIVE-20936

Re: Review Request 69462: HIVE-20936

Re: Review Request 69462: HIVE-20936

Review Request 69462: HIVE-20936

12 matches

Site Navigation

Mail list logo

Footer information