[jira] [Created] (HIVE-21676) use a system table as an alternative proc store

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-21676:
---

 Summary: use a system table as an alternative proc store
 Key: HIVE-21676
 URL: https://issues.apache.org/jira/browse/HIVE-21676
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


We keep hitting these issues:
{noformat}
2019-04-30 23:41:52,164 INFO  [master/master:17000:becomeActiveMaster] 
procedure2.ProcedureExecutor: Starting 16 core workers (bigger of cpus/4 or 16) 
with max (burst) worker count=160
2019-04-30 23:41:52,171 INFO  [master/master:17000:becomeActiveMaster] 
util.FSHDFSUtils: Recover lease on dfs file 
.../MasterProcWALs/pv2-0481.log
2019-04-30 23:41:52,176 INFO  [master/master:17000:becomeActiveMaster] 
util.FSHDFSUtils: Recovered lease, attempt=0 on 
file=.../MasterProcWALs/pv2-0481.log after 5ms
2019-04-30 23:41:52,288 INFO  [master/master:17000:becomeActiveMaster] 
util.FSHDFSUtils: Recover lease on dfs file 
.../MasterProcWALs/pv2-0482.log
2019-04-30 23:41:52,289 INFO  [master/master:17000:becomeActiveMaster] 
util.FSHDFSUtils: Recovered lease, attempt=0 on 
file=.../MasterProcWALs/pv2-0482.log after 1ms
2019-04-30 23:41:52,373 INFO  [master/master:17000:becomeActiveMaster] 
wal.WALProcedureStore: Rolled new Procedure Store WAL, id=483
2019-04-30 23:41:52,375 INFO  [master/master:17000:becomeActiveMaster] 
procedure2.ProcedureExecutor: Recovered WALProcedureStore lease in 206msec
2019-04-30 23:41:52,782 INFO  [master/master:17000:becomeActiveMaster] 
wal.ProcedureWALFormatReader: Read 1556 entries in 
.../MasterProcWALs/pv2-0482.log
2019-04-30 23:41:55,370 INFO  [master/master:17000:becomeActiveMaster] 
wal.ProcedureWALFormatReader: Read 28113 entries in 
.../MasterProcWALs/pv2-0481.log
2019-04-30 23:41:55,384 ERROR [master/master:17000:becomeActiveMaster] 
wal.WALProcedureTree: Missing stack id 166, max stack id is 181, root procedure 
is Procedure(pid=289380, ppid=-1, 
class=org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure)
2019-04-30 23:41:55,384 ERROR [master/master:17000:becomeActiveMaster] 
wal.WALProcedureTree: Missing stack id 178, max stack id is 181, root procedure 
is Procedure(pid=289380, ppid=-1, 
class=org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure)
2019-04-30 23:41:55,389 ERROR [master/master:17000:becomeActiveMaster] 
wal.WALProcedureTree: Missing stack id 359, max stack id is 360, root procedure 
is Procedure(pid=285640, ppid=-1, 
class=org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure)
{noformat}

After which the procedure(s) is/are lost and cluster is stuck permanently.
There were no errors writing these files in the log, and no issues reading them 
from HDFS, so it's purely a data loss issue in the structure. 

I was thinking about debugging it, but on 2nd though what we are trying to 
store PB state by key.
Coincidentally, we have an "HBase" facility that we already deploy, that does 
just that... and it even has a WAL implementation. I don't know why we cannot 
use it for procedure state and have to invent another complex implementation of 
a KV store inside a KV store.
In all/most cases, we don't even support rollback and use the latest state, but 
if we need multiple versions, this HBase product even supports that! 
I think we should add a hbase:proc table that would be maintained similar to 
meta. The latter part esp. given the existing code for meta should be much more 
simple than a separate store impl.
This should be pluggable and optional via ProcStore interface (made more 
abstract as relevant - update state, scan state, get)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69262: HIVE-20853

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69262/#review210357
---




llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
Lines 191 (patched)
<https://reviews.apache.org/r/69262/#comment295021>

the credentials field is optional in pb but this is using it without a check



llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
Lines 318 (patched)
<https://reviews.apache.org/r/69262/#comment295020>

why would it be null


- Sergey Shelukhin


On Nov. 6, 2018, 10:26 p.m., Jaume Marhuenda wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69262/
> ---
> 
> (Updated Nov. 6, 2018, 10:26 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Expose ShuffleHandler.registerDag in the llap daemon API
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/tez/LlapProtocolClientProxy.java
>  211696a0b5 
>   
> llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
>  8fecc1e920 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/impl/LlapProtocolClientImpl.java
>  bdffbbfc22 
>   llap-common/src/protobuf/LlapDaemonProtocol.proto d70dd41a83 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/ContainerRunner.java 
> 035960e347 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonUtils.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  ef5922ef41 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 52990c5f05 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  d856b2580a 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  ab84dcc5b3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
>  18a37a2adc 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/LlapDaemonTestUtils.java
>  PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestContainerRunnerImpl.java
>  PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  d3aa53942b 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  5d4ce223d9 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  7e8299d156 
>   llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTezUtils.java 
> e4af660fff 
> 
> 
> Diff: https://reviews.apache.org/r/69262/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jaume Marhuenda
> 
>

[jira] [Created] (HIVE-20840) sysdb test is not updated and fails on update

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20840:
---

 Summary: sysdb test is not updated and fails on update
 Key: HIVE-20840
 URL: https://issues.apache.org/jira/browse/HIVE-20840
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
 Attachments: HIVE-20840.test.patch





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69148/
---

(Updated Oct. 26, 2018, 11:23 p.m.)


Review request for hive, Jason Dere and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
e226a1f82d44550f389308f91d578e7aa4ea170a 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 c3e1e8e88c69d7713e16c7061ce8cf73a0d5e833 
  metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 
a69046f961cdf0fff7989492c489bb62f2a66d72 
  metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql 
4c770206fe3dcceb8570be1c1ef078b376f5cafd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
807f159daa98d40e667914adc6c53fb8ecabf998 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
012a670064c155a31dab5337dc97ac5bb7b39aec 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
e57db935d9420508ed6091e12ca6b6cd3382db5d 
  ql/src/test/queries/clientpositive/resourceplan.q 
fae9701ebaeaa521904a383f5fb741c13be08d8e 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 
c11daf728cdd5bd6fe36618aff113b3d60579129 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 90b5764a2e24d5ddf71d227f1364a0d09d63c185 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 d1c0c4d1f60016f28cea69348b1b30ecb61bf083 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
4b7b61520a2d55635f474317053a17410f3a4bb7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 8cd46e3f44e7c4e47fbf7f2ce2b6350a5814106f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 9c158040497cd3d2762620ce35e2b46bb6d5fffe 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 c3914b668fac18ead6196a4fc449e909f5af01b1 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 47ac68c667bea8f09f5301a6364c854bc18b3c0d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
 ac51f2d77145b37da468ce8df2ac5c42f4d6c538 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
fef6a42038bb2aa0cba6dfda8d710fd37cb720e7 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 c889bbdf96b887b29be858e41ee854f0731cd5cd 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 aca5227a5bb6192da6c5f070c04d2941d636bad2 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 91ba134325094e413887a89e1d605efa99218288 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 f0d861b3a9bc982c1e24fa49415dcfc6c105cd68 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 3af2ebb253f82bb85976d229d4ac2225deffdbde 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 ee0f691b524a6e822ac14e09d24d3a49ae8565b1 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 33aa08015a9e17585c42d64d44b364be96e69eaf 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 bbb4a39ec4f6f616c9a1a9042a35cafd45cf9796 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 ea088d77fdaec85834b8fd3f01eacdfac58dd245 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 2a2d70ae802eb6f5b9ab7f4f9519a0af30d2c5b4 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 966979891b71f1cbfe50f56c40c35af8b304c47f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 593d562c3498660861201f58d83c27d59d184046 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 4293579ad8b55d59f2230040f23e9a693d838ca7 


Diff: https://reviews.apache.org/r/69148/diff/3/

Changes: https://reviews.apache.org/r/69148/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

[Sergey Shelukhin]

> On Oct. 26, 2018, 7:16 p.m., Prasanth_J wrote:
> > ql/src/test/results/clientpositive/llap/resourceplan.q.out
> > Line 3918 (original), 3970 (patched)
> > <https://reviews.apache.org/r/69148/diff/2/?file=2102661#file2102661line3976>
> >
> > hmm.. expected?

This one is a bug...


> On Oct. 26, 2018, 7:16 p.m., Prasanth_J wrote:
> > ql/src/test/results/clientpositive/llap/resourceplan.q.out
> > Line 4051 (original), 4101 (patched)
> > <https://reviews.apache.org/r/69148/diff/2/?file=2102661#file2102661line4109>
> >
> > Similarly are these support to show up in clientpositive tests?

Yes, there's a setting that skips errors that Harish found, that allows one to 
avoid creating 1000 negative tests :) 
However with large out file changes like this it can be confusing to tell apart 
bugs from genuine errors.


> On Oct. 26, 2018, 7:16 p.m., Prasanth_J wrote:
> > ql/src/test/results/clientpositive/llap/resourceplan.q.out
> > Line 4713 (original), 4763 (patched)
> > <https://reviews.apache.org/r/69148/diff/2/?file=2102661#file2102661line4771>
> >
> > can you add a non-default ns test as well?

This would require a config change at minicluster init time. I added a unit 
test instead for that scenario.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69148/#review210105
---


On Oct. 25, 2018, 9:43 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69148/
> ---
> 
> (Updated Oct. 25, 2018, 9:43 p.m.)
> 
> 
> Review request for hive, Jason Dere and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> e226a1f82d44550f389308f91d578e7aa4ea170a 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  c3e1e8e88c69d7713e16c7061ce8cf73a0d5e833 
>   metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 
> a69046f961cdf0fff7989492c489bb62f2a66d72 
>   metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql 
> 4c770206fe3dcceb8570be1c1ef078b376f5cafd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 807f159daa98d40e667914adc6c53fb8ecabf998 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 4de038913a5c9a2c199f71702b8f70ca84d0856b 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
> e57db935d9420508ed6091e12ca6b6cd3382db5d 
>   ql/src/test/queries/clientpositive/resourceplan.q 
> fae9701ebaeaa521904a383f5fb741c13be08d8e 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 
> c11daf728cdd5bd6fe36618aff113b3d60579129 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  d1c0c4d1f60016f28cea69348b1b30ecb61bf083 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 4b7b61520a2d55635f474317053a17410f3a4bb7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  8cd46e3f44e7c4e47fbf7f2ce2b6350a5814106f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  c3914b668fac18ead6196a4fc449e909f5af01b1 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  47ac68c667bea8f09f5301a6364c854bc18b3c0d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  ac51f2d77145b37da468ce8df2ac5c42f4d6c538 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> fef6a42038bb2aa0cba6dfda8d710fd37cb720e7 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  c889bbdf96b887b29be858e41ee854f0731cd5cd 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  aca5227a5bb6192da6c5f070c04d2941d636bad2 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69148/
---

(Updated Oct. 25, 2018, 9:43 p.m.)


Review request for hive, Jason Dere and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
e226a1f82d44550f389308f91d578e7aa4ea170a 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 c3e1e8e88c69d7713e16c7061ce8cf73a0d5e833 
  metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 
a69046f961cdf0fff7989492c489bb62f2a66d72 
  metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql 
4c770206fe3dcceb8570be1c1ef078b376f5cafd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
807f159daa98d40e667914adc6c53fb8ecabf998 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
4de038913a5c9a2c199f71702b8f70ca84d0856b 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
e57db935d9420508ed6091e12ca6b6cd3382db5d 
  ql/src/test/queries/clientpositive/resourceplan.q 
fae9701ebaeaa521904a383f5fb741c13be08d8e 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 
c11daf728cdd5bd6fe36618aff113b3d60579129 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 d1c0c4d1f60016f28cea69348b1b30ecb61bf083 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
4b7b61520a2d55635f474317053a17410f3a4bb7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 8cd46e3f44e7c4e47fbf7f2ce2b6350a5814106f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 9c158040497cd3d2762620ce35e2b46bb6d5fffe 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 c3914b668fac18ead6196a4fc449e909f5af01b1 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 47ac68c667bea8f09f5301a6364c854bc18b3c0d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
 ac51f2d77145b37da468ce8df2ac5c42f4d6c538 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
fef6a42038bb2aa0cba6dfda8d710fd37cb720e7 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 c889bbdf96b887b29be858e41ee854f0731cd5cd 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 aca5227a5bb6192da6c5f070c04d2941d636bad2 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 91ba134325094e413887a89e1d605efa99218288 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 f0d861b3a9bc982c1e24fa49415dcfc6c105cd68 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 3af2ebb253f82bb85976d229d4ac2225deffdbde 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 ee0f691b524a6e822ac14e09d24d3a49ae8565b1 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 33aa08015a9e17585c42d64d44b364be96e69eaf 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 bbb4a39ec4f6f616c9a1a9042a35cafd45cf9796 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 ea088d77fdaec85834b8fd3f01eacdfac58dd245 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 2a2d70ae802eb6f5b9ab7f4f9519a0af30d2c5b4 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 966979891b71f1cbfe50f56c40c35af8b304c47f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 593d562c3498660861201f58d83c27d59d184046 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 4293579ad8b55d59f2230040f23e9a693d838ca7 


Diff: https://reviews.apache.org/r/69148/diff/2/

Changes: https://reviews.apache.org/r/69148/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20810) option for q files to create sysdb without hardcoding the path

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20810:
---

 Summary: option for q files to create sysdb without hardcoding the 
path
 Key: HIVE-20810
 URL: https://issues.apache.org/jira/browse/HIVE-20810
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Just noticed the master sysdb and resourceplan tests (maybe others too) still 
run 3.1 versions of the sysdb script, because the only way to run it right now 
it so hardcode the path to some sql file. I'm going to fix that for now in some 
other JIRA.
There should be a better way to init sysdb for tests, like we do for other 
datasets like src.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 69148: HIVE-20793 add RP namespacing to workload management

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69148/
---

Review request for hive, Jason Dere and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
ed6d3d80e3f6a9ba66f46223d97666b48c388028 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 c3e1e8e88c69d7713e16c7061ce8cf73a0d5e833 
  metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 
a69046f961cdf0fff7989492c489bb62f2a66d72 
  metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql 
4c770206fe3dcceb8570be1c1ef078b376f5cafd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
807f159daa98d40e667914adc6c53fb8ecabf998 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
4de038913a5c9a2c199f71702b8f70ca84d0856b 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
e57db935d9420508ed6091e12ca6b6cd3382db5d 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 d1c0c4d1f60016f28cea69348b1b30ecb61bf083 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
4b7b61520a2d55635f474317053a17410f3a4bb7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 8cd46e3f44e7c4e47fbf7f2ce2b6350a5814106f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 9c158040497cd3d2762620ce35e2b46bb6d5fffe 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 c3914b668fac18ead6196a4fc449e909f5af01b1 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 47ac68c667bea8f09f5301a6364c854bc18b3c0d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
 ac51f2d77145b37da468ce8df2ac5c42f4d6c538 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
fef6a42038bb2aa0cba6dfda8d710fd37cb720e7 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 c889bbdf96b887b29be858e41ee854f0731cd5cd 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 aca5227a5bb6192da6c5f070c04d2941d636bad2 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 91ba134325094e413887a89e1d605efa99218288 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 f0d861b3a9bc982c1e24fa49415dcfc6c105cd68 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 3af2ebb253f82bb85976d229d4ac2225deffdbde 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 ee0f691b524a6e822ac14e09d24d3a49ae8565b1 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 33aa08015a9e17585c42d64d44b364be96e69eaf 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 bbb4a39ec4f6f616c9a1a9042a35cafd45cf9796 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 ea088d77fdaec85834b8fd3f01eacdfac58dd245 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 2a2d70ae802eb6f5b9ab7f4f9519a0af30d2c5b4 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 966979891b71f1cbfe50f56c40c35af8b304c47f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 593d562c3498660861201f58d83c27d59d184046 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 4293579ad8b55d59f2230040f23e9a693d838ca7 


Diff: https://reviews.apache.org/r/69148/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20793) add RP namespacing to workload management

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20793:
---

 Summary: add RP namespacing to workload management
 Key: HIVE-20793
 URL: https://issues.apache.org/jira/browse/HIVE-20793
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


The idea is to be able to use the same warehouse for multiple clusters in the 
cloud use cases. This scenario is not currently supported by WM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20787) MapJoinBytesTableContainer dummyRow case doesn't handle reuse

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20787:
---

 Summary: MapJoinBytesTableContainer dummyRow case doesn't handle 
reuse
 Key: HIVE-20787
 URL: https://issues.apache.org/jira/browse/HIVE-20787
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


Discovered while investigating some (probably) unrelated issue.
MapJoinBytesTableContainer was not intended to be reused, but it looks like 
some code might reuse it. If that happens, the dummyRow case will not work 
correctly (dummyRow is cleared on first(), so another call to first() will 
behave differently).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69073: HIVE-20772 record per-task CPU counters in LLAP

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69073/
---

(Updated Oct. 19, 2018, 9:16 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

.


Diffs (updated)
-

  llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java 50c0e22837 
  llap-common/src/java/org/apache/hadoop/hive/llap/counters/LlapIOCounters.java 
059d5b9ae3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheCounters.java
 91df036bfd 
  
llap-server/src/java/org/apache/hadoop/hive/llap/counters/QueryFragmentCounters.java
 be4dfad95c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/StatsRecordingThreadPool.java
 27462e1bcb 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 27a5b0f3e4 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java
 d5c2d48db1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 40248a37f8 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 4f5b0a9e65 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 658bc7d621 


Diff: https://reviews.apache.org/r/69073/diff/2/

Changes: https://reviews.apache.org/r/69073/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20774) use less calls for GetTables operation

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20774:
---

 Summary: use less calls for GetTables operation
 Key: HIVE-20774
 URL: https://issues.apache.org/jira/browse/HIVE-20774
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 69073: HIVE-20772 record per-task CPU counters in LLAP

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69073/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

.


Diffs
-

  llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java 50c0e22837 
  llap-common/src/java/org/apache/hadoop/hive/llap/counters/LlapIOCounters.java 
059d5b9ae3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheCounters.java
 91df036bfd 
  
llap-server/src/java/org/apache/hadoop/hive/llap/counters/QueryFragmentCounters.java
 be4dfad95c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/StatsRecordingThreadPool.java
 27462e1bcb 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 27a5b0f3e4 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java
 d5c2d48db1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 40248a37f8 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 4f5b0a9e65 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 658bc7d621 


Diff: https://reviews.apache.org/r/69073/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20772) record per-task CPU counters in LLAP

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20772:
---

 Summary: record per-task CPU counters in LLAP
 Key: HIVE-20772
 URL: https://issues.apache.org/jira/browse/HIVE-20772
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20766) support get_aggr_stats for txn stats in CachedStore

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20766:
---

 Summary: support get_aggr_stats for txn stats in CachedStore
 Key: HIVE-20766
 URL: https://issues.apache.org/jira/browse/HIVE-20766
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20765) validate txn stats for partitions with one call

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20765:
---

 Summary: validate txn stats for partitions with one call
 Key: HIVE-20765
 URL: https://issues.apache.org/jira/browse/HIVE-20765
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20657) pre-allocate LLAP cache at init time

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20657:
---

 Summary: pre-allocate LLAP cache at init time
 Key: HIVE-20657
 URL: https://issues.apache.org/jira/browse/HIVE-20657
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68802: HIVE-20620 manifest collisions when inserting into bucketed sorted MM tables with dynamic partitioning

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68802/
---

(Updated Sept. 21, 2018, 10:05 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 3672c7afef 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9593975c6c 
  ql/src/test/queries/clientpositive/mm_dp.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/mm_dp.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/68802/diff/2/

Changes: https://reviews.apache.org/r/68802/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 68802: HIVE-20620 manifest collisions when inserting into bucketed sorted MM tables with dynamic partitioning

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68802/
---

Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

see jira


Diffs
-

  itests/src/test/resources/testconfiguration.properties 3672c7afef 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9593975c6c 
  ql/src/test/queries/clientpositive/mm_dp.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/mm_dp.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/68802/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20620) manifest collisions when inserting into bucketed sorted MM tables with dynamic partitioning

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20620:
---

 Summary: manifest collisions when inserting into bucketed sorted 
MM tables with dynamic partitioning
 Key: HIVE-20620
 URL: https://issues.apache.org/jira/browse/HIVE-20620
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20605) merge master-tez092 branch into master

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20605:
---

 Summary: merge master-tez092 branch into master
 Key: HIVE-20605
 URL: https://issues.apache.org/jira/browse/HIVE-20605
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


I got tired of waiting for Tez 0.92 release (it's been pending for half a year) 
so I created a branch to prevent various patches from conflicting with each 
other.
This jira is to merge them into master after Tez 0.92 is finally released.
The jiras here: 
https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20fixVersion%20%3D%20master-tez092
 should then be updated with the corresponding Hive release version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68709: HIVE-20547 HS2: support Tez sessions started by someone else (part 1)

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68709/
---

(Updated Sept. 13, 2018, 8:11 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa58d7445c 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java bfa3d5d7d2 
  ql/pom.xml d73deba440 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 79cb54e552 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AbstractTriggerValidator.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KillTriggerActionHandler.java 
f357775c86 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionsRegistryClient.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSession.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 89954cba67 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
2633390861 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
d3748edb86 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
08e65a4a6d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java f2ed07add5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TriggerValidatorRunnable.java 
670184b0ac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java fa2b02e591 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
5326e3590f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerFederation.java 
4b5022a101 
  ql/src/java/org/apache/hadoop/hive/ql/session/KillQuery.java 01dc7e2cd7 
  ql/src/java/org/apache/hadoop/hive/ql/session/NullKillQuery.java eac2936719 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 71e130b608 
  ql/src/java/org/apache/hadoop/hive/ql/wm/SessionTriggerProvider.java 
16106f481b 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
f5ab981f26 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d5b683f788 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java b67aec371d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
4659ecb97b 
  service/src/java/org/apache/hive/service/server/KillQueryImpl.java 490a04da67 


Diff: https://reviews.apache.org/r/68709/diff/2/

Changes: https://reviews.apache.org/r/68709/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 68709: HIVE-20547 HS2: support Tez sessions started by someone else (part 1)

[Sergey Shelukhin]

> On Sept. 13, 2018, 8:03 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java
> > Lines 72 (patched)
> > <https://reviews.apache.org/r/68709/diff/1/?file=2088718#file2088718line72>
> >
> > These resources may not be required. External sessions are assumed to 
> > have all the resources that it requires.

It's used in Tez object construction... I think Eric told me it was important 
for the reconnect patch to still add the resources. So for now I'll keep this 
in.


> On Sept. 13, 2018, 8:03 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionsRegistryClient.java
> > Lines 37 (patched)
> > <https://reviews.apache.org/r/68709/diff/1/?file=2088719#file2088719line37>
> >
> > Yes. Tez needs severals fixes in addition to registry client
> > 1) One that cuts of yarn/RM interaction
> > 2) Registry configs via TezConfiguration
> > 3) Handling DAG recovery for external sessions

can you file some Tez jiras for that?


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68709/#review208584
---


On Sept. 13, 2018, 2:19 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68709/
> ---
> 
> (Updated Sept. 13, 2018, 2:19 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa58d7445c 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> bfa3d5d7d2 
>   ql/pom.xml a55cbe380d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AbstractTriggerValidator.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KillTriggerActionHandler.java 
> f357775c86 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionsRegistryClient.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSession.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 
> 89954cba67 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 2633390861 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
> d3748edb86 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
> 08e65a4a6d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java f2ed07add5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TriggerValidatorRunnable.java 
> 670184b0ac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java fa2b02e591 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> 5326e3590f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerFederation.java 
> 4b5022a101 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 71e130b608 
>   ql/src/java/org/apache/hadoop/hive/ql/wm/SessionTriggerProvider.java 
> 16106f481b 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
> f5ab981f26 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d5b683f788 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java b67aec371d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 4659ecb97b 
> 
> 
> Diff: https://reviews.apache.org/r/68709/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-20547) HS2: support Tez sessions started by someone else (part 1)

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20547:
---

 Summary: HS2: support Tez sessions started by someone else (part 1)
 Key: HIVE-20547
 URL: https://issues.apache.org/jira/browse/HIVE-20547
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Attachments: HIVE-20547.patch

The registry/configs/some code is based on a private patch by [~prasanth_j].
The patch refactors tez pool session to use composition instead of 
implementation inheritance from TezSessionState, to allow for two 
implementations of TezSessionState.


For now it's blocked on getClient API in Tez that will be available after 0.9.3 
release; however I commented out that path to check that refactoring passes 
tests.
When 0.9.3 becomes available, we can uncomment and commit.

In part 2, we may add some tests, and also consider other changes that are 
required for external sessions (e.g. KillQuery, where we cannot assume YARN is 
present).

We may also consider a WM change that allows for proportional session 
distribution when the number of external sessions and the number of 
admin-specified sessions doesn't match, or at least some validation to see that 
the external sessions are available when applying a RP.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68709: HIVE-20547 HS2: support Tez sessions started by someone else (part 1)

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68709/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa58d7445c 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java bfa3d5d7d2 
  ql/pom.xml a55cbe380d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AbstractTriggerValidator.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KillTriggerActionHandler.java 
f357775c86 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionsRegistryClient.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSession.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 89954cba67 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
2633390861 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
d3748edb86 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
08e65a4a6d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java f2ed07add5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TriggerValidatorRunnable.java 
670184b0ac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java fa2b02e591 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
5326e3590f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerFederation.java 
4b5022a101 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 71e130b608 
  ql/src/java/org/apache/hadoop/hive/ql/wm/SessionTriggerProvider.java 
16106f481b 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
f5ab981f26 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d5b683f788 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java b67aec371d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
4659ecb97b 


Diff: https://reviews.apache.org/r/68709/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20471) issues getting the default database path

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20471:
---

 Summary: issues getting the default database path
 Key: HIVE-20471
 URL: https://issues.apache.org/jira/browse/HIVE-20471
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


{noformat}
Caused by: java.lang.IllegalArgumentException: Can not create a Path from an 
empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:168)
at org.apache.hadoop.fs.Path.(Path.java:237)
at 
org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:143)
at 
org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:147)
at 
org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:160)
at 
org.apache.hadoop.hive.metastore.Warehouse.getDatabasePath(Warehouse.java:235)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2335)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2074)

{noformat}

given that the path passed to getDnsPath is already a path that is probably 
valid, this method needs to be a little bit more reliable. Not sure which 
argument turns out to be empty, but I wonder why this complexity is needed for 
the default database.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20453) backport HIVE-16233 (LLAP cache defragmentation) to branch-2

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20453:
---

 Summary: backport HIVE-16233 (LLAP cache defragmentation) to 
branch-2
 Key: HIVE-20453
 URL: https://issues.apache.org/jira/browse/HIVE-20453
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20443) txn stats cleanup in compaction txn handler is unneeded

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20443:
---

 Summary: txn stats cleanup in compaction txn handler is unneeded
 Key: HIVE-20443
 URL: https://issues.apache.org/jira/browse/HIVE-20443
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: External vs. Managed Tables - Current state

[Sergey Shelukhin]

I think it’s very simple conceptually (well the way I’ve been thinking
about it), and any limitations can be derived from this using logic. If
someone has specific questions we can clarify; if some limitations are
missing (e.g. when Hive was using stats for hbase tables) it is a bug.

For managed tables, Hive owns and manages the data:

* Data, properties, etc. can only be changed via Hive commands or APIs;
Hive assumes it’s true and any other modification on the underlying
storage is undefined behavior (not just results - basically all bets are
off ;)).
* Data is attached to the Hive entity (partition, table). Whatever happens
to the entity happens to the data (e.g. the behavior mentioned in the
wiki, where  dropping the table/partition deletes the data).


For external tables (also SH tables before Hive made those explicitly
external), Hive doesn’t manage or own the data:

* Hive assumes the data can be modified externally at any time. Any
feature that requires assumptions about data being unchanged (queries from
stats, ACID, some MV, query cache) won’t work. If it works, it’s a bug.
* Data is detached from Hive entities, e.g. dropping the table by default
doesn’t delete the data - we assume someone else owns it and we are just
pointing at it.
* Additionally, since Hive cannot track modifications, external
products/users/etc. are fully responsible for any concurrency control. In
particular, modifying data under the running query may produce
inconsistent results (no atomicity between tasks), undefined results due
to retries, etc.





On 18/8/21, 10:31, "Mithun RK"  wrote:

>+1. I'm out of the loop as well. I'd be keen on grokking what's not
>available with EXTERNAL tables, given that these are popular on our
>clusters as well. It would be good to have that documented in one place.
>
>On Tue, Aug 21, 2018 at 8:19 AM Lars Francke 
>wrote:
>
>> As this came up with yet another customer and I've heard more things
>>that I
>> believe to be false:
>>
>> Please, anyone have any kind of feedback for me here? Is my list
>>correct,
>> can you add stuff to it, have more details to certain points etc.?
>>
>> Thank you!
>>
>> On Wed, Aug 1, 2018 at 12:17 PM, Lars Francke 
>> wrote:
>>
>> > I have opened https://issues.apache.org/jira/browse/HIVE-20287 and
>>would
>> > love to get feedback
>> >
>> > On Wed, Aug 1, 2018 at 11:14 AM, Lars Francke 
>> > wrote:
>> >
>> >> Thanks Lefty! I haven't created a JIRA but it's on my list.
>> >>
>> >> Still interested in any input anyone might have?
>> >>
>> >> Some of you guys _must_ have some knowledge on this stuff.
>> >>
>> >> You're correct that the DROP stuff is documented. What I meant is
>>that I
>> >> couldn't find the relevant parts in the code.
>> >>
>> >> On Sat, Jul 21, 2018 at 6:15 AM, Lefty Leverenz <
>> leftylever...@gmail.com>
>> >> wrote:
>> >>
>> >>> Agreed, the Hive wiki is woefully incomplete.  But it does mention
>>DROP
>> >>> for
>> >>> external tables in two sections of the DDL doc:
>> >>>
>> >>>- External Tables
>> >>>> >>> anual+DDL#LanguageManualDDL-ExternalTables>:
>> >>> "When dropping an EXTERNAL table, data in the table is NOT
>>deleted
>> >>> from
>> >>>the file system."
>> >>>- Drop Table
>> >>>> >>> anual+DDL#LanguageManualDDL-DropTable>:
>> >>> "When dropping an EXTERNAL table, data in the table will NOT be
>> >>> deleted
>> >>>from the file system."
>> >>>
>> >>> A wiki page comparing managed and external tables would be very
>> helpful,
>> >>> and the list is a good start.  I suggest you open a JIRA issue.
>> >>>
>> >>> Who can help Lars with this task?
>> >>>
>> >>> -- Lefty
>> >>>
>> >>>
>> >>> On Thu, Jul 19, 2018 at 3:04 PM Lars Francke
>>
>> >>> wrote:
>> >>>
>> >>> > Hi,
>> >>> >
>> >>> > I've been wondering if anyone can tell me what the differences and
>> >>> > limitations of managed vs. external tables are these days.
>> >>> >
>> >>> > I've seen the docs from Hortonworks[1] that list these features as
>> not
>> >>> > supported by External tables:
>> >>> > * Query cache
>> >>> > * Materialized views, except in a limited way
>> >>> > * Default statistics gathering
>> >>> > * Compute queries using statistics
>> >>> > * Automatic runtime filtering
>> >>> > * File merging after insert
>> >>> >
>> >>> > But there are no details. For some of them I'm not even sure what
>> they
>> >>> are
>> >>> > supposed to mean because I couldn't find any documentation in our
>> Wiki
>> >>> :(
>> >>> >
>> >>> > So I'd love to create a Wiki page that describes the differences
>> >>> between
>> >>> > managed & external tables and list relevant JIRAs but I need your
>> help
>> >>> for
>> >>> > that.
>> >>> >
>> >>> > I scanned the code for all references to TableType and this is the
>> >>> list I
>> >>> > found but I'm almost certainly missing something:
>> >>> >
>> >>> > * ARCHIVE/UNARCHIVE - DDLTask - Only 

[jira] [Created] (HIVE-20434) analyze on an empty partition doesn't produce correct stats

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20434:
---

 Summary: analyze on an empty partition doesn't produce correct 
stats
 Key: HIVE-20434
 URL: https://issues.apache.org/jira/browse/HIVE-20434
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Ashutosh Chauhan


{noformat}
set hive.stats.dbclass=fs;
set hive.stats.fetch.column.stats=true;
set datanucleus.cache.collections=false;
set hive.merge.mapfiles=false;
set hive.merge.mapredfiles=false;
set hive.mapred.mode=nonstrict;

set hive.stats.autogather=true;
set hive.stats.column.autogather=true;
set hive.compute.query.using.stats=true;
set hive.explain.user=false;

set hive.fetch.task.conversion=none;
set hive.query.results.cache.enabled=false;

create table stats_part(key int,value string) partitioned by (p int) 
tblproperties ("transactional"="false");
insert into table stats_part partition(p=101) values (1, "foo");
explain select count(key) from stats_part; -- <== stats are correct

ALTER TABLE stats_part CHANGE COLUMN key key2 int;
explain select count(key2) from stats_part; -- <== stats are now incorrect

analyze table stats_part partition(p) compute statistics for columns;
explain select count(key2) from stats_part; -- <== stats are now correct again

alter table stats_part add partition(p=105);
explain select count(key2) from stats_part; -- <== stats are now incorrect

analyze table stats_part partition(p) compute statistics for columns;
explain select count(key2) from stats_part; -- <== stats are still incorrect!


drop table stats_part;
{noformat}

Seems like it's impossible to have correct stats on a table with an empty 
partition.








--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68434: HIVE-20431 txn stats write ID check triggers on set location

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68434/
---

Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
2b9f763692 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java 680e0297ed 
  ql/src/test/queries/clientpositive/acid_stats4.q 20d1159541 
  ql/src/test/results/clientpositive/acid_stats4.q.out afd5adb4ff 


Diff: https://reviews.apache.org/r/68434/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20431) txn stats write ID check triggers on set location

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20431:
---

 Summary: txn stats write ID check triggers on set location
 Key: HIVE-20431
 URL: https://issues.apache.org/jira/browse/HIVE-20431
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20418) LLAP IO may not handle ORC files with row index disabled correctly for queries with no columns selected

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20418:
---

 Summary: LLAP IO may not handle ORC files with row index disabled 
correctly for queries with no columns selected
 Key: HIVE-20418
 URL: https://issues.apache.org/jira/browse/HIVE-20418
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
 Attachments: HIVE-20418.patch





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20417) Hive converts strings to doubles in a strange manner

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20417:
---

 Summary: Hive converts strings to doubles in a strange manner
 Key: HIVE-20417
 URL: https://issues.apache.org/jira/browse/HIVE-20417
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Matt McCline


This is not on master, but on a close enough version, so it needs to be dbl 
checked... running with vectorization on:
{noformat}
POSTHOOK: query: create table cross_numbers(i string)
POSTHOOK: query: insert into table cross_numbers select key from src limit 20
POSTHOOK: query: select c1.i, c2.i, c1.i + c2.i, c1.i + '' + c2.i from 
cross_numbers c1 cross 
401 265 666.0   NULL
401 255 656.0   NULL
401 165 566.0   NULL
{noformat}

Why are strings added as numbers, and string + '' + string converted to null is 
not clear..

[~ashutoshc] is plus supposed to work for string?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20414) expand HIVE-20400 to all locations in Hive (create/alter table, partition, database, etc.)

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20414:
---

 Summary: expand HIVE-20400 to all locations in Hive (create/alter 
table, partition, database, etc.)
 Key: HIVE-20414
 URL: https://issues.apache.org/jira/browse/HIVE-20414
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20411) Hive.loadPartition doesn't support catalogs

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20411:
---

 Summary: Hive.loadPartition doesn't support catalogs
 Key: HIVE-20411
 URL: https://issues.apache.org/jira/browse/HIVE-20411
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20408) add tests for HiveStrictManagedMigration

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20408:
---

 Summary: add tests for HiveStrictManagedMigration
 Key: HIVE-20408
 URL: https://issues.apache.org/jira/browse/HIVE-20408
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Jason Dere


There are currently no tests for this class... there should be some test 
coverage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20407) add txn table alter support to HiveStrictManagedMigration

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20407:
---

 Summary: add txn table alter support to HiveStrictManagedMigration
 Key: HIVE-20407
 URL: https://issues.apache.org/jira/browse/HIVE-20407
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


On top of HIVE-20397



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20399) CTAS w/a custom table location that is not fully qualified fails for MM tables

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20399:
---

 Summary: CTAS w/a custom table location that is not fully 
qualified fails for MM tables
 Key: HIVE-20399
 URL: https://issues.apache.org/jira/browse/HIVE-20399
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20400) create table should always use a fully qualified path to avoid potential FS ambiguity

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20400:
---

 Summary: create table should always use a fully qualified path to 
avoid potential FS ambiguity
 Key: HIVE-20400
 URL: https://issues.apache.org/jira/browse/HIVE-20400
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

ptests fail due to a port conflict

[Sergey Shelukhin]

Could this be related to some tests running in parallel?

https://builds.apache.org/job/PreCommit-HIVE-Build/13209/testReport/org.apa
che.hive.jdbc/TestActivePassiveHA/testManualFailoverUnauthorized/


2018-08-14T04:11:45,837 ERROR [Thread-10] server.HiveServer2: Error
starting Web UI: 
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_102]
...
at org.apache.hive.http.HttpServer.start(HttpServer.java:254)
~[hive-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:732)
[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hive.jdbc.miniHS2.MiniHS2.start(MiniHS2.java:371)
[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]

[jira] [Created] (HIVE-20380) explore storing multiple CBs in a single cache buffer in LLAP cache

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20380:
---

 Summary: explore storing multiple CBs in a single cache buffer in 
LLAP cache
 Key: HIVE-20380
 URL: https://issues.apache.org/jira/browse/HIVE-20380
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Lately ORC CBs are becoming ridiculously small. First there's the 4Kb minimum 
(instead of 256Kb), then after we moved metadata cache off-heap, the index 
streams that are all tiny take up a lot of CBs and waste space. 
Wasted space can require larger cache and lead to cache OOMs on some workloads.
Reducing min.alloc solves this problem, but then there's a lot of heap (and 
probably compute) overhead to track all these buffers. Arguably even the 4Kb 
min.alloc is too small.

We should store contiguous CBs in the same buffer; to start, we can do it for 
ROW_INDEX streams. That probably means reading all ROW_INDEX streams instead of 
doing projection when we see that they are too small.
We need to investigate what the pattern is for ORC data blocks. One option is 
to increase min.alloc and then consolidate multiple 4-8Kb CBs, but only for the 
same stream. However larger min.alloc will result in wastage for really small 
streams, so we can also consolidate multiple streams (potentially across 
columns) if needed. This will result in some priority anomalies but they 
probably ok.

Another consideration is making tracking less object oriented, in particular 
passing around integer indexes instead of objects and storing state in giant 
arrays somewhere (potentially with some optimizations for less common things), 
instead of every buffers getting its own object. 

cc [~gopalv] [~prasanth_j]





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20378) don't update stats during alter for txn table conversion

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20378:
---

 Summary: don't update stats during alter for txn table conversion
 Key: HIVE-20378
 URL: https://issues.apache.org/jira/browse/HIVE-20378
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20311) add txn stats checks to some more paths

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20311:
---

 Summary: add txn stats checks to some more paths
 Key: HIVE-20311
 URL: https://issues.apache.org/jira/browse/HIVE-20311
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


These were set to false in the original patch for no reason as far as I see.
I later added notes but not TODOs to switch them over, so they remained as 
non-txn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20299) potential race in LLAP signer unit test

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20299:
---

 Summary: potential race in LLAP signer unit test
 Key: HIVE-20299
 URL: https://issues.apache.org/jira/browse/HIVE-20299
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New committer: Slim Bouguerra

[Sergey Shelukhin]

Congrats!

On 18/7/30, 12:53, "Gunther Hagleitner" 
wrote:

>Congratulations!
>
>Thanks,
>Gunther.
>
>From: Xuefu Zhang 
>Sent: Monday, July 30, 2018 12:11 PM
>To: dev@hive.apache.org
>Subject: Re: [ANNOUNCE] New committer: Slim Bouguerra
>
>congratulations!!!
>
>On Mon, Jul 30, 2018 at 12:10 PM, Jesus Camacho Rodriguez <
>jcamachorodrig...@hortonworks.com> wrote:
>
>> Congrats Slim!
>>
>> On 7/30/18, 10:53 AM, "Andrew Sherman" 
>> wrote:
>>
>> Congratulations Slim!
>>
>> On Mon, Jul 30, 2018 at 12:46 AM Peter Vary
>>> >
>> wrote:
>>
>> > Congratulations Slim!
>> >
>> > > On Jul 30, 2018, at 02:00, Ashutosh Chauhan
>>
>> > wrote:
>> > >
>> > > Apache Hive's Project Management Committee (PMC) has invited
>>Slim
>> > Bouguerra
>> > > to become a committer, and we are pleased to announce that he
>>has
>> > accepted.
>> > >
>> > > Slim, welcome, thank you for your contributions, and we look
>> forward your
>> > > further interactions with the community!
>> > >
>> > > Ashutosh Chauhan (on behalf of the Apache Hive PMC)
>> >
>> >
>>
>>
>>

Re: Review Request 68072: HIVE-20247 cleanup issues in LLAP IO after cache OOM

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68072/
---

(Updated July 27, 2018, 1:38 a.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

.


Diffs (updated)
-

  llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
fcfc22a712 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
348f9df773 


Diff: https://reviews.apache.org/r/68072/diff/2/

Changes: https://reviews.apache.org/r/68072/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20248) clean up some TODOs after txn stats merge

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20248:
---

 Summary: clean up some TODOs after txn stats merge
 Key: HIVE-20248
 URL: https://issues.apache.org/jira/browse/HIVE-20248
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68072: HIVE-20247 cleanup issues in LLAP IO after cache OOM

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68072/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

.


Diffs
-

  llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
fcfc22a712 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
348f9df773 


Diff: https://reviews.apache.org/r/68072/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20247) cleanup issues in LLAP IO after cache OOM

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20247:
---

 Summary: cleanup issues in LLAP IO after cache OOM
 Key: HIVE-20247
 URL: https://issues.apache.org/jira/browse/HIVE-20247
 Project: Hive
  Issue Type: Bug
Reporter: Prasanth Jayachandran
Assignee: Sergey Shelukhin


LLAP IO creates unallocated buffer objects inside the read-related data 
structures, then allocates them in bulk, then decompresses into them and 
increfs them.
If allocate or decompress steps fail, it's hard for the higher-level cleanup to 
tell what the state of the buffers in the read-related structures is - they may 
be unallocated, allocated but not incref-ed, or incref-ed.
Some cleanup paths only deal with the latter case, resulting in bugs.

This only happens on one paths, others allocate and use buffers in a single 
place.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68070: HIVE-20244 forward port HIVE-19704 to master

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68070/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
fcfc22a712 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
 4297cfc61d 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/MemoryManager.java 
542041d133 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 b76b0ded98 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 5b54af5d3b 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/MetadataCache.java 
426d599b29 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java 
1e6f3ac96d 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelLrfuCachePolicy.java
 6eb2eb5089 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
 df20f20c8f 
  ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java dcb24b8018 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
f6b949e51b 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
348f9df773 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/StoppableAllocator.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
 f64efe26f5 
  storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
d1da7f5de8 


Diff: https://reviews.apache.org/r/68070/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 68034: HIVE-19568 Active/Passive HS2 HA: Disallow direct connection to passive HS2 instance

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68034/
---

(Updated July 26, 2018, 7:03 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
4055f13fd0 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestQueryDisplay.java
 8b28e2dec0 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
a78dd739b1 
  service/src/java/org/apache/hive/service/cli/CLIService.java dbfaf7154e 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
e9649824f1 
  
service/src/java/org/apache/hive/service/cli/thrift/EmbeddedThriftBinaryCLIService.java
 8b61874f93 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 432a341bef 
  service/src/test/org/apache/hive/service/auth/TestPlainSaslHelper.java 
14e28323e0 
  
service/src/test/org/apache/hive/service/cli/TestCLIServiceConnectionLimits.java
 5ecea9a08b 
  service/src/test/org/apache/hive/service/cli/TestCLIServiceRestore.java 
6b69d4d0af 
  
service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java
 7bae62d977 
  
service/src/test/org/apache/hive/service/cli/session/TestPluggableHiveSessionImpl.java
 90237c088f 
  
service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java
 af7a72e70f 
  
service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java
 d954692e98 


Diff: https://reviews.apache.org/r/68034/diff/5/

Changes: https://reviews.apache.org/r/68034/diff/4-5/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 68034: HIVE-19568 Active/Passive HS2 HA: Disallow direct connection to passive HS2 instance

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68034/
---

(Updated July 26, 2018, 7 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
4055f13fd0 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestQueryDisplay.java
 8b28e2dec0 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
a78dd739b1 
  service/src/java/org/apache/hive/service/cli/CLIService.java dbfaf7154e 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
e9649824f1 
  
service/src/java/org/apache/hive/service/cli/thrift/EmbeddedThriftBinaryCLIService.java
 8b61874f93 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 432a341bef 
  service/src/test/org/apache/hive/service/auth/TestPlainSaslHelper.java 
14e28323e0 
  
service/src/test/org/apache/hive/service/cli/TestCLIServiceConnectionLimits.java
 5ecea9a08b 
  service/src/test/org/apache/hive/service/cli/TestCLIServiceRestore.java 
6b69d4d0af 
  
service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java
 7bae62d977 
  
service/src/test/org/apache/hive/service/cli/session/TestPluggableHiveSessionImpl.java
 90237c088f 
  
service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java
 af7a72e70f 
  
service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java
 d954692e98 


Diff: https://reviews.apache.org/r/68034/diff/4/

Changes: https://reviews.apache.org/r/68034/diff/3-4/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20244) forward port HIVE-19704 to master

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20244:
---

 Summary: forward port HIVE-19704 to master
 Key: HIVE-20244
 URL: https://issues.apache.org/jira/browse/HIVE-20244
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


Apparently this logic is still there and can be engaged in some cases, like 
when one file takes the entire cache from a single large read.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68034: HIVE-19568 Active/Passive HS2 HA: Disallow direct connection to passive HS2 instance

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68034/
---

(Updated July 25, 2018, 11:55 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
4055f13fd0 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
a78dd739b1 
  service/src/java/org/apache/hive/service/cli/CLIService.java dbfaf7154e 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
e9649824f1 
  
service/src/java/org/apache/hive/service/cli/thrift/EmbeddedThriftBinaryCLIService.java
 8b61874f93 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 432a341bef 
  service/src/test/org/apache/hive/service/auth/TestPlainSaslHelper.java 
14e28323e0 
  
service/src/test/org/apache/hive/service/cli/TestCLIServiceConnectionLimits.java
 5ecea9a08b 
  service/src/test/org/apache/hive/service/cli/TestCLIServiceRestore.java 
6b69d4d0af 
  
service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java
 7bae62d977 
  
service/src/test/org/apache/hive/service/cli/session/TestPluggableHiveSessionImpl.java
 90237c088f 
  
service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java
 af7a72e70f 
  
service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java
 d954692e98 


Diff: https://reviews.apache.org/r/68034/diff/3/

Changes: https://reviews.apache.org/r/68034/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 68034: HIVE-19568 Active/Passive HS2 HA: Disallow direct connection to passive HS2 instance

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68034/
---

(Updated July 25, 2018, 1:43 a.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
4055f13fd0 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
a78dd739b1 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
e9649824f1 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 432a341bef 


Diff: https://reviews.apache.org/r/68034/diff/2/

Changes: https://reviews.apache.org/r/68034/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 68034: HIVE-19568 Active/Passive HS2 HA: Disallow direct connection to passive HS2 instance

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68034/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
e9649824f1 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 432a341bef 


Diff: https://reviews.apache.org/r/68034/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20235) AggregateStatsCache may return more partitions than requested

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20235:
---

 Summary: AggregateStatsCache may return more partitions than 
requested
 Key: HIVE-20235
 URL: https://issues.apache.org/jira/browse/HIVE-20235
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


""stats.StatsUtils: Column stats requested for : 2 partitions. Able to retrieve 
for 3 partitions"
Looks like aggregate stats cache best match may have more partitions than 
needed.
I wonder if, when there's a mismatch with the number requested, the stats 
should be recomputed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68007: HIVE-19532 merge master-txnstats branch

[Sergey Shelukhin]

/hive/metastore/TestHiveAlterHandler.java
 adc82b0b9c1e5e1a08dcb6bd30726510c2b93f6e 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
 cb32236d548ecff58fd190f1f4df8da45a42fa1a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
 fb4a761c284ea5ae87cff0ce83101ca8ac9e3e0e 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 833e2bdabf139d274a7e9af71ca26ffee23145e4 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestOldSchema.java
 717c5ee848b158f28b3233c976d2f4891e7a1650 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
 62ed380dfcc22e90ccdf4b1516cc33f128f6b3e7 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
 54bf3d7e25b6c0a6498a502e0722d0e950375739 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAppendPartitions.java
 8ce8531e9a15bcebdd017018f986d0a4c161330c 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesCreateDropAlterTruncate.java
 816a7358b2b73f3a807a27805873711ed52095c2 
  standalone-metastore/pom.xml 2da4f2b2c828a5fa9c8d0c4352122021693a549d 
  storage-api/src/java/org/apache/hadoop/hive/common/ValidTxnWriteIdList.java 
9867a81a7b88684f7152cba9d902499be0ac2504 
  storage-api/src/java/org/apache/hive/common/util/TxnIdUtils.java 
17f3777fbdd391cba52760e8f0b4fd83da2daf57 


Diff: https://reviews.apache.org/r/68007/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20219) analyze and analyze for columns don't manage stats state correctly

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20219:
---

 Summary: analyze and analyze for columns don't manage stats state 
correctly
 Key: HIVE-20219
 URL: https://issues.apache.org/jira/browse/HIVE-20219
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


Step 1. Have a table with fully correct stats.
Step 2. Run some query that sets stats to invalid.
Step 3. Run analyze without for columns...

Result is that while only basic stats are updated, the flag is set to true with 
old column stats values.
I blame json.
Not sure if this would affect master, I'll add a test; ACID stats invalidation 
definitely needs to invalidate all the stats, not just basic stats flag, when 
invalidating.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67969: HIVE-20115 Acid tables should not use footer scan for analyze

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67969/
---

(Updated July 19, 2018, 8:10 p.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
64f9c70f05 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
03cceace40 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 49709e596e 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 
28d4de7f7b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsNoJobTask.java 
3128ee8200 
  ql/src/test/queries/clientpositive/acid_no_buckets.q bcf9e0634b 
  ql/src/test/results/clientpositive/acid_table_stats.q.out 841a5a42ae 
  ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out 36a6a5d5d1 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_4.q.out
 eda3985d0a 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_5.q.out
 99832ff847 


Diff: https://reviews.apache.org/r/67969/diff/3/

Changes: https://reviews.apache.org/r/67969/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 67969: HIVE-20115 Acid tables should not use footer scan for analyze

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67969/
---

(Updated July 19, 2018, 8:04 p.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
64f9c70f05 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
03cceace40 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 49709e596e 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 
28d4de7f7b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsNoJobTask.java 
3128ee8200 
  ql/src/test/queries/clientpositive/acid_no_buckets.q bcf9e0634b 
  ql/src/test/results/clientpositive/acid_table_stats.q.out 841a5a42ae 
  ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out 36a6a5d5d1 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_4.q.out
 eda3985d0a 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_5.q.out
 99832ff847 


Diff: https://reviews.apache.org/r/67969/diff/2/

Changes: https://reviews.apache.org/r/67969/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 67969: HIVE-20115 Acid tables should not use footer scan for analyze

[Sergey Shelukhin]

> On July 19, 2018, 7:29 p.m., Eugene Koifman wrote:
> > ql/src/test/queries/clientpositive/acid_no_buckets.q
> > Lines 37 (patched)
> > <https://reviews.apache.org/r/67969/diff/1/?file=2061551#file2061551line37>
> >
> > I don't understand this comment.  There was update/insert done (line 
> > 25) since last analyze at line 22-23.  Shouldn't analyze at 34-35 change 
> > stats?
> > Or are they auto updated after each statement?

Yes. Updated the comment


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67969/#review206245
-------


On July 18, 2018, 11:19 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67969/
> ---
> 
> (Updated July 18, 2018, 11:19 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> 64f9c70f05 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
> 03cceace40 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 49709e596e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java
>  28d4de7f7b 
>   ql/src/test/queries/clientpositive/acid_no_buckets.q bcf9e0634b 
>   ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out 36a6a5d5d1 
> 
> 
> Diff: https://reviews.apache.org/r/67969/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 67969: HIVE-20115 Acid tables should not use footer scan for analyze

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67969/
---

Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
64f9c70f05 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
03cceace40 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 49709e596e 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 
28d4de7f7b 
  ql/src/test/queries/clientpositive/acid_no_buckets.q bcf9e0634b 
  ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out 36a6a5d5d1 


Diff: https://reviews.apache.org/r/67969/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20143) analyze doesn't mark partition column stats as accurate after truncate

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20143:
---

 Summary: analyze doesn't mark partition column stats as accurate 
after truncate
 Key: HIVE-20143
 URL: https://issues.apache.org/jira/browse/HIVE-20143
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Discovered while looking at txn stats. This works fine for truncate+analyze for 
non-partitioned tables, but not for partitions:
{noformat}
set hive.stats.dbclass=fs;
set hive.stats.fetch.column.stats=true;
set hive.stats.autogather=true;
set hive.stats.column.autogather=true;
set hive.compute.query.using.stats=true;
set hive.mapred.mode=nonstrict;
set hive.explain.user=false;
set hive.fetch.task.conversion=none;
set hive.query.results.cache.enabled=false;

create table stats_part1(key int,value string) partitioned by (p int);
insert into table stats_part1 partition(p=101) values (1, "foo");
insert into table stats_part1 partition(p=102) values (2, "bar");
explain select count(key) from stats_part1; -- from stats

truncate table stats_part1 partition(p=101);
explain select count(key) from stats_part1; -- not from stats, ok

analyze table stats_part1 partition(p) compute statistics for columns;
explain select count(key) from stats_part1; -- not from stats still
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20128) add partial read support to LLAP Parquet cache

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20128:
---

 Summary: add partial read support to LLAP Parquet cache
 Key: HIVE-20128
 URL: https://issues.apache.org/jira/browse/HIVE-20128
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Follow-up from HIVE-20127 to allow the reader to accumulate reads for several 
buffers until the aggregate data read covers the pre-determined cache 
boundaries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20127) fix some issues with LLAP Parquet cache

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20127:
---

 Summary: fix some issues with LLAP Parquet cache
 Key: HIVE-20127
 URL: https://issues.apache.org/jira/browse/HIVE-20127
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20117) schema changes for txn stats

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20117:
---

 Summary: schema changes for txn stats
 Key: HIVE-20117
 URL: https://issues.apache.org/jira/browse/HIVE-20117
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20109) get rid of COLUMN_STATS_ACCURATE

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20109:
---

 Summary: get rid of COLUMN_STATS_ACCURATE
 Key: HIVE-20109
 URL: https://issues.apache.org/jira/browse/HIVE-20109
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


I don't know why anyone would come up with an idea of storing a set of booleans 
in a database using JSON. This has caused various problems in the past (text 
field limitations, perf issues when parsing a giant string; also bugs because 
the way it is set is brittle).
However, now that we are implementing transactional stats, it becomes 
especially problematic and error prone because the code in Hive sets C_S_A in 
random places with reckless abandon, whereas we want to change the state of the 
stats in well defined places where txn semantics can be verified.
Currently in HIVE-19416, we are handling random things that touch it (from 
metastore itself to output committers, various stats tasks, commands like 
truncate, etc.) via a pile of hacks, but the best solution would be to remove 
it completely and replace with a DB table/columns in stats tables that would 
need to be set explicitly, not via generic alter_table.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20082) HiveDecimal to string conversion doesn't format the decimal correctly - master

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20082:
---

 Summary: HiveDecimal to string conversion doesn't format the 
decimal correctly - master
 Key: HIVE-20082
 URL: https://issues.apache.org/jira/browse/HIVE-20082
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be "0.0"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20083) HiveDecimal to string conversion doesn't format the decimal correctly - branch-2

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20083:
---

 Summary: HiveDecimal to string conversion doesn't format the 
decimal correctly - branch-2
 Key: HIVE-20083
 URL: https://issues.apache.org/jira/browse/HIVE-20083
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be "0.0"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20081) remove EnvironmentContext usage and add proper request APIs

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20081:
---

 Summary: remove EnvironmentContext usage and add proper request 
APIs
 Key: HIVE-20081
 URL: https://issues.apache.org/jira/browse/HIVE-20081
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


Optional, since because of old unrelated changes we cannot entirely get rid of 
EnvironmentContext.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20062) TestReplicationScenarios doesn't clean injection properly, causing bizarre interdependent failures

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20062:
---

 Summary: TestReplicationScenarios doesn't clean injection 
properly, causing bizarre interdependent failures
 Key: HIVE-20062
 URL: https://issues.apache.org/jira/browse/HIVE-20062
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20061) add a flag to turn off txn stats

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20061:
---

 Summary: add a flag to turn off txn stats
 Key: HIVE-20061
 URL: https://issues.apache.org/jira/browse/HIVE-20061
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20055) SQL injection via metastore ACID APIs (and maybe queries, although that's unlikely)

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20055:
---

 Summary: SQL injection via metastore ACID APIs (and maybe queries, 
although that's unlikely)
 Key: HIVE-20055
 URL: https://issues.apache.org/jira/browse/HIVE-20055
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Thejas M Nair


[~thejas] asked me to create this JIRA based on my earlier email :)

{noformat}
This might be doable with a specially crafted query, I’m not sure what APIs 
calls have what checks (e.g. via Hive parser) that would prevent the below.
However, for remote metastore (default on many clusters currently, afaik it’s 
the default for ACID) we expose thrift API that accepts strings e.g. 
get_valid_write_ids.
That passes the string table names to TxnHandler::getValidWriteIdsForTable, 
that inserts them into the query string w/quoteString call; quoteString doesn’t 
do any validation.

Some ready made delete statements also exist e.g.  "delete from REPL_TXN_MAP 
where RTM_SRC_TXN_ID = " + sourceTxnId + " and RTM_REPL_POLICY = " + 
quoteString(rqst.getReplPolicy());
I think my replication policy might be {' OR '1' = '1} ;)

So, SQL injection might be possible thru these APIs.
I wonder if this class should be switched to parameter based execution? 
DirectSQL could be used as an example, although that uses DataNucleus direct 
sql feature… at least we need some checks on these.
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20047) consider removing txnID argument for txn stats methods

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20047:
---

 Summary: consider removing txnID argument for txn stats methods
 Key: HIVE-20047
 URL: https://issues.apache.org/jira/browse/HIVE-20047
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


Followup from HIVE-19975.
W.r.t. write IDs and txn IDs, stats validity check currently verifies one of 
two things - that stats write ID is valid for query write ID list, or that 
stats txn ID (derived from write ID) is the same as the query txn ID.
I'm not sure the latter check is needed; removing it would allow us to make a 
bunch of APIs a little bit simpler.

[~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) 
observe stats written by the same txn; but in such manner that it doesn't have 
the write ID of the same-txn stats writer, in its valid write ID list? I'm 
assuming it's not possible, e.g. in multi statement txn each query would have 
the previous same-txn writer for the same table in its valid write ID list?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20046) remove NUM_FILES check or add a negative test

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20046:
---

 Summary: remove NUM_FILES check or add a negative test
 Key: HIVE-20046
 URL: https://issues.apache.org/jira/browse/HIVE-20046
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


{noformat}
// Since newly initialized empty table has 0 for the parameter.
if (Long.parseLong(statsParams.get(StatsSetupConst.NUM_FILES)) == 0) {
  return true;
}
{noformat}

This doesn't look safe; # of files could be set to 0 by an invalid update, or 
potentially a parallel update that we cannot see (not sure if this is possible; 
there's some code in metastore that updates basic stats outside of the scope of 
the query).

It would be better to remove this, and see if it breaks some tests. If we do 
need this, there should be a negative test at some point



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20029) add negative tests

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20029:
---

 Summary: add negative tests
 Key: HIVE-20029
 URL: https://issues.apache.org/jira/browse/HIVE-20029
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


We need a few tests, esp. for parallel case, where we verify that stats are NOT 
used.
Right now many code paths don't fail but return -1, null or whatever when 
something else is not present, so positive tests might pass because they skip 
the check, not because the check passes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67712: HIVE-19820 add ACID stats support to background stats updater

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67712/
---

(Updated June 27, 2018, 7:55 p.m.)


Review request for hive, Eugene Koifman and Seong (Steve) Yeom.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 580bae9c3f 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java 
ddca70497a 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
dd0929f2b9 
  ql/src/test/org/apache/hadoop/hive/ql/stats/TestStatsUpdaterThread.java 
14f86eabbc 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 706d831435 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 51e081b22f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2c3554edc4 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 8cc9d2c586 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 e4894fa12b 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 001c3edcff 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 d6a882e8e9 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/VerifyingObjectStore.java
 c9a6a471cb 


Diff: https://reviews.apache.org/r/67712/diff/2/

Changes: https://reviews.apache.org/r/67712/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 67712: HIVE-19820 add ACID stats support to background stats updater

[Sergey Shelukhin]

> On June 27, 2018, 4:23 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java
> > Lines 291 (patched)
> > <https://reviews.apache.org/r/67712/diff/1/?file=2044531#file2044531line291>
> >
> > there are several read ops in this txn - what semantics is the txn 
> > trying to achive here?

Added a comment


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67712/#review205443
-------


On June 23, 2018, 12:29 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67712/
> ---
> 
> (Updated June 23, 2018, 12:29 a.m.)
> 
> 
> Review request for hive, Eugene Koifman and Seong (Steve) Yeom.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  580bae9c3f1307325842a08275e085a8e31f9351 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java 
> ddca70497a3f51c3ec9ea532fac2a42aa36149b3 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
> dd0929f2b9748d83d55ccc271cec6aa07933bde1 
>   ql/src/test/org/apache/hadoop/hive/ql/stats/TestStatsUpdaterThread.java 
> 14f86eabbcf4bfc38c92294cd5d71d4905eb5c30 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  4296084381df1e109248820b96739a4eb5ee0490 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  51e081b22fa27b013715bb6eddf7fbbcf6bbd061 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9266879ad0134dbf87598af6f9305b73cc8c40ba 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  8cc9d2c586a411712d01d599ff2986f6ad5e0cfd 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  e4894fa12bfee78f51f3796e0ccaaf51c7ac4136 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  001c3edcff5a4d0ea67b73e83075b1f867342654 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  d6a882e8e98f92eefbdb7900bdf43e3274a21c5d 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/VerifyingObjectStore.java
>  c9a6a471cb7fc28845efb6d774601dba0cef2a85 
> 
> 
> Diff: https://reviews.apache.org/r/67712/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-20018) fix TestReplicationScenarios on the branch

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20018:
---

 Summary: fix TestReplicationScenarios on the branch
 Key: HIVE-20018
 URL: https://issues.apache.org/jira/browse/HIVE-20018
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Steve Yeom


{noformat}
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropPartition
 (batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
 (batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithDropPartitionedTable
 (batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
 (batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry
 (batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus 
(batchId=238)
{noformat}

Most tests have errors, not being able to find a table/partition in 
getPartition. Might be related to catalog handling, or something else.
Some have NPEs, some no obvious errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20005) acid_table_stats - query result change on the branch

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20005:
---

 Summary: acid_table_stats - query result change on the branch
 Key: HIVE-20005
 URL: https://issues.apache.org/jira/browse/HIVE-20005
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Steve Yeom


Queries in acid_table_stats.q test on TestCliDriver have explain changes from 
running the query to using stats.
However, one of the queries also has the result change (from 1000 to 2000); new 
result is likely invalid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: HiveQA is broken

[Sergey Shelukhin]

Might be related to
https://stackoverflow.com/questions/50946420/could-not-transfer-artifact-ht
tps-repo-maven-apache-org-maven2-received-fat, similar issue in
https://blog.gradle.org/unable-to-download-maven-central-bintray

On 18/6/26, 11:51, "Sergey Shelukhin"  wrote:

>Was some change made w/versions?
>
>RROR] Plugin org.apache.maven.plugins:maven-clean-plugin:2.5 or one of
>its dependencies could not be resolved: Failed to read artifact
>descriptor for org.apache.maven.plugins:maven-clean-plugin:jar:2.5: Could
>not transfer artifact org.apache.maven.plugins:maven-clean-plugin:pom:2.5
>from/to central (https://repo.maven.apache.org/maven2): Received fatal
>alert: protocol_version -> [Help 1]
>[ERROR]
>[ERROR] To see the full stack trace of the errors, re-run Maven with the
>-e switch.
>[ERROR] Re-run Maven using the -X switch to enable full debug logging.
>[ERROR]
>[ERROR] For more information about the errors and possible solutions,
>please read the following articles:
>[ERROR] [Help 1] 
>http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
>+ return 1
>+ ret=1
>+ unpack_test_results
>+ '[' -z 
>/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build ']'
>+ cd 
>/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive
>/testutils/ptest2/target
>jenkins-execute-build.sh: line 61: cd:
>/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive
>/testutils/ptest2/target: No such file or directory
>+ [[ -f test-results.tar.gz ]]
>+ exit 1
>+ rm -f /tmp/tmp.evhJJyv3A1
>Build step 'Execute shell' marked build as failure
>Recording test results
>ERROR: Step ‘Publish JUnit test result report’ failed: No test report
>files were found. Configuration error?
>[description-setter] Description set: HIVE-19481 / branch-3-mr2
>Finished: FAILURE
>
>

HiveQA is broken

[Sergey Shelukhin]

Was some change made w/versions?

RROR] Plugin org.apache.maven.plugins:maven-clean-plugin:2.5 or one of its 
dependencies could not be resolved: Failed to read artifact descriptor for 
org.apache.maven.plugins:maven-clean-plugin:jar:2.5: Could not transfer 
artifact org.apache.maven.plugins:maven-clean-plugin:pom:2.5 from/to central 
(https://repo.maven.apache.org/maven2): Received fatal alert: protocol_version 
-> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
+ return 1
+ ret=1
+ unpack_test_results
+ '[' -z /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build 
']'
+ cd 
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target
jenkins-execute-build.sh: line 61: cd: 
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target:
 No such file or directory
+ [[ -f test-results.tar.gz ]]
+ exit 1
+ rm -f /tmp/tmp.evhJJyv3A1
Build step 'Execute shell' marked build as failure
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files 
were found. Configuration error?
[description-setter] Description set: HIVE-19481 / branch-3-mr2
Finished: FAILURE

Review Request 67712: HIVE-19820 add ACID stats support to background stats updater

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67712/
---

Review request for hive, Eugene Koifman and Seong (Steve) Yeom.


Repository: hive-git


Description
---

see jira


Diffs
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 580bae9c3f1307325842a08275e085a8e31f9351 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java 
ddca70497a3f51c3ec9ea532fac2a42aa36149b3 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
dd0929f2b9748d83d55ccc271cec6aa07933bde1 
  ql/src/test/org/apache/hadoop/hive/ql/stats/TestStatsUpdaterThread.java 
14f86eabbcf4bfc38c92294cd5d71d4905eb5c30 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 4296084381df1e109248820b96739a4eb5ee0490 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 51e081b22fa27b013715bb6eddf7fbbcf6bbd061 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 9266879ad0134dbf87598af6f9305b73cc8c40ba 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 8cc9d2c586a411712d01d599ff2986f6ad5e0cfd 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 e4894fa12bfee78f51f3796e0ccaaf51c7ac4136 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 001c3edcff5a4d0ea67b73e83075b1f867342654 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 d6a882e8e98f92eefbdb7900bdf43e3274a21c5d 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/VerifyingObjectStore.java
 c9a6a471cb7fc28845efb6d774601dba0cef2a85 


Diff: https://reviews.apache.org/r/67712/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-19916) master-txnstats branch - integrate with HIVE-19382

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19916:
---

 Summary: master-txnstats branch - integrate with HIVE-19382
 Key: HIVE-19916
 URL: https://issues.apache.org/jira/browse/HIVE-19916
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Steve Yeom


There's some code in original txn stats patch that may go to metastore to get 
write Ids. This code should not go to metastore, it should fail instead. 
HIVE-19382 should ensure that we have correct IDs already present during 
optimizer - they are using by e.g. materialized view optimizer, so they should 
be there; if they are not present, some integration might be needed so that txn 
stats optimizations also have access to those write Ids.

cc [~jcamachorodriguez]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19915) master-txnstats branch - support CachedStore

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19915:
---

 Summary: master-txnstats branch - support CachedStore
 Key: HIVE-19915
 URL: https://issues.apache.org/jira/browse/HIVE-19915
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Steve Yeom






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19914) master-txnstats branch - make sure SQL changes are in correct upgrade scripts

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19914:
---

 Summary: master-txnstats branch - make sure SQL changes are in 
correct upgrade scripts
 Key: HIVE-19914
 URL: https://issues.apache.org/jira/browse/HIVE-19914
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


The initial commit changed multiple files e.g.
{noformat}
standalone-metastore/src/main/sql/mysql/hive-schema-3.0.0.mysql.sql
standalone-metastore/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
standalone-metastore/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql
standalone-metastore/src/main/sql/mysql/upgrade-3.1.0-to-4.0.0.mysql.sql
{noformat}

The target version is currently 4.0 (or 3.1? cc [~hagleitn]), so all the 
changes should be in the scripts upgrading to 4.0
cc [~vgarg]




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19891) inserting into external tables with custom partition directories may cause data loss

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19891:
---

 Summary: inserting into external tables with custom partition 
directories may cause data loss
 Key: HIVE-19891
 URL: https://issues.apache.org/jira/browse/HIVE-19891
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19866) improve cache purge

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19866:
---

 Summary: improve cache purge
 Key: HIVE-19866
 URL: https://issues.apache.org/jira/browse/HIVE-19866
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


1) Memory needs to be accounted for.
2) LRFU eviction doesn't need to maintain state between individual removals.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19858) make interface change from TEZ-3951 non-breaking

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19858:
---

 Summary: make interface change from TEZ-3951 non-breaking
 Key: HIVE-19858
 URL: https://issues.apache.org/jira/browse/HIVE-19858
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 67515: HIVE-19838 simplify & fix ColumnizedDeleteEventRegistry load loop

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67515/
---

Review request for hive, Eugene Koifman and Teddy Choi.


Repository: hive-git


Description
---

see jira


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 a4568de696 


Diff: https://reviews.apache.org/r/67515/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19838:
---

 Summary: simplify & fix ColumnizedDeleteEventRegistry load loop
 Key: HIVE-19838
 URL: https://issues.apache.org/jira/browse/HIVE-19838
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


Apparently sometimes the delete count in ACID stats doesn't match what merger 
actually returns.
It could be due to some deltas having duplicate deletes from parallel queries 
(I guess?) or some other reasons beyond my mortal comprehension.

The loop assumes the merger will return the exact number of records, so it 
fails with array index exception. Also, it could actually be done in a single 
loop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19833) reduce LLAP IO min allocation to match ORC variable CB size

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19833:
---

 Summary: reduce LLAP IO min allocation to match ORC variable CB 
size
 Key: HIVE-19833
 URL: https://issues.apache.org/jira/browse/HIVE-19833
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19826) OrcRawRecordMerger doesn't work for more than one file

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19826:
---

 Summary: OrcRawRecordMerger doesn't work for more than one file
 Key: HIVE-19826
 URL: https://issues.apache.org/jira/browse/HIVE-19826
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


Key object in the map is reused and reset, leading to bizarre merges and wrong 
results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19820) add ACID stats support to background stats updater

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19820:
---

 Summary: add ACID stats support to background stats updater
 Key: HIVE-19820
 URL: https://issues.apache.org/jira/browse/HIVE-19820
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


Follow-up from HIVE-19418.
Right now it checks whether stats are valid in an old-fashioned way... and also 
gets ACID state, and discards it without using.
When ACID stats are implemented, ACID state needs to be used to do 
version-aware valid stats checks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19819) fix/remove flaky test: flaky test: TestCliDriver#input31

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19819:
---

 Summary: fix/remove flaky test:  flaky test: TestCliDriver#input31
 Key: HIVE-19819
 URL: https://issues.apache.org/jira/browse/HIVE-19819
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Follow up from HIVE-19778.
This test uses some sort of test mode and may not be useful.
If it is, it should be fixed (actual query results are flaky); it's disabled 
for now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19793) disable LLAP IO batch-to-row wrapper for ACID deletes/updates

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19793:
---

 Summary: disable LLAP IO batch-to-row wrapper for ACID 
deletes/updates
 Key: HIVE-19793
 URL: https://issues.apache.org/jira/browse/HIVE-19793
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19789) reenable orc_llap test

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19789:
---

 Summary: reenable orc_llap test
 Key: HIVE-19789
 URL: https://issues.apache.org/jira/browse/HIVE-19789
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Matt McCline


Test has been disabled, looks like by mistake (or due to some issue with the 
patch there that was never addressed), in HIVE-11394.
It needs to be reenabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 67406: HIVE-19663 refactor LLAP IO report generation

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67406/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
f4a549c529 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
 6a361fa4fc 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 
10a4bfbb58 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapOomDebugDump.java 
29ddf5c112 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
af1b699771 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
5e102d93de 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
 31cd45f103 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java 
33236362b5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
 f7f80a8958 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
 7787cb4867 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/MemoryManager.java 
65b1d4f654 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SerDeLowLevelCacheImpl.java
 cb89d12e80 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleBufferManager.java 
a1b6caeb60 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
5003d9b5e5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/MetadataCache.java 
0184e3053f 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java 
714efef182 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
 b19cdcf531 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
 58c918c200 


Diff: https://reviews.apache.org/r/67406/diff/1/


Testing
---


Thanks,

Sergey Shelukhin