[HIVE-22601] Some columns will be lost when a UDTF has multiple aliases in some cases

2020-01-20 Thread Makoto Yui 
Hi committers,

Could someone please review this ticket and my colleague's patch?
https://issues.apache.org/jira/browse/HIVE-22601

The following query produces a wrong result in the current stable
release of Hive (2.3.6 and 3.1.2).

SELECT stack(1, 'a', 'b', 'c') AS (c1, c2, c3)
UNION ALL
SELECT stack(1, 'd', 'e', 'f') AS (c1, c2, c3);

+-+
| _u1.c3  |
+-+
| c   |
| f   |
+-+

The same issue happened for

select
 posexplode(split('1,4', ',')) as (seq, col)
union all
select
 posexplode(split('2,6', ',')) as (seq, col)

_u1.col
1
4
2
6

Thanks,
Makoto

[jira] [Created] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors

2020-01-20 Thread Rajesh Balamohan (Jira) 
Rajesh Balamohan created HIVE-22753:
---

 Summary: Fix gradual mem leak: Operationlog related appenders 
should be cleared up on errors 
 Key: HIVE-22753
 URL: https://issues.apache.org/jira/browse/HIVE-22753
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan
 Attachments: image-2020-01-21-11-14-37-911.png

In case of exception in SQLOperation, operational log does not get cleared up. 
This causes gradual build up of HushableRandomAccessFileAppender causing HS2 to 
OOM after some time.

!image-2020-01-21-11-14-37-911.png|width=431,height=267!

!2Q==|width=487,height=204!

 

Prod instance mem

!Z|width=531,height=159!

 

Each HushableRandomAccessFileAppender holds internal ref to 
RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem leak.

Related ticket: HIVE-18820



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22752) HiveMetastore addWriteNotificationLog should be invoked only when listeners are enabled

2020-01-20 Thread Rajesh Balamohan (Jira) 
Rajesh Balamohan created HIVE-22752:
---

 Summary: HiveMetastore addWriteNotificationLog should be invoked 
only when listeners are enabled
 Key: HIVE-22752
 URL: https://issues.apache.org/jira/browse/HIVE-22752
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Rajesh Balamohan


[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L8109]

 

Even though listeners are turned off, it gets executed and causes load on the 
system. This should be guarded by listener checks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22751) Move locking in HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper

2020-01-20 Thread Rajesh Balamohan (Jira) 
Rajesh Balamohan created HIVE-22751:
---

 Summary: Move locking in HiveServer2::isDeregisteredWithZooKeeper 
to ZooKeeperHiveHelper
 Key: HIVE-22751
 URL: https://issues.apache.org/jira/browse/HIVE-22751
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan


[https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/server/HiveServer2.java#L620]

[https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/SessionManager.java#L597]

 

When queries are run in beeline and closed, it causes unwanted delays in 
shutting down beeline.  Here is the threaddump from server side, which shows 
HiveServer2 lock contention.

 

It would be good to move synchronization to 
"zooKeeperHelper.isDeregisteredWithZooKeeper"

 
{noformat}
"main" #1 prio=5 os_prio=0 tid=0x7f78b0078800 nid=0x2d1c waiting on 
condition [0x7f78b968c000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xac8d5ff0> (a 
java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:187)
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123)
- locked <0xa9c5f2a8> (a java.lang.Object)
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115)
at 
org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:790)
at 
org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:763)
at 
org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:687)
- locked <0xa99bd568> (a 
org.apache.hive.service.server.HiveServer2)
at 
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1016)
at 
org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137)
at 
org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1294)
at 
org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
"HiveServer2-HttpHandler-Pool: Thread-50" #50 prio=5 os_prio=0 
tid=0x7f78b3e60800 nid=0x2fa7 waiting for monitor entry [0x7f7884edf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
org.apache.hive.service.server.HiveServer2.isDeregisteredWithZooKeeper(HiveServer2.java:600)
- waiting to lock <0xa99bd568> (a 
org.apache.hive.service.server.HiveServer2)
at 
org.apache.hive.service.cli.session.SessionManager.closeSessionInternal(SessionManager.java:631)
at 
org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:621)
- locked <0xaa1970b0> (a 
org.apache.hive.service.cli.session.SessionManager)
at 
org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:244)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.CloseSession(ThriftCLIService.java:527)
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1517)
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1502)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
at 
org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:237)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at

Re: [VOTE] Apache Hive 3.1.3 Release Candidate 0

2020-01-20 Thread Alan Gates 
+1.  I checked the signature and hash, did a build, and checked the rat
output.

Alan.

On Wed, Jan 15, 2020 at 2:08 PM Naveen Gangam 
wrote:

> Apache Hive 3.1.3 Release Candidate 0 is available here:
> https://people.apache.org/~ngangam/hive-3.1.3-rc-0
>
>
> Maven artifacts are available
> here:
> https://repository.apache.org/content/repositories/orgapachehive-1099/
>
>
> The tag release-3.1.3-rc0 has been applied to the source for this
> release in github, you can see it
> athttps://github.com/apache/hive/tree/release-3.1.3-rc0
>
> Voting will conclude in 72 hours.
>
> Hive PMC Members: Please test and vote.
>
> Thanks.
>

Re: Review Request 72028: HIVE-22729: Provide a failure reason for failed compactions

2020-01-20 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72028/#review219335
---


Ship it!




LGTM, 1 minor comment


ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
Lines 127 (patched)


Wouldn't it be helpul to add treshold param for more clarity ("Compaction 
is not initiated since last 
HiveConf.ConfVars.COMPACTOR_INITIATOR_FAILED_THRESHOLD consecutive compaction 
attemps failed) ?


- Denys Kuzmenko


On Jan. 20, 2020, 12:51 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72028/
> ---
> 
> (Updated Jan. 20, 2020, 12:51 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22729: Provide a failure reason for failed compactions
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsDesc.java
>  9348efc5a12b50f55f5952094882e941158405fd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java
>  517d88237cc3b8f0316727bf1eebfc6535152fae 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 
> 6f642901203ab73699ed694009d48ca77263fb10 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 6017fd31b1c180c144c0d37dddc56c44d9d8a05b 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
> 5aff71e0e981c429f85663300d3e5c21089529a9 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
>  e5895547e6006f30a37b5ba0b1ce42253129d3b6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
>  4aee45ce5f0e534823194bc84d13b88210ce0b3c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponseElement.java
>  8a5682a013b24f8dcf7ad3fdb0b0b606d82cc7c0 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  093ad4be273f3544e013c608ba3ca0a66cfaea20 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  0dcca59b6812a892055d5d0615c1c479e8d8d030 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  8d7c32a7658535b2a55695813f5e98a95f79f0a4 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 72ccdd1a0f62ed1c034b17dfbd84a8ab3f49befd 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
>  ba45f3945274853fdc84487d93c4c00ff2982541 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  aded6f5486cc840f397347b39049310009fd3bad 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
>  da5dd61d08e2ca8fe5e80ffdf9fb4a6f4c4d0ba3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  1dc3867929fcdfef53da4859fbe8de2e89435166 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  67102718867233f29ddb2ea8ec3fbcb6560c6c30 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  ae0a32541a4bb9179b2bb71ae9f9098d7b35a88e 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  221d4f1fffb682aaec3af22a339e7a3077a75f6a 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  bc98d5fc4a5637988c97f2e5a0e02d3be16ae0cb 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  dd761a66db4826580a67d64879e4c85278b8e20c 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  6a040a6a64c2086b5eb68a397697c9e2d2ca4d76 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  f5ec1ba1aff89d02b66d6a2cd1da8de1b3b08d06 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  c7738be2732b839aa2b460733c092e368909f935 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
>  455f98b72578ff977e29301cd2fc595ae80ee4ca 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
>  5c39b0d9f4d27ab82ef44392818c1810cb7664ce 
> 
> 
> Diff: https://reviews.apache.org/r/72028/diff/1/
> 
> 
> Testing
> ---
> 
> 
>

Review Request 72028: HIVE-22729: Provide a failure reason for failed compactions

2020-01-20 Thread Laszlo Pinter via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72028/
---

Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.


Repository: hive-git


Description
---

HIVE-22729: Provide a failure reason for failed compactions


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsDesc.java
 9348efc5a12b50f55f5952094882e941158405fd 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java
 517d88237cc3b8f0316727bf1eebfc6535152fae 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 
6f642901203ab73699ed694009d48ca77263fb10 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
6017fd31b1c180c144c0d37dddc56c44d9d8a05b 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
5aff71e0e981c429f85663300d3e5c21089529a9 
  
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java 
e5895547e6006f30a37b5ba0b1ce42253129d3b6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
 4aee45ce5f0e534823194bc84d13b88210ce0b3c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponseElement.java
 8a5682a013b24f8dcf7ad3fdb0b0b606d82cc7c0 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 093ad4be273f3544e013c608ba3ca0a66cfaea20 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 0dcca59b6812a892055d5d0615c1c479e8d8d030 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 8d7c32a7658535b2a55695813f5e98a95f79f0a4 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
72ccdd1a0f62ed1c034b17dfbd84a8ab3f49befd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 ba45f3945274853fdc84487d93c4c00ff2982541 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 aded6f5486cc840f397347b39049310009fd3bad 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
 da5dd61d08e2ca8fe5e80ffdf9fb4a6f4c4d0ba3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 1dc3867929fcdfef53da4859fbe8de2e89435166 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 67102718867233f29ddb2ea8ec3fbcb6560c6c30 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 ae0a32541a4bb9179b2bb71ae9f9098d7b35a88e 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 221d4f1fffb682aaec3af22a339e7a3077a75f6a 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 bc98d5fc4a5637988c97f2e5a0e02d3be16ae0cb 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 dd761a66db4826580a67d64879e4c85278b8e20c 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 6a040a6a64c2086b5eb68a397697c9e2d2ca4d76 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 f5ec1ba1aff89d02b66d6a2cd1da8de1b3b08d06 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 c7738be2732b839aa2b460733c092e368909f935 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 455f98b72578ff977e29301cd2fc595ae80ee4ca 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 5c39b0d9f4d27ab82ef44392818c1810cb7664ce 


Diff: https://reviews.apache.org/r/72028/diff/1/


Testing
---


Thanks,

Laszlo Pinter

[jira] [Created] (HIVE-22750) Consolidate LockType naming

2020-01-20 Thread Zoltan Chovan (Jira) 
Zoltan Chovan created HIVE-22750:


 Summary: Consolidate LockType naming
 Key: HIVE-22750
 URL: https://issues.apache.org/jira/browse/HIVE-22750
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Reporter: Zoltan Chovan
Assignee: Zoltan Chovan




Extend enum with string literal to remove unnecessary `id` to `char` casting 
for the LockType:


{code:java}
switch (lockType) {
case EXCLUSIVE:
  lockChar = LOCK_EXCLUSIVE;
  break;
case SHARED_READ:
  lockChar = LOCK_SHARED;
  break;
case SHARED_WRITE:
  lockChar = LOCK_SEMI_SHARED;
  break;
  }
{code}


Consolidate LockType naming in code and schema upgrade scripts:


{code:java}
CASE WHEN HL.`HL_LOCK_TYPE` = 'e' THEN 'exclusive' WHEN HL.`HL_LOCK_TYPE` = 'r' 
THEN 'shared' WHEN HL.`HL_LOCK_TYPE` = 'w' THEN *'semi-shared'* END AS 
LOCK_TYPE,

{code}

EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: If Hive Metastore is compatibility with MariaDB version 10.x.?

2020-01-20 Thread Zoltan Haindrich 

Hello,

Locally I use Mariadb 10.4.8 when I validate metastore schema/etc changes. So 
far, I've not uncovered any issues with it...

I'm planning to integrate some kind of smoke tests against all the supported 
DBs to help uncover metastore related issues earlier.
To evaluate that we have everything working with mysql; it might be usefull to 
see how well our existing tests behave on that version.
For that purpose the following things might be usefull:
* there is a way to run qtest-s against different metastores: see HIVE-21954
* the hive-dev-box can be used to configure and launch hive with 
postgres/mysql/ora/mssql https://github.com/kgyrtkirk/hive-dev-box

cheers,
Zoltan


On 1/17/20 1:28 PM, Oleksiy S wrote:

Hi all.

Could you please help? Customer asked if Hive Metastore is compatible with 
MariaDB version 10.x. He is going to use 10.4.10-MariaDB MariaDB Server.

--
Oleksiy