Re: Review Request 66098: LLAP: report guaranteed tasks count in AM registry to check for consistency

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66098/#review199309
---




llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 595 (patched)


why do we have to set uniqId again? it should be set during registration 
right?



llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
Lines 255 (patched)


nit: should this start with 0?



llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
Lines 518 (patched)


nit: log the version?


- Prasanth_J


On March 15, 2018, 8:15 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66098/
> ---
> 
> (Updated March 15, 2018, 8:15 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5b1f 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571b8d 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dddf7 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b805a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java 
> 82b38d55b4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/QueryAllocationManager.java 
> a44690283b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 
> 0962460b8f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java d4c3ab9952 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> 00e2c20eeb 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 8d185ba801 
> 
> 
> Diff: https://reviews.apache.org/r/66098/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-18973) Make transaction system work with catalogs

[Alan Gates (JIRA)]

Alan Gates created HIVE-18973:
-

 Summary: Make transaction system work with catalogs
 Key: HIVE-18973
 URL: https://issues.apache.org/jira/browse/HIVE-18973
 Project: Hive
  Issue Type: Sub-task
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Alan Gates
Assignee: Alan Gates


The transaction tables need to understand catalogs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18972) beeline command suggestion to kill job deprecated

[Bharathkrishna Guruvayoor Murali (JIRA)]

Bharathkrishna Guruvayoor Murali created HIVE-18972:
---

 Summary: beeline command suggestion to kill job deprecated
 Key: HIVE-18972
 URL: https://issues.apache.org/jira/browse/HIVE-18972
 Project: Hive
  Issue Type: Bug
Reporter: Bharathkrishna Guruvayoor Murali
Assignee: Bharathkrishna Guruvayoor Murali


When I run a beeline command that uses YARN:
{code}
INFO : The url to track the job: 
http://vd0514.halxg.cloudera.com:8088/proxy/application_1488996234407_0010/
INFO : Starting Job = job_1488996234407_0010, Tracking URL = 
http://vd0514.halxg.cloudera.com:8088/proxy/application_1488996234407_0010/
INFO : Kill Command = 
/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.9/lib/hadoop/bin/hadoop job 
-kill job_1488996234407_0010
{code}
If I then try to kill the job using that command:
{code}
[systest@vd0514 ~]$ 
/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.9/lib/hadoop/bin/hadoop job 
-kill job_1488996234407_0010
DEPRECATED: Use of this script to execute mapred command is deprecated.
Instead use the mapred command for it.
{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hive pull request #320: HIVE-18755 Modifications to metastore for catalog

[alanfgates]

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/320

HIVE-18755 Modifications to metastore for catalog



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive18755

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/320.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #320


commit 697e0428b4046b0c36d413a74fc46c2cf28d6e67
Author: Alan Gates 
Date:   2018-03-15T22:51:06Z

HIVE-18755 Modifications to metastore for catalog




---

[jira] [Created] (HIVE-18971) add HS2 WM metrics for use in Grafana and such

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18971:
---

 Summary: add HS2 WM metrics for use in Grafana and such
 Key: HIVE-18971
 URL: https://issues.apache.org/jira/browse/HIVE-18971
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


AM metrics need to be tagged with something like dag ID.
HS2 should have metrics added per pool, tagged accordingly. Not clear if HS2 
even sets up metrics right now...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18970) add and improve WM metrics for use in Grafana and such

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18970:
---

 Summary: add and improve WM metrics for use in Grafana and such
 Key: HIVE-18970
 URL: https://issues.apache.org/jira/browse/HIVE-18970
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


AM metrics need to be tagged them with something like dag ID.
HS2 should have metrics added per pool, tagged accordingly. Not clear if HS2 
even sets up metrics right now...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66075: HIVE-18962 add WM task state to Tez AM heartbeat

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66075/#review199290
---


Ship it!




Ship It!

- Jason Dere


On March 15, 2018, 8:53 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66075/
> ---
> 
> (Updated March 15, 2018, 8:53 p.m.)
> 
> 
> Review request for hive and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java
>  d97b1567ea 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  a2dca1b319 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java 
> b784360b6c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  75d8577d6a 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestAMReporter.java
>  bde354649a 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  a4bd4dfd88 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b805a 
>   
> llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
>  90b31e4fcd 
> 
> 
> Diff: https://reviews.apache.org/r/66075/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66075: HIVE-18962 add WM task state to Tez AM heartbeat

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66075/
---

(Updated March 15, 2018, 8:53 p.m.)


Review request for hive and Jason Dere.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java
 d97b1567ea 
  
llap-common/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
 a2dca1b319 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java 
b784360b6c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 75d8577d6a 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestAMReporter.java
 bde354649a 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 a4bd4dfd88 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b805a 
  
llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
 90b31e4fcd 


Diff: https://reviews.apache.org/r/66075/diff/2/

Changes: https://reviews.apache.org/r/66075/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 66075: HIVE-18962 add WM task state to Tez AM heartbeat

[Sergey Shelukhin]

> On March 15, 2018, 5:53 p.m., Jason Dere wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java
> > Line 541 (original), 547 (patched)
> > 
> >
> > The previous implementation will lock the tasks member from any 
> > additional updates when getting the TaskSnapshot. The newer implementation 
> > could potentially allow another caller to update the tasks map - does that 
> > matter at all here?

Shouldn't matter; added a comment.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66075/#review199231
---


On March 14, 2018, 11:45 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66075/
> ---
> 
> (Updated March 14, 2018, 11:45 p.m.)
> 
> 
> Review request for hive and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java
>  d97b1567ea 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  a2dca1b319 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java 
> b784360b6c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  75d8577d6a 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestAMReporter.java
>  bde354649a 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  a4bd4dfd88 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b805a 
>   
> llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
>  90b31e4fcd 
> 
> 
> Diff: https://reviews.apache.org/r/66075/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-18969) Hive throws NPE when writing struct and list types data to a HBase backed table

[Kryvenko Igor (JIRA)]

Kryvenko Igor created HIVE-18969:


 Summary: Hive throws NPE when writing struct and list types data 
to a HBase backed table
 Key: HIVE-18969
 URL: https://issues.apache.org/jira/browse/HIVE-18969
 Project: Hive
  Issue Type: Bug
Reporter: Kryvenko Igor
Assignee: Kryvenko Igor


"INSERT INTO" query fails with NPE when we inserting STRUCT datatype with null 
values into a table stored in Hbase. The same issue for MAP datatype is already 
resolved in https://issues.apache.org/jira/browse/HIVE-13065

*Steps to reproduce*
*1. Create tables:*

{code}
CREATE TABLE IF NOT EXISTS t1 (id INT);
INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);

CREATE TABLE IF NOT EXISTS `htable`(
  `id` INT, 
  `map_column` STRUCT)
ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' 
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
WITH SERDEPROPERTIES ( 
  'hbase.columns.mapping'=':key,id:id', 
  'serialization.format'='1')
TBLPROPERTIES ( 'hbase.table.name'='/tmp/h');

{code}
*2. Insert into table stored the struct with NULL value in it:*
{code}
INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS 
INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
{code}

The mapreduce job for insert query fails. Error messages are as below:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.hadoop.hive.serde2.SerDeException: java.lan$
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
at 
org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:63)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:348)
... 17 more
Caused by: org.apache.hadoop.hive.serde2.SerDeException: 
java.lang.NullPointerException
at 
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:301)
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:714)
... 22 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObject$
at 
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:239)
at 
org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
at 
org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:295)
at 
org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
at 
org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
at 
org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
at 
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:297)

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 66098: LLAP: report guaranteed tasks count in AM registry to check for consistency

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66098/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5b1f 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571b8d 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dddf7 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b805a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java 
82b38d55b4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/QueryAllocationManager.java 
a44690283b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 0962460b8f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java d4c3ab9952 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
00e2c20eeb 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
8d185ba801 


Diff: https://reviews.apache.org/r/66098/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-18968) LLAP: report guaranteed tasks count in AM registry to check for consistency

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18968:
---

 Summary: LLAP: report guaranteed tasks count in AM registry to 
check for consistency
 Key: HIVE-18968
 URL: https://issues.apache.org/jira/browse/HIVE-18968
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18967) Standalone metastore SQL upgrade scripts do not properly set schema version

[Alan Gates (JIRA)]

Alan Gates created HIVE-18967:
-

 Summary: Standalone metastore SQL upgrade scripts do not properly 
set schema version
 Key: HIVE-18967
 URL: https://issues.apache.org/jira/browse/HIVE-18967
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Affects Versions: 3.0.0
Reporter: Alan Gates
Assignee: Alan Gates
 Fix For: 3.0.0


The new combined upgrade scripts for Hive 2.3 to 3.0 transition do not properly 
set the schema version after they have completed the upgrade.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18966) LLAP should not shut down when some random thread goes down

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18966:
---

 Summary: LLAP should not shut down when some random thread goes 
down
 Key: HIVE-18966
 URL: https://issues.apache.org/jira/browse/HIVE-18966
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


LlapDaemonUncaughtExceptionHandler shuts down LLAP on any thread failure.
Not sure it's the best idea... 3rd party libraries like permitted UDFs or 
Storage Handlers (e.g. Druid recently) can have errors that should not bring 
the entire daemon down.
Perhaps we can go by thread name pattern?
Overall logging the error, unless it's an OOM or other Error, might be better.

We can also add error handling to important threads like schedulers, if it's 
missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18965) HIVE-17990 didn't update derby SQL scripts

[Alan Gates (JIRA)]

Alan Gates created HIVE-18965:
-

 Summary: HIVE-17990 didn't update derby SQL scripts
 Key: HIVE-18965
 URL: https://issues.apache.org/jira/browse/HIVE-18965
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Alan Gates
Assignee: Alan Gates
 Fix For: 3.0.0


Owing to HIVE-17983, there are currently two copies of the derby installation 
and upgrade script.  HIVE-17990 only updated the one in standalone-metastore.  
The one in metastore needs to be updated as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66075: HIVE-18962 add WM task state to Tez AM heartbeat

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66075/#review199231
---




llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java
Lines 412 (patched)


nit whitespace



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java
Line 541 (original), 547 (patched)


The previous implementation will lock the tasks member from any additional 
updates when getting the TaskSnapshot. The newer implementation could 
potentially allow another caller to update the tasks map - does that matter at 
all here?



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java
Lines 590 (patched)


nit


- Jason Dere


On March 14, 2018, 11:45 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66075/
> ---
> 
> (Updated March 14, 2018, 11:45 p.m.)
> 
> 
> Review request for hive and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java
>  d97b1567ea 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  a2dca1b319 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java 
> b784360b6c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  75d8577d6a 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestAMReporter.java
>  bde354649a 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  a4bd4dfd88 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b805a 
>   
> llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
>  90b31e4fcd 
> 
> 
> Diff: https://reviews.apache.org/r/66075/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-18964) HiveServer2 should not log errors when clients issue connection reset

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-18964:
--

 Summary: HiveServer2 should not log errors when clients issue 
connection reset
 Key: HIVE-18964
 URL: https://issues.apache.org/jira/browse/HIVE-18964
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 2.3.2, 3.0.0
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


HiveServer2 logs an ugly exception trace when clients issue a connection reset. 
There is nothing we can do when the connection is reset. This exception trace 
should either be ignored or logged only in debug mode.

Things become worse if you are using load balancer like HAProxy which has its 
own health checks. HAProxy issues a connection reset to "quickly" close the 
connection once it finds that HS2 is up and available. This spams the logs at a 
very high frequency and makes them unusable for debugging purposes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65985: HIVE-18783: ALTER TABLE post-commit listener does not include the transactional listener responses

[Sergio Pena via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65985/
---

(Updated March 15, 2018, 5:45 p.m.)


Review request for hive, Alexander Kolbasov, Sahil Takiar, and Vihang 
Karajgaonkar.


Bugs: HIVE-18783
https://issues.apache.org/jira/browse/HIVE-18783


Repository: hive-git


Description
---

HIVE-16164 introduced a mechanism to pass HMS notification events ID to the 
post-commit listeners for all DDL operations, but it didn't add it to the ALTER 
TABLE event. This patch in review adds the same behavior for ALTER TABLE events.


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
 e0e29652da94bbdaca515a17955d1409824c1742 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 0dd3eb101709969a77998e1488e1c97214426cd3 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 66353e769b2d633ad9dfad2bcae25e8ad90f61d1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IHMSHandler.java
 e6de0013bc1be12b2772e2e97102ed476cf5 


Diff: https://reviews.apache.org/r/65985/diff/2/

Changes: https://reviews.apache.org/r/65985/diff/1-2/


Testing
---

All tests passed.


Thanks,

Sergio Pena

Re: Review Request 65985: HIVE-18783: ALTER TABLE post-commit listener does not include the transactional listener responses

[Sergio Pena via Review Board]

> On March 12, 2018, 4:45 p.m., Na Li wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
> > Lines 372 (patched)
> > 
> >
> > why when db name is changed at alter table,  the 
> > transactionalListenerResponses is not past into 
> > MetaStoreListenerNotifier.notifyEvent()? And the command won't be able to 
> > block until sentry gets the notification of alter table.

Good catch. I added other variables for each transaction response.


- Sergio


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65985/#review199027
---


On March 8, 2018, 4:11 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65985/
> ---
> 
> (Updated March 8, 2018, 4:11 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-18783
> https://issues.apache.org/jira/browse/HIVE-18783
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16164 introduced a mechanism to pass HMS notification events ID to the 
> post-commit listeners for all DDL operations, but it didn't add it to the 
> ALTER TABLE event. This patch in review adds the same behavior for ALTER 
> TABLE events.
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
>  e0e29652da94bbdaca515a17955d1409824c1742 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  89354a2d34249903a9ff13c4ed913a68de93057e 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  662de9a66767f27f31998f14c68f854e59993ab6 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IHMSHandler.java
>  e6de0013bc1be12b2772e2e97102ed476cf5 
> 
> 
> Diff: https://reviews.apache.org/r/65985/diff/1/
> 
> 
> Testing
> ---
> 
> All tests passed.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 65985: HIVE-18783: ALTER TABLE post-commit listener does not include the transactional listener responses

[Sergio Pena via Review Board]

> On March 13, 2018, 3:35 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
> > Lines 362 (patched)
> > 
> >
> > Would it make sense to move this out to its own try/catch block? This 
> > would avoid confusion with transaction try/catch block.

Yes, I moved it.


> On March 13, 2018, 3:35 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
> > Lines 365 (patched)
> > 
> >
> > This may cause weird situation where operation succeeds and is 
> > commitetd but throws an exception.

I moved it to its own try/catch to avoid such weird situation.


> On March 13, 2018, 3:35 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
> > Lines 366 (patched)
> > 
> >
> > It may be easier to read if you use 
> > 
> > if (listeners.isEmpty()) {
> >   return;
> > }

I'd like to keep the if (!listeners.isEmpty()) in case more code is added in 
the future that should be executed after the finally or this if statement and 
avoid the return if listeners is empty.


> On March 13, 2018, 3:35 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
> > Lines 372 (patched)
> > 
> >
> > Why are we creating dropTable/createTable notifications here? I 
> > understand that this mimics the code above, but it looks suspicious.

I left a comment there. The reason is if a rename happens between databases, 
then a drop/create table events between both databases should happen. Honestly, 
I don't think that is necessary and an ALTER event should be enough, but this 
code is already there and I don't know would affect other clients using 
notifications. Perhaps this is something to consider on the new HMS v2 design.


> On March 13, 2018, 3:35 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IHMSHandler.java
> > Lines 99 (patched)
> > 
> >
> > There is only a single consumer of this interface - it seems odd to 
> > have an interface which is used in some special case. Should all access to 
> > listeners be converted to the interface?

I don't know. All listeners are used in the HiveMetaStore class which contains 
the HMSHandler subclass. Being listeners and transctionalListeners a variable 
of HMSHandler, then the HiveMetaStore can call them directly without need the 
interface. The only consumer is the HiveAlterHandler which should be re-design 
in my opinion as all DDL operations are inside the HiveMetaStore and only the 
alter is in its own class. Could this also be considered in the HMS v2 design?


- Sergio


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65985/#review199063
---


On March 8, 2018, 4:11 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65985/
> ---
> 
> (Updated March 8, 2018, 4:11 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-18783
> https://issues.apache.org/jira/browse/HIVE-18783
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16164 introduced a mechanism to pass HMS notification events ID to the 
> post-commit listeners for all DDL operations, but it didn't add it to the 
> ALTER TABLE event. This patch in review adds the same behavior for ALTER 
> TABLE events.
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
>  e0e29652da94bbdaca515a17955d1409824c1742 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  89354a2d34249903a9ff13c4ed913a68de93057e 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  662de9a66767f27f31998f14c68f854e59993ab6 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IHMSHandler.java
>  e6de0013bc1be12b2772e2e97102ed476cf5 
> 
> 
> Diff: https://reviews.apache.org/r/65985/diff/1/
> 
> 
> Testing
> ---
> 
> All tests passed.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated March 15, 2018, 7:03 a.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

minor update to avoid zk exceptions during shutdown. also unregister listener 
is removed as latch close will automatically remove listeners and no need to do 
explicit removal.


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 907d064 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
46cfe56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/6/

Changes: https://reviews.apache.org/r/65498/diff/5-6/


Testing
---


Thanks,

Prasanth_J