Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-14 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated March 15, 2018, 5:51 a.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

some changes to notification handling. added comments about how notifications 
are received and handled.


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 907d064 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
46cfe56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/5/

Changes: https://reviews.apache.org/r/65498/diff/4-5/


Testing
---


Thanks,

Prasanth_J

[jira] [Created] (HIVE-18963) JDBC: Provide an option to simplify beeline usage

2018-03-14 Thread Vaibhav Gumashta (JIRA) 
Vaibhav Gumashta created HIVE-18963:
---

 Summary: JDBC: Provide an option to simplify beeline usage
 Key: HIVE-18963
 URL: https://issues.apache.org/jira/browse/HIVE-18963
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Reporter: Vaibhav Gumashta


Currently, after opening Beeline CLI, the user needs to supply a connection 
string to use the HS2 instance and set up the jdbc driver. Since we plan to 
replace Hive CLI with Beeline in future (HIVE-10511), it will help the 
usability if the user can simply type {{beeline}} and get start the hive 
session. The jdbc url can be specified in a beeline-site.xml (which can contain 
other named jdbc urls as well, and they can be accessed by something like: 
{{beeline -c namedUrl}}. The use of beeline-site.xml can also be potentially 
expanded later if needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66075: HIVE-18962 add WM task state to Tez AM heartbeat

2018-03-14 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66075/#review199227
---




llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
Lines 3020 (patched)


bogus comment


- Sergey Shelukhin


On March 14, 2018, 11:45 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66075/
> ---
> 
> (Updated March 14, 2018, 11:45 p.m.)
> 
> 
> Review request for hive and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java
>  d97b1567ea 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  a2dca1b319 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java 
> b784360b6c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  75d8577d6a 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestAMReporter.java
>  bde354649a 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  a4bd4dfd88 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b805a 
>   
> llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
>  90b31e4fcd 
> 
> 
> Diff: https://reviews.apache.org/r/66075/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 66075: HIVE-18962 add WM task state to Tez AM heartbeat

2018-03-14 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66075/
---

Review request for hive and Jason Dere.


Repository: hive-git


Description
---

see jira


Diffs
-

  
llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java
 d97b1567ea 
  
llap-common/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
 a2dca1b319 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java 
b784360b6c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 75d8577d6a 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestAMReporter.java
 bde354649a 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 a4bd4dfd88 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b805a 
  
llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
 90b31e4fcd 


Diff: https://reviews.apache.org/r/66075/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-18962) add WM task state to Tez AM heartbeat

2018-03-14 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-18962:
---

 Summary: add WM task state to Tez AM heartbeat
 Key: HIVE-18962
 URL: https://issues.apache.org/jira/browse/HIVE-18962
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18961) Error in results cache when query has space in the name

2018-03-14 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-18961:
-

 Summary: Error in results cache when query has space in the name
 Key: HIVE-18961
 URL: https://issues.apache.org/jira/browse/HIVE-18961
 Project: Hive
  Issue Type: Sub-task
  Components: Query Planning
Reporter: Jason Dere
Assignee: Jason Dere


Found by [~gopalv]:

{noformat}
2018-03-14T05:08:32,551 ERROR [0c4b7a6c-ed37-428e-ac04-ca38716f211e 
HiveServer2-HttpHandler-Pool: Thread-8961]: parse.CalcitePlanner (:()) - 
Unexpected ]
org.apache.hadoop.hive.ql.parse.ParseException: line 1:100 missing EOF at 
'Count' near 'IMSI'
at 
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:220) 
~[hive-exec-3.0.0.3.0.0.2-132-jdere.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74) 
~[hive-exec-3.0.0.3.0.0.2-132-jdere.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringForCache(SemanticAnalyzer.java:14067)
 ~[hive-exec-3.0.0.3.0.0.2-132-jdere.jar:]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createLookupInfoForQuery(SemanticAnalyzer.java:14080)
 ~[hive-exec-3.0.0.3.0.0.2-132-jdere.ja]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11683)
 ~[hive-exec-3.0.0.3.0.0.2-132-jdere.jar:3.0.0-S]
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:304)
 ~[hive-exec-3.0.0.3.0.0.2-132-jdere.jar:3.0.0-SNAPSHO]
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:273)
 ~[hive-exec-3.0.0.3.0.0.2-132-jdere.jar:3.0.0-SNA]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:614) 
~[hive-exec-3.0.0.3.0.0.2-132-jdere.jar:3.0.0-SNAPSHOT]
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66068: [HIVE-16882] Improvements For Avro SerDe Package

2018-03-14 Thread David Mollitor 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66068/
---

(Updated March 14, 2018, 7:56 p.m.)


Review request for hive.


Repository: hive-git


Description
---

[HIVE-16882] Improvements For Avro SerDe Package


Diffs (updated)
-

  serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 
b7b3d12 
  
serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java 
e1e4d8b 
  serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1746a0f 
  serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java 2d52020 


Diff: https://reviews.apache.org/r/66068/diff/2/

Changes: https://reviews.apache.org/r/66068/diff/1-2/


Testing
---


Thanks,

David Mollitor

[jira] [Created] (HIVE-18960) Make Materialized view invalidation cache work with catalogs

2018-03-14 Thread Alan Gates (JIRA) 
Alan Gates created HIVE-18960:
-

 Summary: Make Materialized view invalidation cache work with 
catalogs
 Key: HIVE-18960
 URL: https://issues.apache.org/jira/browse/HIVE-18960
 Project: Hive
  Issue Type: Sub-task
  Components: Materialized views, Metastore
Affects Versions: 3.0.0
Reporter: Alan Gates
Assignee: Alan Gates


MaterializationsInvalidationCache needs to be made catalog aware.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66068: [HIVE-16882] Improvements For Avro SerDe Package

2018-03-14 Thread Vihang Karajgaonkar via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66068/#review199185
---


Fix it, then Ship it!




LGTM


serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
Line 29 (original), 29 (patched)


Didn't the author really mean memoizes 
https://en.wikipedia.org/wiki/Memoization


- Vihang Karajgaonkar


On March 14, 2018, 4:10 p.m., David Mollitor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66068/
> ---
> 
> (Updated March 14, 2018, 4:10 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> [HIVE-16882] Improvements For Avro SerDe Package
> 
> 
> Diffs
> -
> 
>   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 
> b7b3d12 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java
>  e1e4d8b 
>   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1746a0f 
>   serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java 
> 2d52020 
> 
> 
> Diff: https://reviews.apache.org/r/66068/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> David Mollitor
> 
>

[jira] [Created] (HIVE-18959) Avoid creating extra pool of threads within LLAP

2018-03-14 Thread slim bouguerra (JIRA) 
slim bouguerra created HIVE-18959:
-

 Summary: Avoid creating extra pool of threads within LLAP
 Key: HIVE-18959
 URL: https://issues.apache.org/jira/browse/HIVE-18959
 Project: Hive
  Issue Type: Task
  Components: Druid integration
 Environment: Kerberos Cluster
Reporter: slim bouguerra
Assignee: slim bouguerra
 Fix For: 3.0.0


The current Druid-Kerberos-Http client is using an external single threaded 
pool to handle retry auth calls (eg when a cookie expire or other transient 
auth issues). 

First, this is not buying us anything since all the Druid Task is executed as 
one synchronous task.

Second, this can cause a major issue if an exception occurs that leads to 
shutting down the LLAP main thread.

 Thus to fix this we should avoid using an external thread pool and handle 
retrying in a synchronous way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18958) Fix Spark config warnings

2018-03-14 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-18958:
---

 Summary: Fix Spark config warnings
 Key: HIVE-18958
 URL: https://issues.apache.org/jira/browse/HIVE-18958
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar


Getting a few configuration warnings in the logs that we should fix:

{code}
2018-03-14T10:06:19,164  WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
spark.SparkConf: The configuration key 'spark.yarn.driver.memoryOverhead' has 
been deprecated as of Spark 2.3 and may be removed in the future. Please use 
the new key 'spark.driver.memoryOverhead' instead.
2018-03-14T10:06:19,165  WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
spark.SparkConf: The configuration key spark.akka.logLifecycleEvents is not 
supported any more because Spark doesn't use Akka since 2.0
2018-03-14T10:06:19,165  WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
spark.SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has 
been deprecated as of Spark 2.3 and may be removed in the future. Please use 
the new key 'spark.executor.memoryOverhead' instead.
2018-03-14T10:06:20,351  INFO 
[RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
client.SparkClientImpl: Warning: Ignoring non-spark config property: 
hive.spark.client.server.connect.timeout=9
2018-03-14T10:06:20,351  INFO 
[RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
client.SparkClientImpl: Warning: Ignoring non-spark config property: 
hive.spark.client.rpc.threads=8
2018-03-14T10:06:20,351  INFO 
[RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
client.SparkClientImpl: Warning: Ignoring non-spark config property: 
hive.spark.client.connect.timeout=3
2018-03-14T10:06:20,351  INFO 
[RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
client.SparkClientImpl: Warning: Ignoring non-spark config property: 
hive.spark.client.secret.bits=256
2018-03-14T10:06:20,351  INFO 
[RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
client.SparkClientImpl: Warning: Ignoring non-spark config property: 
hive.spark.client.rpc.max.size=52428800
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18957) Upgrade Calcite version to 1.16.0

2018-03-14 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-18957:
--

 Summary: Upgrade Calcite version to 1.16.0
 Key: HIVE-18957
 URL: https://issues.apache.org/jira/browse/HIVE-18957
 Project: Hive
  Issue Type: Task
  Components: CBO
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 66069: HIVE-18926 Imporve operator-tree matching

2018-03-14 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66069/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-18926
https://issues.apache.org/jira/browse/HIVE-18926


Repository: hive-git


Description
---

* match operators based on opsigs
* always calc operator sigs


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java 
90b2fd3dad8ee9ec14213d286d05ef21f53d45a0 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java 
c6d1a6aaca27bc487654b41612073fb2e20c2023 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/Signature.java 
c228a8e4f52f31deb5781a5b875ca5c521722ec7 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/SignatureUtils.java 
2269322b69ab0d003d7ebc83a6784267477e5551 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 22b052cd07928fb0011787424b5af7b75dcc6bf4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 
5a81add7066c4d886bf30e2c50c71194763174db 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 
ea22131c56aede0b523d861808214520f031ea2f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 
85a4683491d6f22bdea1c40b042fedcc24cfbf31 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 
91ea159d524771b4492b3ff3bb402deb37e106d4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 
f2955af25ee09fa0521da0cb601181ceb9d18417 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java 
36d7e589a844359e1e91e5150541298f8cac837f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java 
424dd7956b9e50bc7729289a889452a07b59351a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java 
21a06781533b684269f88daa7f1c9522383b1971 
  
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java 
6f340b8450a95569156b3b7eddf28fdd53fc9065 
  ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java 
93031712dc6fb60a6d618c8754e50def489a12af 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java
 8c899e7fef1a1ebe06fdd9269f3c922f8baab84d 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java 
cfb0ca38b2289c29ff93bc8cfd6868de17ed4fad 
  ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out 
9b58ce0e5d99a704c4a6305487e937b2ea727624 


Diff: https://reviews.apache.org/r/66069/diff/1/


Testing
---


Thanks,

Zoltan Haindrich

[jira] [Created] (HIVE-18956) AvroSerDe Race Condition

2018-03-14 Thread BELUGA BEHR (JIRA) 
BELUGA BEHR created HIVE-18956:
--

 Summary: AvroSerDe Race Condition
 Key: HIVE-18956
 URL: https://issues.apache.org/jira/browse/HIVE-18956
 Project: Hive
  Issue Type: Bug
  Components: Serializers/Deserializers
Affects Versions: 2.3.2, 3.0.0
Reporter: BELUGA BEHR


{code}
  @Override
  public Writable serialize(Object o, ObjectInspector objectInspector) throws 
SerDeException {
if(badSchema) {
  throw new BadSchemaException();
}
return getSerializer().serialize(o, objectInspector, columnNames, 
columnTypes, schema);
  }

  @Override
  public Object deserialize(Writable writable) throws SerDeException {
if(badSchema) {
  throw new BadSchemaException();
}
return getDeserializer().deserialize(columnNames, columnTypes, writable, 
schema);
  }

...

  private AvroDeserializer getDeserializer() {
if(avroDeserializer == null) {
  avroDeserializer = new AvroDeserializer();
}

return avroDeserializer;
  }

  private AvroSerializer getSerializer() {
if(avroSerializer == null) {
  avroSerializer = new AvroSerializer();
}

return avroSerializer;
  }
{code}

{{getDeserializer}} and {{getSerializer}} methods are not thread safe, so 
neither are {{deserialize}} and {{serialize}} methods.  It probably didn't 
matter with MapReduce, but now that we have Spark/Tez, it may be an issue.

You could visualize a scenario where three threads all enter {{getSerializer}} 
and all see that {{avroSerializer}} is _null_ and create three instances, then 
they would fight to assign the new object to the {{avroSerializer}} variable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 66068: [HIVE-16882] Improvements For Avro SerDe Package

2018-03-14 Thread David Mollitor 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66068/
---

Review request for hive.


Repository: hive-git


Description
---

[HIVE-16882] Improvements For Avro SerDe Package


Diffs
-

  serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 
b7b3d12 
  
serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java 
e1e4d8b 
  serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1746a0f 
  serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java 2d52020 


Diff: https://reviews.apache.org/r/66068/diff/1/


Testing
---


Thanks,

David Mollitor

[jira] [Created] (HIVE-18955) HoS: Unable to create Channel from class NioServerSocketChannel

2018-03-14 Thread Rui Li (JIRA) 
Rui Li created HIVE-18955:
-

 Summary: HoS: Unable to create Channel from class 
NioServerSocketChannel
 Key: HIVE-18955
 URL: https://issues.apache.org/jira/browse/HIVE-18955
 Project: Hive
  Issue Type: Bug
  Components: Spark
Reporter: Rui Li


Hit the issue when trying launch spark job. Stack trace:
{noformat}
Caused by: java.lang.NoSuchMethodError: 
io.netty.channel.DefaultChannelId.newInstance()Lio/netty/channel/DefaultChannelId;
at io.netty.channel.AbstractChannel.newId(AbstractChannel.java:111) 
~[netty-all-4.1.17.Final.jar:4.1.17.Final]
at io.netty.channel.AbstractChannel.(AbstractChannel.java:83) 
~[netty-all-4.1.17.Final.jar:4.1.17.Final]
at 
io.netty.channel.nio.AbstractNioChannel.(AbstractNioChannel.java:84) 
~[netty-all-4.1.17.Final.jar:4.1.17.Final]
at 
io.netty.channel.nio.AbstractNioMessageChannel.(AbstractNioMessageChannel.java:42)
 ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
at 
io.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:86)
 ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
at 
io.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:72)
 ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method) ~[?:1.8.0_151]
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 ~[?:1.8.0_151]
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 ~[?:1.8.0_151]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) 
~[?:1.8.0_151]
at 
io.netty.channel.ReflectiveChannelFactory.newChannel(ReflectiveChannelFactory.java:38)
 ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
... 32 more
{noformat}

It seems we have conflicts versions of class 
{{io.netty.channel.DefaultChannelId}} from async-http-client.jar and 
netty-all.jar



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-14 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated March 14, 2018, 8:24 a.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

Addressed review comments.


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 30c24d9 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
46cfe56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/4/

Changes: https://reviews.apache.org/r/65498/diff/3-4/


Testing
---


Thanks,

Prasanth_J

[jira] [Created] (HIVE-18954) insert hive storage handler table from a parquet table error

2018-03-14 Thread JIRA 
翟玉勇 created HIVE-18954:
--

 Summary: insert hive storage handler table from a parquet table 
error
 Key: HIVE-18954
 URL: https://issues.apache.org/jira/browse/HIVE-18954
 Project: Hive
  Issue Type: Bug
  Components: StorageHandler
Affects Versions: 2.1.0
Reporter: 翟玉勇


{code}


drop table if exists temp.temp_dw_sia_jc_kpa7_brand_factor_list_week;
create table temp.temp_dw_sia_jc_kpa7_brand_factor_list_week( 
 `brand_id` bigint,
 `brand_name` string,
 `reason_key` string,
 `reason_value` double ,
 `reason_imp_order_num` double ,
 `data_type` string ,
 `biz_date` string 
)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES ( 
'es.resource'='index/type',
 'es.nodes'='',
 'es.port'='9200',
 'es.nodes.wan.only'='true',
 'es.index.translog.flush_threshold_size'='3gb',
 'es.index.translog.interval'='60s',
 'es.index.translog,sync_interval'='60s',
 'index.translog.durability'='async'
 );

insert overwrite table temp.temp_dw_sia_jc_kpa7_brand_factor_list
select
now_brand_id as brand_id,
now_brand_name as brand_name,
factor as reason_key,
value as reason_value,
value_order as reason_imp_order_num,
model as data_type,
biz_date
from dw.dw_jc_kpa7_brand_factor_list
where dt = '123'
and value is not null and value_order is not null and model is not null;

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)