[
https://issues.apache.org/jira/browse/HIVE-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-15565:
------------------------------------
Description:
{{GroupByOperator::isTez}} would be true in LLAP mode. Current memory
computations can go wrong with {{isTez}} checks in {{GroupByOperator}}. For
e.g, in a LLAP instance with Xmx128G and 12 executors, it would start flushing
hash table for every record once it reaches around 42GB
(hive.tez.container.size=7100, hive.map.aggr.hash.percentmemory=0.5).
{noformat}
2017-01-08T23:40:21,339 INFO [TezTaskRunner
(1480722417364_1922_7_03_000004_1)]
org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size = 0
2017-01-08T23:40:21,339 INFO [TezTaskRunner
(1480722417364_1922_7_03_000012_1)]
org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size = 0
2017-01-08T23:40:21,339 INFO [TezTaskRunner
(1480722417364_1922_7_03_000004_1)]
org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 1
2017-01-08T23:40:21,339 INFO [TezTaskRunner
(1480722417364_1922_7_03_000012_1)]
org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 1
{noformat}
was:
{{GroupByOperator::isTez}} would be true in LLAP mode. Current memory
computations can go wrong with {{isTez}} checks in {{GroupByOperator}}. For
e.g, in a LLAP instance with Xmx128G and 12 executors, it would start flushing
hash table for every record once it reaches around 42GB
(hive.tez.container.size=7100, hive.map.aggr.hash.percentmemory=0.5).
> LLAP: GroupByOperator flushes hash table too frequently
> -------------------------------------------------------
>
> Key: HIVE-15565
> URL: https://issues.apache.org/jira/browse/HIVE-15565
> Project: Hive
> Issue Type: Bug
> Components: llap
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
> Attachments: HIVE-15565.1.patch
>
>
> {{GroupByOperator::isTez}} would be true in LLAP mode. Current memory
> computations can go wrong with {{isTez}} checks in {{GroupByOperator}}. For
> e.g, in a LLAP instance with Xmx128G and 12 executors, it would start
> flushing hash table for every record once it reaches around 42GB
> (hive.tez.container.size=7100, hive.map.aggr.hash.percentmemory=0.5).
> {noformat}
> 2017-01-08T23:40:21,339 INFO [TezTaskRunner
> (1480722417364_1922_7_03_000004_1)]
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size
> = 0
> 2017-01-08T23:40:21,339 INFO [TezTaskRunner
> (1480722417364_1922_7_03_000012_1)]
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size
> = 0
> 2017-01-08T23:40:21,339 INFO [TezTaskRunner
> (1480722417364_1922_7_03_000004_1)]
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table =
> 1
> 2017-01-08T23:40:21,339 INFO [TezTaskRunner
> (1480722417364_1922_7_03_000012_1)]
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table =
> 1
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
