subject:"\"\\\\\\\[jira\\\\\\\] \\\\\\\[Updated\\\\\\\] \\\\\\\(HIVE\\\\\\\-15565\\\\\\\) LLAP\\\\\\\: GroupByOperator flushes hash table too frequently\""

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

2017-01-22 Thread Rajesh Balamohan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-15565:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks [~prasanth_j], [~sershe].  Committed to master. 

> LLAP: GroupByOperator flushes hash table too frequently
> ---
>
> Key: HIVE-15565
> URL: https://issues.apache.org/jira/browse/HIVE-15565
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15565.1.patch, HIVE-15565.2.patch
>
>
> {{GroupByOperator::isTez}} would be true in LLAP mode. Current memory 
> computations can go wrong with {{isTez}} checks in {{GroupByOperator}}. For 
> e.g, in a LLAP instance with Xmx128G and 12 executors, it would start 
> flushing hash table for every record once it reaches around 42GB 
> (hive.tez.container.size=7100, hive.map.aggr.hash.percentmemory=0.5).
> {noformat}
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_04_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size 
> = 0
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_12_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size 
> = 0
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_04_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 
> 1
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_12_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 
> 1
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

2017-01-18 Thread Rajesh Balamohan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-15565:

Status: Patch Available  (was: Reopened)

> LLAP: GroupByOperator flushes hash table too frequently
> ---
>
> Key: HIVE-15565
> URL: https://issues.apache.org/jira/browse/HIVE-15565
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15565.1.patch, HIVE-15565.2.patch
>
>
> {{GroupByOperator::isTez}} would be true in LLAP mode. Current memory 
> computations can go wrong with {{isTez}} checks in {{GroupByOperator}}. For 
> e.g, in a LLAP instance with Xmx128G and 12 executors, it would start 
> flushing hash table for every record once it reaches around 42GB 
> (hive.tez.container.size=7100, hive.map.aggr.hash.percentmemory=0.5).
> {noformat}
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_04_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size 
> = 0
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_12_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size 
> = 0
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_04_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 
> 1
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_12_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 
> 1
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

2017-01-18 Thread Rajesh Balamohan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-15565:

Attachment: HIVE-15565.2.patch

> LLAP: GroupByOperator flushes hash table too frequently
> ---
>
> Key: HIVE-15565
> URL: https://issues.apache.org/jira/browse/HIVE-15565
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15565.1.patch, HIVE-15565.2.patch
>
>
> {{GroupByOperator::isTez}} would be true in LLAP mode. Current memory 
> computations can go wrong with {{isTez}} checks in {{GroupByOperator}}. For 
> e.g, in a LLAP instance with Xmx128G and 12 executors, it would start 
> flushing hash table for every record once it reaches around 42GB 
> (hive.tez.container.size=7100, hive.map.aggr.hash.percentmemory=0.5).
> {noformat}
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_04_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size 
> = 0
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_12_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size 
> = 0
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_04_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 
> 1
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_12_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 
> 1
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

2017-01-11 Thread Rajesh Balamohan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-15565:

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Thanks [~sershe], [~prasanth_j]. Committed to master.

> LLAP: GroupByOperator flushes hash table too frequently
> ---
>
> Key: HIVE-15565
> URL: https://issues.apache.org/jira/browse/HIVE-15565
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15565.1.patch
>
>
> {{GroupByOperator::isTez}} would be true in LLAP mode. Current memory 
> computations can go wrong with {{isTez}} checks in {{GroupByOperator}}. For 
> e.g, in a LLAP instance with Xmx128G and 12 executors, it would start 
> flushing hash table for every record once it reaches around 42GB 
> (hive.tez.container.size=7100, hive.map.aggr.hash.percentmemory=0.5).
> {noformat}
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_04_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size 
> = 0
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_12_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size 
> = 0
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_04_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 
> 1
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_12_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 
> 1
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

2017-01-08 Thread Rajesh Balamohan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-15565:

Description: 
{{GroupByOperator::isTez}} would be true in LLAP mode. Current memory 
computations can go wrong with {{isTez}} checks in {{GroupByOperator}}. For 
e.g, in a LLAP instance with Xmx128G and 12 executors, it would start flushing 
hash table for every record once it reaches around 42GB 
(hive.tez.container.size=7100, hive.map.aggr.hash.percentmemory=0.5).

{noformat}
2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
(1480722417364_1922_7_03_04_1)] 
org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size = 0
2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
(1480722417364_1922_7_03_12_1)] 
org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size = 0
2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
(1480722417364_1922_7_03_04_1)] 
org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 1
2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
(1480722417364_1922_7_03_12_1)] 
org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 1
{noformat}

  was:
{{GroupByOperator::isTez}} would be true in LLAP mode. Current memory 
computations can go wrong with {{isTez}} checks in {{GroupByOperator}}. For 
e.g, in a LLAP instance with Xmx128G and 12 executors, it would start flushing 
hash table for every record once it reaches around 42GB 
(hive.tez.container.size=7100, hive.map.aggr.hash.percentmemory=0.5).



> LLAP: GroupByOperator flushes hash table too frequently
> ---
>
> Key: HIVE-15565
> URL: https://issues.apache.org/jira/browse/HIVE-15565
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15565.1.patch
>
>
> {{GroupByOperator::isTez}} would be true in LLAP mode. Current memory 
> computations can go wrong with {{isTez}} checks in {{GroupByOperator}}. For 
> e.g, in a LLAP instance with Xmx128G and 12 executors, it would start 
> flushing hash table for every record once it reaches around 42GB 
> (hive.tez.container.size=7100, hive.map.aggr.hash.percentmemory=0.5).
> {noformat}
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_04_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size 
> = 0
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_12_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size 
> = 0
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_04_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 
> 1
> 2017-01-08T23:40:21,339 INFO  [TezTaskRunner 
> (1480722417364_1922_7_03_12_1)] 
> org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 
> 1
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

2017-01-08 Thread Rajesh Balamohan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-15565:

Status: Patch Available  (was: Open)

> LLAP: GroupByOperator flushes hash table too frequently
> ---
>
> Key: HIVE-15565
> URL: https://issues.apache.org/jira/browse/HIVE-15565
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15565.1.patch
>
>
> {{GroupByOperator::isTez}} would be true in LLAP mode. Current memory 
> computations can go wrong with {{isTez}} checks in {{GroupByOperator}}. For 
> e.g, in a LLAP instance with Xmx128G and 12 executors, it would start 
> flushing hash table for every record once it reaches around 42GB 
> (hive.tez.container.size=7100, hive.map.aggr.hash.percentmemory=0.5).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

2017-01-08 Thread Rajesh Balamohan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-15565:

Attachment: HIVE-15565.1.patch

> LLAP: GroupByOperator flushes hash table too frequently
> ---
>
> Key: HIVE-15565
> URL: https://issues.apache.org/jira/browse/HIVE-15565
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15565.1.patch
>
>
> {{GroupByOperator::isTez}} would be true in LLAP mode. Current memory 
> computations can go wrong with {{isTez}} checks in {{GroupByOperator}}. For 
> e.g, in a LLAP instance with Xmx128G and 12 executors, it would start 
> flushing hash table for every record once it reaches around 42GB 
> (hive.tez.container.size=7100, hive.map.aggr.hash.percentmemory=0.5).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

[jira] [Updated] (HIVE-15565) LLAP: GroupByOperator flushes hash table too frequently

7 matches

Site Navigation

Mail list logo

Footer information