[
https://issues.apache.org/jira/browse/HIVE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15673923#comment-15673923
]
Barna Zsombor Klara commented on HIVE-15221:
--------------------------------------------
Hi [~ferhui],
thanks for the patch, but I'm not sure how I feel about an explicit call to
System.gc.
I understand your problem that a portion of the used heap at the point of the
exception could be garbage, but a call to System.gc will not guarantee a
garbage collection cycle, the jvm may ignore it. So the change would not
guarantee that we only abort if the heap is over utilised.
Sporadic and unexpected calls for a garbage collection would also ravage
attempts at fine tuning garbage collection cycles.
If a user is worried about false MemoryExhaustionExceptions then I would
recommend to try and fine tune the collector in use to increase the garbage
collection frequency of the jvm.
> Improvement for MapJoin checkMemoryStatus, adding gc before throwing Exception
> ------------------------------------------------------------------------------
>
> Key: HIVE-15221
> URL: https://issues.apache.org/jira/browse/HIVE-15221
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 2.1.0, 2.0.1
> Reporter: Fei Hui
> Assignee: Fei Hui
> Attachments: HIVE-15221.1.patch
>
>
> i see in the current master version
> percentage = (double) usedMemory / (double) maxHeapSize;
> if percentage > maxMemoryUsage, then throw MapJoinMemoryExhaustionException
> in my opinion, running is better than fail. after System.gc, ' if percentage
> > maxMemoryUsage, then throw MapJoinMemoryExhaustionException' maybe better
> And original checking way has a problem: 1) consuming much memory cause gc
> (e.g young gc), then check after adding row and pass. 2) consuming much
> memory does not cause gc, then check after adding rows but throw Exception
> sometimes 2) occurs, but it contians less rows than 1).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
