[
https://issues.apache.org/jira/browse/PIG-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852511#action_12852511
]
Ashutosh Chauhan commented on PIG-1348:
---------------------------------------
To reproduce, cogroup page_views(from PigMix's dataset) with page_views on user
and this exception should occur. Apart from making InternalCachedBag more
robust, important thing to figure out here is to see where 90% of available
memory is getting used. Also, a related fix went in for this recently: PIG-1307
Might be related to that issue.
> InternalCachedBag running out of memory
> ---------------------------------------
>
> Key: PIG-1348
> URL: https://issues.apache.org/jira/browse/PIG-1348
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Richard Ding
>
> InternalCachedBag makes estimate of memory available to the VM by using
> Runtime.getRuntime().maxMemory(). It then uses 10%(by default, though
> configurable) of this memory and divides this memory into number of bags. It
> keeps track of the memory used by bags and then proactively spills if bags
> memory usage reach close to these limits. Given all this in theory when
> presented with data more then it can handle InternalCachedBag should not run
> out of memory. But in practice we find OOM happening.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
