[
https://issues.apache.org/jira/browse/PIG-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohini Palaniswamy updated PIG-4375:
------------------------------------
Attachment: PIG-4375-1.patch
Also fixed another case of OOM with PigCombiner apart from the
ObjectRegistryImpl leak. The reporter has reference to the context which has
sort buffer with default size of 100MB and usually set to higher value in
production. As combiner is invoked multiple times during merge, keeping strong
reference to ProgressableReporter in the PhysicalOperator's ThreadLocal can
quickly cause OOM. Set to null freeing it up for GC.
[~cheolsoo],
You might want to try this patch with the join + group by job you had.
There is a possibility that this memory leak could have caused more slowness
and retries on failure due to OOM than having less number of intermediate
reducers.
> ObjectCache should use ProcessorContext.getObjectRegistry()
> -----------------------------------------------------------
>
> Key: PIG-4375
> URL: https://issues.apache.org/jira/browse/PIG-4375
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.14.0
> Reporter: Rohini Palaniswamy
> Fix For: 0.14.1, 0.15.0
>
> Attachments: PIG-4375-1.patch
>
>
> We have been instantiating our ObjectRegistryImpl which is wrong and will
> leak memory. It was a copy from hive code in the beginning and I think lot
> has changed in Tez after that and we have missed it. We also need to expose
> the ObjectRegistry to users so that they can cache their own stuff for
> IndexedLoadFunc implementations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
