[
https://issues.apache.org/jira/browse/TEZ-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277971#comment-14277971
]
Siddharth Seth commented on TEZ-1945:
-------------------------------------
bq. SPILLED_RECORDS can be > 0 as it is accounted for in finalMerge (mem +
disk). Will create a separate JIRA for post_merge_mem_limit.
So it's a merge not triggered by the fetchMemoryLimit but by the
postMergeMemoryLimit. Should be accounted for somehow in the counters; will
create a follow up jira.
> Remove 2 GB memlimit restriction in MergeManager
> ------------------------------------------------
>
> Key: TEZ-1945
> URL: https://issues.apache.org/jira/browse/TEZ-1945
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: TEZ-1945.1.patch
>
>
> In certain situations (data coming in larger chunks, but yet to complete),
> fetchers might wait in MerManager.waitForShuffleToMergeMemory() for memory to
> become available.
> Removing the 2 GB resitrction on MergeManager.memlimit would help in such
> situations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
