[
https://issues.apache.org/jira/browse/PIG-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317274#comment-15317274
]
Rohini Palaniswamy edited comment on PIG-4922 at 6/6/16 9:33 PM:
-----------------------------------------------------------------
Moved the sorting of the list that does getMemorySize outside
synchronized(spillables) block as we have already made the copy of the list to
spillablesSR.
getMemorySize has always been in synchronized(spillables). But seeing the issue
for the first time now. Most likely because in Tez, the input and output take
more memory than mapreduce and making it hit the spill code.
was (Author: rohini):
Moved the sorting of the list that does getMemorySize outside
synchronized(spillables) block as we have already made the copy of the list to
spillablesSR.
> Deadlock between SpillableMemoryManager and
> InternalSortedBag$SortedDataBagIterator
> -----------------------------------------------------------------------------------
>
> Key: PIG-4922
> URL: https://issues.apache.org/jira/browse/PIG-4922
> Project: Pig
> Issue Type: Bug
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.17.0, 0.16.1
>
> Attachments: PIG-4922-1.patch
>
>
> This one ran into a deadlock, when the data was really huge and
> InternalSortedBag was reading spilled data from disk.
> {code}
> grpd = FOREACH (GROUP data BY $0){
> sorted = ORDER data BY timestamp DESC;
> latest = LIMIT sorted 1;
> GENERATE latest;
> };
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
