[jira] [Commented] (PIG-3783) We can predict when small local jobs will cause an OOM and change io.sort.mb in that case
[ https://issues.apache.org/jira/browse/PIG-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13929917#comment-13929917 ] Ravi Prakash commented on PIG-3783: --- Thanks Aniket! I'll defer to your better judgement and close this JIRA as a duplicate of PIG-3731. > We can predict when small local jobs will cause an OOM and change io.sort.mb > in that case > - > > Key: PIG-3783 > URL: https://issues.apache.org/jira/browse/PIG-3783 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Ravi Prakash > Labels: newbie > Attachments: PIG-3783.branch-0.12.patch > > > Seems like a lot of new users run into this problem: > http://stackoverflow.com/questions/10165648/apache-pig-outofmemory-exception-with-simple-group-by-in-local-mode > http://sumedha.blogspot.in/2012/01/solving-apache-pig-javalangoutofmemorye.html > http://stackoverflow.com/questions/16499432/pig-local-mode-group-or-join-java-lang-outofmemoryerror-java-heap-space -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3783) We can predict when small local jobs will cause an OOM and change io.sort.mb in that case
[ https://issues.apache.org/jira/browse/PIG-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920629#comment-13920629 ] Aniket Mokashi commented on PIG-3783: - How about solving this with a separate namespace for local mode configuration so that user can choose what suits their job the best- https://issues.apache.org/jira/browse/PIG-3731? > We can predict when small local jobs will cause an OOM and change io.sort.mb > in that case > - > > Key: PIG-3783 > URL: https://issues.apache.org/jira/browse/PIG-3783 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Ravi Prakash > Labels: newbie > Attachments: PIG-3783.branch-0.12.patch > > > Seems like a lot of new users run into this problem: > http://stackoverflow.com/questions/10165648/apache-pig-outofmemory-exception-with-simple-group-by-in-local-mode > http://sumedha.blogspot.in/2012/01/solving-apache-pig-javalangoutofmemorye.html > http://stackoverflow.com/questions/16499432/pig-local-mode-group-or-join-java-lang-outofmemoryerror-java-heap-space -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3783) We can predict when small local jobs will cause an OOM and change io.sort.mb in that case
[ https://issues.apache.org/jira/browse/PIG-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920627#comment-13920627 ] Aniket Mokashi commented on PIG-3783: - [~raviprak], AFAIK, io.sort.mb also affects mappers. We should take that into consideration. Also, pig can run more than one job in parallel- this does not help it. Third, pig can also convert job to local mode at runtime, so we need to find a fix for that one too. > We can predict when small local jobs will cause an OOM and change io.sort.mb > in that case > - > > Key: PIG-3783 > URL: https://issues.apache.org/jira/browse/PIG-3783 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Ravi Prakash > Labels: newbie > Attachments: PIG-3783.branch-0.12.patch > > > Seems like a lot of new users run into this problem: > http://stackoverflow.com/questions/10165648/apache-pig-outofmemory-exception-with-simple-group-by-in-local-mode > http://sumedha.blogspot.in/2012/01/solving-apache-pig-javalangoutofmemorye.html > http://stackoverflow.com/questions/16499432/pig-local-mode-group-or-join-java-lang-outofmemoryerror-java-heap-space -- This message was sent by Atlassian JIRA (v6.2#6252)
