[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2017-08-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142227#comment-16142227 ] Apache Spark commented on SPARK-17321: -- User 'jerryshao' has created a pull request for this issue:

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2017-08-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142229#comment-16142229 ] Apache Spark commented on SPARK-17321: -- User 'jerryshao' has created a pull request for this issue:

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2017-08-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139502#comment-16139502 ] Saisai Shao commented on SPARK-17321: - 1. if NM recovery is enabled, then yarn will provide a

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2017-08-23 Thread lishuming (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139488#comment-16139488 ] lishuming commented on SPARK-17321: --- [~jerryshao] I agree with what you said, however there are some

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2017-08-23 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138347#comment-16138347 ] Thomas Graves commented on SPARK-17321: --- Yes that sounds good. It wouldn't hurt to verify the

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2017-08-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138320#comment-16138320 ] Saisai Shao commented on SPARK-17321: - We're facing the same issue. I think YARN shuffle service

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2017-07-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105164#comment-16105164 ] Thomas Graves commented on SPARK-17321: --- Can you clarify? as stated above you should not be using

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2017-07-27 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104373#comment-16104373 ] roncenzhao commented on SPARK-17321: Hi, I encounter this problem too. Any process about this bug?

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15487203#comment-15487203 ] Thomas Graves commented on SPARK-17321: --- yes that makes sense and as I stated I think the fix for

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-13 Thread Alexander Kasper (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15487166#comment-15487166 ] Alexander Kasper commented on SPARK-17321: -- No, we're not using NM recovery. What we observed is

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484748#comment-15484748 ] Thomas Graves commented on SPARK-17321: --- Not sure I follow this comment. So you are using NM

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-12 Thread Alexander Kasper (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483413#comment-15483413 ] Alexander Kasper commented on SPARK-17321: -- I guess then we encountered the 1% where the NM

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477061#comment-15477061 ] Thomas Graves commented on SPARK-17321: --- Note that if we want to fix this without yarn NM recovery

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477028#comment-15477028 ] Thomas Graves commented on SPARK-17321: --- it is possible but the point of the recovery dir is it

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-09 Thread Alexander Kasper (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476355#comment-15476355 ] Alexander Kasper commented on SPARK-17321: -- But NM recovery only kicks in if the NM goes down,

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474557#comment-15474557 ] Thomas Graves commented on SPARK-17321: --- so there are 2 possible things here: 1) You are using

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-08 Thread Alexander Kasper (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474342#comment-15474342 ] Alexander Kasper commented on SPARK-17321: -- We discovered the same issue. It seems the shuffle

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-08-31 Thread yunjiong zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15451942#comment-15451942 ] yunjiong zhao commented on SPARK-17321: --- If yarn.nodemanager.recovery.enabled = false & the first

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-08-31 Thread yunjiong zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15451925#comment-15451925 ] yunjiong zhao commented on SPARK-17321: --- Below logs shows that NodeManager already detect the disk

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-08-31 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15451742#comment-15451742 ] Sean Owen commented on SPARK-17321: --- Duplicate of https://issues.apache.org/jira/browse/SPARK-14963 ?

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-08-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450424#comment-15450424 ] Apache Spark commented on SPARK-17321: -- User 'zhaoyunjiong' has created a pull request for this