[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-12 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/15037 Merging to master and branch-2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-12 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/15037 This LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15037 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15037 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65178/ Test PASSed. ---

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15037 **[Test build #65178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65178/consoleFull)** for PR 15037 at commit

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15037 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65175/ Test PASSed. ---

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15037 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15037 **[Test build #65175 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65175/consoleFull)** for PR 15037 at commit

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15037 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15037 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65172/ Test PASSed. ---

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15037 **[Test build #65172 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65172/consoleFull)** for PR 15037 at commit

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15037 **[Test build #65178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65178/consoleFull)** for PR 15037 at commit

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/15037 It looks like this location refresh logic was added in #11241 in order to resolve some issues which could occur on dynamic allocation clusters with high churn. While I definitely think that we

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/15037 Ah, I see the issue: the problem is that we might end up repeatedly refreshing in case the number of returned locations is is much smaller than the max fetch attempt count. I think that

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/15037 Although the location refresh will reset `runningFailureCount`, the loop here should still terminate because it's based off of `totalFailureCount`, which is monotonically increasing. I agree that

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15037 **[Test build #65175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65175/consoleFull)** for PR 15037 at commit

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread srinathshankar
Github user srinathshankar commented on the issue: https://github.com/apache/spark/pull/15037 How does the refreshing of locations here : https://github.com/apache/spark/pull/15037/files?diff=unified#diff-2b643ea78c1add0381754b1f47eec132R581 affect maxFetchFailures. Seems possible

[GitHub] spark issue #15037: [SPARK-17485] Prevent failed remote reads of cached bloc...

2016-09-09 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15037 **[Test build #65172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65172/consoleFull)** for PR 15037 at commit