[GitHub] spark pull request: [SPARK-4946] [CORE] Using AkkaUtils.askWithRep...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3785 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4946] [CORE] Using AkkaUtils.askWithRep...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3785#issuecomment-68293600 Alright, I'm going to merge this into `master` (1.3.0). Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4946] [CORE] Using AkkaUtils.askWithRep...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3785#issuecomment-68292416 On the surface, this seems like an okay change. I wonder whether this retry logic could have unexpected consequences. Let me try to reason it out: - `askTracker` is only called with `GetMapOutputStatuses`. - In the master actor, it calls `getSerializedMapOutputStatuses`. This method never throws exceptions: if a shuffle is missing, then it just stores an empty array and serializes it. - It's possible that the serialized map statuses could exceed the Akka frame size (although extremely unlikely and perhaps impossible with the new output status compression techniques). In this case, though, the master would throw an exception and fail to send a reply back to the asker. In this case, with this patch we'd end up performing a bunch of retries for an operation that will ultimately fail, so we'll take longer to detect a failure. In the common cases, though, this seems fine, even if the map output statuses are missing (since it won't introduce a bunch of futile retries). Therefore, I think we should pull this in; I don't know if this fixes an actual bug, but it seems like it could make things more robust. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4946] [CORE] Using AkkaUtils.askWithRep...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3785#issuecomment-68034001 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24765/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4946] [CORE] Using AkkaUtils.askWithRep...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3785#issuecomment-68033992 [Test build #24765 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24765/consoleFull) for PR 3785 at commit [`9ca6541`](https://github.com/apache/spark/commit/9ca65418c4d859b7ded77697e81d09f33a43b9a4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4946] [CORE] Using AkkaUtils.askWithRep...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3785#issuecomment-68030034 [Test build #24765 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24765/consoleFull) for PR 3785 at commit [`9ca6541`](https://github.com/apache/spark/commit/9ca65418c4d859b7ded77697e81d09f33a43b9a4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4946] [CORE] Using AkkaUtils.askWithRep...
GitHub user YanTangZhai opened a pull request: https://github.com/apache/spark/pull/3785 [SPARK-4946] [CORE] Using AkkaUtils.askWithReply in MapOutputTracker.askTracker to reduce the chance of the communicating problem Using AkkaUtils.askWithReply in MapOutputTracker.askTracker to reduce the chance of the communicating problem You can merge this pull request into a Git repository by running: $ git pull https://github.com/YanTangZhai/spark SPARK-4946 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3785.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3785 commit cdef539abc5d2d42d4661373939bdd52ca8ee8e6 Author: YanTangZhai Date: 2014-08-06T13:07:08Z Merge pull request #1 from apache/master update commit cbcba66ad77b96720e58f9d893e87ae5f13b2a95 Author: YanTangZhai Date: 2014-08-20T13:14:08Z Merge pull request #3 from apache/master Update commit 8a0010691b669495b4c327cf83124cabb7da1405 Author: YanTangZhai Date: 2014-09-12T06:54:58Z Merge pull request #6 from apache/master Update commit 03b62b043ab7fd39300677df61c3d93bb9beb9e3 Author: YanTangZhai Date: 2014-09-16T12:03:22Z Merge pull request #7 from apache/master Update commit 76d40277d51f709247df1d3734093bf2c047737d Author: YanTangZhai Date: 2014-10-20T12:52:22Z Merge pull request #8 from apache/master update commit d26d98248a1a4d0eb15336726b6f44e05dd7a05a Author: YanTangZhai Date: 2014-11-04T09:00:31Z Merge pull request #9 from apache/master Update commit e249846d9b7967ae52ec3df0fb09e42ffd911a8a Author: YanTangZhai Date: 2014-11-11T03:18:24Z Merge pull request #10 from apache/master Update commit 6e643f81555d75ec8ef3eb57bf5ecb6520485588 Author: YanTangZhai Date: 2014-12-01T11:23:56Z Merge pull request #11 from apache/master Update commit 718afebe364bd54ac33be425e24183eb1c76b5d3 Author: YanTangZhai Date: 2014-12-05T11:08:31Z Merge pull request #12 from apache/master update commit e4c2c0a18bdc78cc17823cbc2adf3926944e6bc5 Author: YanTangZhai Date: 2014-12-24T03:15:22Z Merge pull request #15 from apache/master update commit 9ca65418c4d859b7ded77697e81d09f33a43b9a4 Author: yantangzhai Date: 2014-12-24T06:17:32Z [SPARK-4946] [CORE] Using AkkaUtils.askWithReply in MapOutputTracker.askTracker to reduce the chance of the communicating problem --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
