[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user squito closed the pull request at: https://github.com/apache/spark/pull/6648 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155564301 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45533/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155564296 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155564092 **[Test build #45533 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45533/consoleFull)** for PR 6648 at commit [`fbd129b`](https://github.com/apache/spark/commit/fbd129b37fbb821053088bb5b704611262029dd0). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)`\n * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)`\n * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155516219 **[Test build #45533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45533/consoleFull)** for PR 6648 at commit [`fbd129b`](https://github.com/apache/spark/commit/fbd129b37fbb821053088bb5b704611262029dd0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155515272 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155515244 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155514893 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155497993 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155497998 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45528/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155497795 **[Test build #45528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45528/consoleFull)** for PR 6648 at commit [`fbd129b`](https://github.com/apache/spark/commit/fbd129b37fbb821053088bb5b704611262029dd0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)`\n * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)`\n * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155441284 **[Test build #45528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45528/consoleFull)** for PR 6648 at commit [`fbd129b`](https://github.com/apache/spark/commit/fbd129b37fbb821053088bb5b704611262029dd0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155439233 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155439196 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155439140 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155267487 **[Test build #45484 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45484/consoleFull)** for PR 6648 at commit [`fbd129b`](https://github.com/apache/spark/commit/fbd129b37fbb821053088bb5b704611262029dd0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155267494 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45484/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155267429 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155267492 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155267449 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155267074 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155264424 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45437/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155264423 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155264348 **[Test build #45437 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45437/consoleFull)** for PR 6648 at commit [`fbd129b`](https://github.com/apache/spark/commit/fbd129b37fbb821053088bb5b704611262029dd0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)`\n * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)`\n * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155231380 **[Test build #45437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45437/consoleFull)** for PR 6648 at commit [`fbd129b`](https://github.com/apache/spark/commit/fbd129b37fbb821053088bb5b704611262029dd0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155228605 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155228626 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155228271 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155192069 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155191969 **[Test build #45389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45389/consoleFull)** for PR 6648 at commit [`fbd129b`](https://github.com/apache/spark/commit/fbd129b37fbb821053088bb5b704611262029dd0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class MasterWebUI(`\n * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)`\n * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)`\n * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)`\n * `public class JavaAFTSurvivalRegressionExample `\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155149920 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155150330 **[Test build #45389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45389/consoleFull)** for PR 6648 at commit [`fbd129b`](https://github.com/apache/spark/commit/fbd129b37fbb821053088bb5b704611262029dd0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-155149944 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-147553320 BTW, with that design, I also wouldn't even implement the delete message in the first patch, unless we've actually seen block corruptions happen; but it sounds like we haven't seen such things and we probably wouldn't have a great way to detect them now anyway (i.e. the reduce task would mark a fetch successful and just crash). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-147552582 Hey Imran, Given the number of changes required for this approach, I wonder whether an atomic rename design wouldn't be simpler (in particular, the "first attempt wins" in the doc). The doc seems to be worried that a file output might be corrupted, but in that case, why not send a message to the node asking it to delete its old output files, and then send a new map task? It can just be the delete-block message that the block manager already supports. This seems much nicer because it doesn't require any changes to the data structures in the rest of Spark. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-147519993 I will get @JoshRosen to take a look at this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146979682 I looked at the diffs since my last review, looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146973041 @vanzin @JoshRosen made external shuffle service backwards compatible and got rid of DAGSchedulerFailureRecoverySuite --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146946481 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43475/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146946480 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146946253 [Test build #43475 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43475/console) for PR 6648 at commit [`c9a9e08`](https://github.com/apache/spark/commit/c9a9e08ed5db1b13d8b6e77149dbd1552a9794ec). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)` * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` * `public final class UnsafeRow extends MutableRow implements Externalizable, KryoSerializable ` * ` /** Run a function within Hive state (SessionState, HiveConf, Hive client and class loader) */` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146904630 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43474/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146904627 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146903055 [Test build #43475 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43475/consoleFull) for PR 6648 at commit [`c9a9e08`](https://github.com/apache/spark/commit/c9a9e08ed5db1b13d8b6e77149dbd1552a9794ec). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146900480 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146902032 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146901974 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146900537 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146900213 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146893976 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146893923 [Test build #43470 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43470/console) for PR 6648 at commit [`37ac799`](https://github.com/apache/spark/commit/37ac799f53bc4d09319c51a45e1079c9f9cee166). * This patch **fails MiMa tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_: * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)` * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146893979 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43470/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146889726 [Test build #43470 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43470/consoleFull) for PR 6648 at commit [`37ac799`](https://github.com/apache/spark/commit/37ac799f53bc4d09319c51a45e1079c9f9cee166). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146886622 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146886660 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/6648#discussion_r41575239
--- Diff:
core/src/main/java/org/apache/spark/shuffle/unsafe/UnsafeShuffleWriter.java ---
@@ -125,6 +129,7 @@ public UnsafeShuffleWriter(
taskContext.taskMetrics().shuffleWriteMetrics_$eq(Option.apply(writeMetrics));
this.taskContext = taskContext;
this.sparkConf = sparkConf;
+this.allowSpillMove =
sparkConf.getBoolean("spark.shuffle.unsafe.testing.allowSpillMove", true);
--- End diff --
I think at the same time you were adding this comment, I put in a comment,
but on the variable declaration:
https://github.com/apache/spark/pull/6648/files#diff-642ce9f439435408382c3ac3b5c5e0a0R99
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/6648#discussion_r41575081
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
---
@@ -163,12 +163,12 @@ public void registerExecutor(
/**
* Obtains a FileSegmentManagedBuffer from a shuffle block id. We expect
the blockId has the
- * format "shuffle_ShuffleId_MapId_ReduceId" (from ShuffleBlockId), and
additionally make
- * assumptions about how the hash and sort based shuffles store their
data.
+ * format "shuffle_ShuffleId_MapId_ReduceId_StageAttemptId" (from
ShuffleBlockId), and
+ * additionally make assumptions about how the hash and sort based
shuffles store their data.
*/
public ManagedBuffer getBlockData(String appId, String execId, String
blockId) {
String[] blockIdParts = blockId.split("_");
-if (blockIdParts.length < 4) {
+if (blockIdParts.length != 5) {
--- End diff --
I think Marcelo has a really good point. We should probably support both,
no sense in making it impossible to support both. I'll update for this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146659581 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43401/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146659319 [Test build #43401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43401/console) for PR 6648 at commit [`f37be91`](https://github.com/apache/spark/commit/f37be9120499d699a2da34cb69620cd8dba9cb8a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)` * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146659579 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6648#discussion_r41548771
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
---
@@ -163,12 +163,12 @@ public void registerExecutor(
/**
* Obtains a FileSegmentManagedBuffer from a shuffle block id. We expect
the blockId has the
- * format "shuffle_ShuffleId_MapId_ReduceId" (from ShuffleBlockId), and
additionally make
- * assumptions about how the hash and sort based shuffles store their
data.
+ * format "shuffle_ShuffleId_MapId_ReduceId_StageAttemptId" (from
ShuffleBlockId), and
+ * additionally make assumptions about how the hash and sort based
shuffles store their data.
*/
public ManagedBuffer getBlockData(String appId, String execId, String
blockId) {
String[] blockIdParts = blockId.split("_");
-if (blockIdParts.length < 4) {
+if (blockIdParts.length != 5) {
--- End diff --
This happens to work today, as far as I know, but I don't think that we
explicitly stated that we support this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146639169 Looks sane, but this isn't really my area of expertise. Just a reminder that you should either enable DAGSchedulerFailureRecoverySuite or remove it from the patch. Also, left a question about backwards compatibilty. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6648#discussion_r41546189
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
---
@@ -163,12 +163,12 @@ public void registerExecutor(
/**
* Obtains a FileSegmentManagedBuffer from a shuffle block id. We expect
the blockId has the
- * format "shuffle_ShuffleId_MapId_ReduceId" (from ShuffleBlockId), and
additionally make
- * assumptions about how the hash and sort based shuffles store their
data.
+ * format "shuffle_ShuffleId_MapId_ReduceId_StageAttemptId" (from
ShuffleBlockId), and
+ * additionally make assumptions about how the hash and sort based
shuffles store their data.
*/
public ManagedBuffer getBlockData(String appId, String execId, String
blockId) {
String[] blockIdParts = blockId.split("_");
-if (blockIdParts.length < 4) {
+if (blockIdParts.length != 5) {
--- End diff --
Hmm... is the contract that a new shuffle service should work with apps
running an old Spark version? In that case, shouldn't this have different code
paths for `length == 4` and `length == 5`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6648#discussion_r41544610
--- Diff: core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala
---
@@ -36,8 +43,15 @@ private[spark] trait ShuffleManager {
numMaps: Int,
dependency: ShuffleDependency[K, V, C]): ShuffleHandle
- /** Get a writer for a given partition. Called on executors by map
tasks. */
- def getWriter[K, V](handle: ShuffleHandle, mapId: Int, context:
TaskContext): ShuffleWriter[K, V]
+ /** Get a writer for a given partition. Called on executors by map tasks.
--- End diff --
super nit: comment should follow javadoc formatting:
/**
* Comment.
*/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6648#discussion_r41543763
--- Diff:
core/src/main/java/org/apache/spark/shuffle/unsafe/UnsafeShuffleWriter.java ---
@@ -125,6 +129,7 @@ public UnsafeShuffleWriter(
taskContext.taskMetrics().shuffleWriteMetrics_$eq(Option.apply(writeMetrics));
this.taskContext = taskContext;
this.sparkConf = sparkConf;
+this.allowSpillMove =
sparkConf.getBoolean("spark.shuffle.unsafe.testing.allowSpillMove", true);
--- End diff --
Can you add a comment in the code explaining it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146608768 [Test build #43401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43401/consoleFull) for PR 6648 at commit [`f37be91`](https://github.com/apache/spark/commit/f37be9120499d699a2da34cb69620cd8dba9cb8a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146603453 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146603482 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146578338 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43396/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146578278 [Test build #43396 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43396/console) for PR 6648 at commit [`c7b3017`](https://github.com/apache/spark/commit/c7b3017487ee0d2f0a34fb29ba7dc4875a51ffcb). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)` * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146578332 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146574433 [Test build #43396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43396/consoleFull) for PR 6648 at commit [`c7b3017`](https://github.com/apache/spark/commit/c7b3017487ee0d2f0a34fb29ba7dc4875a51ffcb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146572501 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146572450 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146567168 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43395/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146567159 [Test build #43395 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43395/console) for PR 6648 at commit [`5547611`](https://github.com/apache/spark/commit/5547611896f644b84cdd8651c96b2efcd8a56e7c). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)` * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146567165 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146566496 [Test build #43395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43395/consoleFull) for PR 6648 at commit [`5547611`](https://github.com/apache/spark/commit/5547611896f644b84cdd8651c96b2efcd8a56e7c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146564323 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-146564355 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128171315 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128171222 [Test build #237 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/237/console) for PR 6648 at commit [`26b6ea6`](https://github.com/apache/spark/commit/26b6ea6350b24bbdcf089a852092505772bc790d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)` * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128160443 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128160391 [Test build #39908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39908/console) for PR 6648 at commit [`26b6ea6`](https://github.com/apache/spark/commit/26b6ea6350b24bbdcf089a852092505772bc790d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)` * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128132364 [Test build #39908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39908/consoleFull) for PR 6648 at commit [`26b6ea6`](https://github.com/apache/spark/commit/26b6ea6350b24bbdcf089a852092505772bc790d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128131872 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128131741 [Test build #237 has started](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/237/consoleFull) for PR 6648 at commit [`26b6ea6`](https://github.com/apache/spark/commit/26b6ea6350b24bbdcf089a852092505772bc790d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128131850 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128131589 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128131558 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128131451 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128122426 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128122356 [Test build #39883 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39883/console) for PR 6648 at commit [`26b6ea6`](https://github.com/apache/spark/commit/26b6ea6350b24bbdcf089a852092505772bc790d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)` * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128071372 [Test build #39883 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39883/consoleFull) for PR 6648 at commit [`26b6ea6`](https://github.com/apache/spark/commit/26b6ea6350b24bbdcf089a852092505772bc790d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128070990 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128070970 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-124159452 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-124159280 [Test build #38226 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38226/console) for PR 6648 at commit [`90ee54a`](https://github.com/apache/spark/commit/90ee54a767589ba579030a3eab8407bff452a583). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-124141027 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-124140799 [Test build #38221 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38221/console) for PR 6648 at commit [`9cd9c75`](https://github.com/apache/spark/commit/9cd9c75df281eda6a39540760ff3d07a6d3fe426). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int, stageAttemptId: Int)` * `case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` * `case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, stageAttemptId: Int, reduceId: Int)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
