[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8007#issuecomment-137297485 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38605830 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -795,7 +796,9 @@ private[spark] object JsonProtocol { case `taskKil

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38605840 --- Diff: core/src/test/scala/org/apache/spark/PartitioningSuite.scala --- @@ -17,13 +17,12 @@ package org.apache.spark -import scala.c

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38605816 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -795,7 +796,9 @@ private[spark] object JsonProtocol { case `taskKil

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38605811 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -390,7 +390,7 @@ private[spark] class MesosSched

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8007#issuecomment-137297118 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8007#issuecomment-137297119 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [SPARK-10332][CORE] Fix yarn spark executor va...

2015-09-02 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/8580#issuecomment-137297074 @vanzin : Do you think this is something we should try getting into 1.5 since its a regression, or just try and put it in master? --- If your project is set up for it,

[GitHub] spark pull request: [SPARK-10411][SQL]Move visualization above exp...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8570#issuecomment-137296885 [Test build #41957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41957/consoleFull) for PR 8570 at commit [`9978206`](https://gith

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8007#issuecomment-137296753 [Test build #41956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41956/consoleFull) for PR 8007 at commit [`e967fa4`](https://gith

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38605550 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -91,6 +94,36 @@ private[spark] abstract class YarnSched

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38605520 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -91,6 +94,36 @@ private[spark] abstract class YarnSched

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-09-02 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/5636#issuecomment-137296698 thanks for reviews @kayousterhout and @andrewor14 , and the quick updates @ilganeli ! --- If your project is set up for it, you can reply to this email and have your rep

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-09-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r38605474 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,283 @@ class DAGSchedulerSuite assertDataStructure

[GitHub] spark pull request: [SPARK-10411][SQL]Move visualization above exp...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8570#issuecomment-137296634 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8007#issuecomment-137296602 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8007#issuecomment-137296635 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-10411][SQL]Move visualization above exp...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8570#issuecomment-137296595 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-10411][SQL]Move visualization above exp...

2015-09-02 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/8570#issuecomment-137296538 Fixed it. These are the new screenshots. https://cloud.githubusercontent.com/assets/1000778/9648751/8a66378a-521e-11e5-90db-e4373af4e110.png";> https://cloud.

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38605369 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -803,13 +829,17 @@ private class LogisticAggregator(

[GitHub] spark pull request: [SPARK-10332][CORE] Fix yarn spark executor va...

2015-09-02 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8580#issuecomment-137296256 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled a

[GitHub] spark pull request: [Spark-9769][ML][PY] add python api for countv...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8561#issuecomment-137296017 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [Spark-9769][ML][PY] add python api for countv...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8561#issuecomment-137296013 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [Spark-9769][ML][PY] add python api for countv...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8561#issuecomment-137295702 [Test build #41954 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41954/console) for PR 8561 at commit [`87bbe8c`](https://github.

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/8007#issuecomment-137295407 Ah sorry @mccheah I was actually not done yet. I will resume now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitH

[GitHub] spark pull request: [SPARK-8137][core] Improve treeAggregate to co...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/7461#issuecomment-137295244 Yeah the problem is there's no reliable way to tell whether the number of executors that your job will be run with in advance. New executors can register within the s

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8007#issuecomment-137295051 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8007#issuecomment-137295040 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread mccheah
Github user mccheah commented on the pull request: https://github.com/apache/spark/pull/8007#issuecomment-137294983 Accidentally missed one of @andrewor14 's comments but everything else is addressed I think. One point to note: I moved the loss-reason-request-reply logic into

[GitHub] spark pull request: [SPARK-10422] [SQL] String column in InMemoryC...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8578#issuecomment-137294378 [Test build #1715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1715/console) for PR 8578 at commit [`1469cb0`](https://github

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-09-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r38604678 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,282 @@ class DAGSchedulerSuite assertDataStructure

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread rotationsymmetry
Github user rotationsymmetry commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38604536 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -803,13 +829,17 @@ private class LogisticAggregato

[GitHub] spark pull request: [Spark-9769][ML][PY] add python api for countv...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8561#issuecomment-137293348 [Test build #41954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41954/consoleFull) for PR 8561 at commit [`87bbe8c`](https://gith

[GitHub] spark pull request: [SPARK-10422] [SQL] String column in InMemoryC...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8578#issuecomment-137292157 [Test build #41952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41952/consoleFull) for PR 8578 at commit [`2905fd5`](https://gith

[GitHub] spark pull request: [SPARK-10332][CORE] Fix yarn spark executor va...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8580#issuecomment-137292083 [Test build #41953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41953/consoleFull) for PR 8580 at commit [`f7e5b00`](https://gith

[GitHub] spark pull request: [Spark-9769][ML][PY] add python api for countv...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8561#issuecomment-137291471 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [Spark-9769][ML][PY] add python api for countv...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8561#issuecomment-137291419 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-09-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r38604281 --- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala --- @@ -110,3 +133,8 @@ private[spark] abstract class Stage( case _ => false

[GitHub] spark pull request: [SPARK-10332][CORE] Fix yarn spark executor va...

2015-09-02 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/8580#issuecomment-137290430 cc @srowen & @vanzin who were on the ticket. I'm not sure if this should be targeted for 1.5 but if RC3 doesn't make it seems like it might be reasonable (since its a re

[GitHub] spark pull request: [SPARK-10332][CORE] Fix yarn spark executor va...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8580#issuecomment-137289466 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-10332][CORE] Fix yarn spark executor va...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8580#issuecomment-137289378 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-10422] [SQL] String column in InMemoryC...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8578#issuecomment-137289465 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-10422] [SQL] String column in InMemoryC...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8578#issuecomment-137289402 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-10422] [SQL] String column in InMemoryC...

2015-09-02 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/8578#discussion_r38603982 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/columnar/InMemoryColumnarQuerySuite.scala --- @@ -191,4 +191,24 @@ class InMemoryColumnarQuerySuite exte

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38603904 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala --- @@ -59,8 +64,7 @@ class LogisticRegressionSuite extends Sp

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8579#issuecomment-137287379 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-10332][CORE] Fix yarn spark executor va...

2015-09-02 Thread holdenk
GitHub user holdenk opened a pull request: https://github.com/apache/spark/pull/8580 [SPARK-10332][CORE] Fix yarn spark executor validation From Jira: Running spark-submit with yarn with number-executors equal to 0 when not using dynamic allocation should error out. In spar

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8579#issuecomment-137287380 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [SPARK-9851] Support submitting map stages ind...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8180#issuecomment-137287375 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-9851] Support submitting map stages ind...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8180#issuecomment-137287377 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [SPARK-9851] Support submitting map stages ind...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8180#issuecomment-137287309 [Test build #41947 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41947/console) for PR 8180 at commit [`dac96b7`](https://github.

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8579#issuecomment-137287302 [Test build #41948 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41948/console) for PR 8579 at commit [`16756a6`](https://github.

[GitHub] spark pull request: [SPARK-10049][SPARKR] Support collecting data ...

2015-09-02 Thread sun-rui
Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/8458#discussion_r38601927 --- Diff: core/src/main/scala/org/apache/spark/api/r/RBackendHandler.scala --- @@ -171,35 +183,53 @@ private[r] class RBackendHandler(server: RBackend)

[GitHub] spark pull request: [SPARK-10049][SPARKR] Support collecting data ...

2015-09-02 Thread sun-rui
Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/8458#discussion_r38601919 --- Diff: R/pkg/R/SQLContext.R --- @@ -49,7 +49,7 @@ infer_type <- function(x) { stopifnot(length(x) > 0) names <- names(x) if (is.n

[GitHub] spark pull request: [SPARK-8137][core] Improve treeAggregate to co...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/7461#discussion_r38601860 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -1080,6 +1080,21 @@ abstract class RDD[T: ClassTag]( var partiallyAggregated

[GitHub] spark pull request: [SPARK-10422] [SQL] String column in InMemoryC...

2015-09-02 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/8578#discussion_r38601580 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/columnar/InMemoryColumnarQuerySuite.scala --- @@ -191,4 +191,13 @@ class InMemoryColumnarQuerySuite ext

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8579#issuecomment-137277982 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8579#issuecomment-137277936 [Test build #41946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41946/console) for PR 8579 at commit [`8a81df4`](https://github.

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8579#issuecomment-137277981 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38600527 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -264,9 +266,13 @@ class CoarseGrainedScheduler

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38600407 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala --- @@ -135,11 +135,11 @@ private[spark] class SparkDep

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38600357 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -264,9 +266,13 @@ class CoarseGrainedScheduler

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38600287 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala --- @@ -92,6 +94,11 @@ private[spark] object CoarseGrai

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38600072 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -31,14 +31,18 @@ import org.apache.spark.util.Utils * old stages to be resubmit

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38600038 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -722,10 +728,7 @@ private[spark] class TaskSetManager( put(

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38600012 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -709,6 +709,12 @@ private[spark] class TaskSetManager( }

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38599910 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorLossReason.scala --- @@ -23,13 +23,19 @@ import org.apache.spark.executor.ExecutorExitCode

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38599854 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorLossReason.scala --- @@ -23,13 +23,19 @@ import org.apache.spark.executor.ExecutorExitCode

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38599699 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -202,8 +209,17 @@ case class TaskCommitDenied(jobID: Int, partitionID: Int, attem

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38599463 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -194,6 +200,7 @@ case object TaskKilled extends TaskFailedReason { case class

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38599567 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -202,8 +209,17 @@ case class TaskCommitDenied(jobID: Int, partitionID: Int, attem

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38599409 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -194,6 +200,7 @@ case object TaskKilled extends TaskFailedReason { case class

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38599389 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -31,14 +31,18 @@ import org.apache.spark.util.Utils * old stages to be resub

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r38599322 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -31,14 +31,18 @@ import org.apache.spark.util.Utils * old stages to be resub

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5636#issuecomment-137274766 [Test build #41951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41951/consoleFull) for PR 5636 at commit [`5bb1ae6`](https://gith

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-09-02 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/8512#issuecomment-137274773 @piaozhexiu This improvement looks really exciting, great work! We have some other places that also need this kind of improvements, I'm learning how to borrow the

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5636#issuecomment-137273921 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5636#issuecomment-137273907 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r38598339 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,283 @@ class DAGSchedulerSuite assertDataStruc

[GitHub] spark pull request: [SPARK-10422] [SQL] String column in InMemoryC...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8578#issuecomment-137272654 [Test build #1715 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1715/consoleFull) for PR 8578 at commit [`1469cb0`](https://git

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-09-02 Thread ilganeli
Github user ilganeli commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r38597940 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,283 @@ class DAGSchedulerSuite assertDataStructu

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8567#issuecomment-137272142 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8567#issuecomment-137272144 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8567#issuecomment-137272059 [Test build #41950 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41950/console) for PR 8567 at commit [`bf324c4`](https://github.

[GitHub] spark pull request: [SPARK-8137][core] Improve treeAggregate to co...

2015-09-02 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7461#issuecomment-137272017 btw @kmadhugit -- your point about `numExecutors` is very true. I thought about this a bit more and we could in some cases have a very large cluster and not all executo

[GitHub] spark pull request: [SPARK-8137][core] Improve treeAggregate to co...

2015-09-02 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/7461#discussion_r38597517 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -1080,6 +1080,21 @@ abstract class RDD[T: ClassTag]( var partiallyAggregated =

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-09-02 Thread ilganeli
Github user ilganeli commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r38597257 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,283 @@ class DAGSchedulerSuite assertDataStructu

[GitHub] spark pull request: Merge pull request #1 from apache/master

2015-09-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/8576#issuecomment-137271147 @Jianfeng-chs looks like this is opened by mistake. Please close this PR. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-8137][core] Improve treeAggregate to co...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/7461#discussion_r38596921 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -1080,6 +1080,21 @@ abstract class RDD[T: ClassTag]( var partiallyAggregated

[GitHub] spark pull request: [SPARK-8137][core] Improve treeAggregate to co...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/7461#discussion_r3859 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -1080,6 +1080,21 @@ abstract class RDD[T: ClassTag]( var partiallyAggregated

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38596112 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala --- @@ -781,4 +808,117 @@ class LogisticRegressionSuite extend

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38595932 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -218,31 +217,51 @@ class LogisticRegression(override val uid

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38595939 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -803,13 +829,17 @@ private class LogisticAggregator(

[GitHub] spark pull request: [SPARK-10422] [SQL] String column in InMemoryC...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8578#issuecomment-137268317 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-10422] [SQL] String column in InMemoryC...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8578#issuecomment-137268318 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [SPARK-10422] [SQL] String column in InMemoryC...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8578#issuecomment-137268254 [Test build #41944 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41944/console) for PR 8578 at commit [`1469cb0`](https://github.

[GitHub] spark pull request: [SPARK-4449][Core]Specify port range in spark

2015-09-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5722#issuecomment-137268180 (EDIT: I posted a suggestion that I didn't like in retrospect, so I deleted it. Please disregard.) --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...

2015-09-02 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8218#discussion_r38595606 --- Diff: network/common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java --- @@ -109,15 +111,34 @@ public void connectionTerminated(

[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...

2015-09-02 Thread tedyu
Github user tedyu commented on a diff in the pull request: https://github.com/apache/spark/pull/8218#discussion_r38595554 --- Diff: network/common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java --- @@ -109,15 +111,34 @@ public void connectionTerminated(C

[GitHub] spark pull request: [SPARK-4449][Core]Specify port range in spark

2015-09-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5722#issuecomment-137267811 OK I think one thing we all agree is that SPARK-4449 means have different port retries across different ports. One way to do it is to have the special syntax

<    1   2   3   4   5   6   7   >