[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4123#discussion_r23283721 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -19,7 +19,7 @@ package org.apache.spark.deploy.yarn import sc

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70799955 We should warn about this in standalone mode and mesos coarse grained mode as well, right? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-5345][DEPLOY] Fix unstable test case in...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4133#issuecomment-70800435 [Test build #25880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25880/consoleFull) for PR 4133 at commit [`77678fe`](https://gith

[GitHub] spark pull request: [SPARK-5345][DEPLOY] Fix unstable test case in...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4133#issuecomment-70800443 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-4874] [CORE] Collect record count metri...

2015-01-21 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4067#discussion_r23284115 --- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala --- @@ -238,6 +245,10 @@ case class InputMetrics(readMethod: DataReadMethod.Value) {

[GitHub] spark pull request: [SPARK-4874] [CORE] Collect record count metri...

2015-01-21 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4067#discussion_r23284154 --- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala --- @@ -317,4 +338,9 @@ class ShuffleWriteMetrics extends Serializable { * Tim

[GitHub] spark pull request: [SPARK-4874] [CORE] Collect record count metri...

2015-01-21 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4067#discussion_r23284162 --- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala --- @@ -41,7 +41,7 @@ import org.apache.spark._ import org.apache.spark.annotation.Develo

[GitHub] spark pull request: [SPARK-4988][SQL] Fix: 'Create table ..as sele...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3821#issuecomment-70801020 [Test build #25881 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25881/consoleFull) for PR 3821 at commit [`a9b78d3`](https://gith

[GitHub] spark pull request: [SPARK-4874] [CORE] Collect record count metri...

2015-01-21 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4067#discussion_r23284264 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/BlockStoreShuffleFetcher.scala --- @@ -82,7 +82,15 @@ private[hash] object BlockStoreShuffleFetcher

[GitHub] spark pull request: [SPARK-4988][SQL] Fix: 'Create table ..as sele...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3821#issuecomment-70801024 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-4874] [CORE] Collect record count metri...

2015-01-21 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4067#discussion_r23284251 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/BlockStoreShuffleFetcher.scala --- @@ -82,7 +82,15 @@ private[hash] object BlockStoreShuffleFetcher

[GitHub] spark pull request: [SPARK-5118][SQL] Fix: create table test store...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3921#issuecomment-70801372 [Test build #25879 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25879/consoleFull) for PR 3921 at commit [`9da56f8`](https://gith

[GitHub] spark pull request: [SPARK-5118][SQL] Fix: create table test store...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3921#issuecomment-70801376 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [MLLIB] [SPARK-2222] Add multiclass evaluation...

2015-01-21 Thread tolgap
Github user tolgap commented on the pull request: https://github.com/apache/spark/pull/1155#issuecomment-70801697 @avulanov How many neurons does the output layer have in this case? 1 or 10? Because my current implementation has an output layer of 10 neurons, e.g: ```scala

[GitHub] spark pull request: [SPARK-4654][CORE] Clean up DAGScheduler getMi...

2015-01-21 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/4134 [SPARK-4654][CORE] Clean up DAGScheduler getMissingParentStages / stageDependsOn methods This is a follow up of https://github.com/apache/spark/pull/3515 with some minor fix. @JoshRosen do yo

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70802105 @sryza In standalone and mesos mode it is a bit different. JIRA is here:https://issues.apache.org/jira/browse/SPARK-5337 Seems like @CodingCat has created

[GitHub] spark pull request: [SPARK-4654][CORE] Clean up DAGScheduler getMi...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4134#issuecomment-70802189 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

2015-01-21 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/4129#issuecomment-70802269 What if app.coresLeft < app.desc.coreNumPerTask? And should we fix the same issue in mesos mode in this PR? --- If your project is set up for it, you can repl

[GitHub] spark pull request: [SPARK-5325] [SQL] Shrink the Hive shim layer

2015-01-21 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/4107#issuecomment-70803084 @yhuai @marmbrus It would be great if you can help reviewing this when you have time. This is not super urgent though. --- If your project is set up for it, you can re

[GitHub] spark pull request: [SPARK-4699][SQL] make caseSensitive configura...

2015-01-21 Thread jackylk
Github user jackylk commented on the pull request: https://github.com/apache/spark/pull/3558#issuecomment-70803953 Please run the test again. I tested all sql and hive test case locally, and no error is reported. I am not sure why online test fails --- If your project is set up for i

[GitHub] spark pull request: [SPARK-5345][DEPLOY] Fix unstable test case in...

2015-01-21 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4133#issuecomment-70804009 Spark.testing should already be set for all tests. Is that needed ? What is the problem briefly and what does this fix? --- If your project is set up for it, you can repl

[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...

2015-01-21 Thread uncleGen
GitHub user uncleGen opened a pull request: https://github.com/apache/spark/pull/4135 [SPARK-5205][Streaming]:Inconsistent behaviour between Streaming job and others, when click kill link in WebUI The "kill" link is used to kill a stage in job. It works in any kinds of Spark job bu

[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...

2015-01-21 Thread uncleGen
Github user uncleGen commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-70804703 @JoshRosen Since my local branch is out of date and this PR contains merge conflicts, I will open a [new PR](https://github.com/apache/spark/pull/4135) and close this fi

[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...

2015-01-21 Thread uncleGen
Github user uncleGen closed the pull request at: https://github.com/apache/spark/pull/4008 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is en

[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70805185 [Test build #25882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25882/consoleFull) for PR 4135 at commit [`c90a288`](https://githu

[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70805248 [Test build #25882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25882/consoleFull) for PR 4135 at commit [`c90a288`](https://gith

[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70805253 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70805768 [Test build #25883 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25883/consoleFull) for PR 4123 at commit [`d03d6fa`](https://githu

[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70806362 [Test build #25884 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25884/consoleFull) for PR 4135 at commit [`fc3be2c`](https://githu

[GitHub] spark pull request: [SPARK-4988][SQL] Fix: 'Create table ..as sele...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3821#issuecomment-70806372 [Test build #25885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25885/consoleFull) for PR 3821 at commit [`f3709f4`](https://githu

[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70806543 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70806538 [Test build #25884 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25884/consoleFull) for PR 4135 at commit [`fc3be2c`](https://gith

[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70807559 [Test build #25886 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25886/consoleFull) for PR 4135 at commit [`d85dede`](https://githu

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70808154 [Test build #25887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25887/consoleFull) for PR 4123 at commit [`9632d3a`](https://githu

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70808340 Actually when cluster has not enough cores, the driver will also hang up. I made a trivial change here and hope it would not make much noise for this pr. --- If

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70814056 [Test build #25883 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25883/consoleFull) for PR 4123 at commit [`d03d6fa`](https://gith

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70814065 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-4988][SQL] Fix: 'Create table ..as sele...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3821#issuecomment-70814842 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-4988][SQL] Fix: 'Create table ..as sele...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3821#issuecomment-70814836 [Test build #25885 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25885/consoleFull) for PR 3821 at commit [`f3709f4`](https://gith

[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70816298 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70816290 [Test build #25886 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25886/consoleFull) for PR 4135 at commit [`d85dede`](https://gith

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70817246 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70817239 [Test build #25887 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25887/consoleFull) for PR 4123 at commit [`9632d3a`](https://gith

[GitHub] spark pull request: [SPARK-5337] respect spark.task.cpus when sche...

2015-01-21 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/4129#issuecomment-70823105 Hi, @WangTaoTheTonic , if app.coresLeft < app.desc.coreNumPerTask, we should not assign more cores to the executor...as those cores will just be wasted since we canno

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/4129#issuecomment-70825357 just upload the fix for Mesos... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does no

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4129#issuecomment-70825694 [Test build #25888 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25888/consoleFull) for PR 4129 at commit [`619c8b9`](https://githu

[GitHub] spark pull request: [SPARK-4654][CORE] Clean up DAGScheduler getMi...

2015-01-21 Thread lianhuiwang
Github user lianhuiwang commented on a diff in the pull request: https://github.com/apache/spark/pull/4134#discussion_r23294907 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -349,34 +349,7 @@ class DAGScheduler( } private def

[GitHub] spark pull request: [SPARK-4959][SQL] Attributes are case sensitiv...

2015-01-21 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/4013#issuecomment-70830314 Oh? Should I close this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does no

[GitHub] spark pull request: [MLLib]SPARK-5027:add SVMWithLBFGS interface i...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3890#issuecomment-70832740 [Test build #25889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25889/consoleFull) for PR 3890 at commit [`020a9e2`](https://githu

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4129#issuecomment-70833826 [Test build #25888 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25888/consoleFull) for PR 4129 at commit [`619c8b9`](https://gith

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4129#issuecomment-70833836 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/4129#discussion_r23299327 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -541,17 +541,17 @@ private[spark] class Master( // in the qu

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/4129#discussion_r23299355 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -567,10 +567,12 @@ private[spark] class Master( } else {

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/4129#discussion_r23299503 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala --- @@ -78,7 +78,11 @@ private[spark] class SparkD

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/4129#discussion_r23299588 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala --- @@ -78,7 +78,11 @@ private[spark] class SparkD

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/4129#discussion_r23299681 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala --- @@ -78,7 +78,11 @@ private[spark] class SparkDeployS

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/4129#discussion_r23299757 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -87,7 +89,12 @@ private[spark] class

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/4129#discussion_r23299862 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -87,7 +89,12 @@ private[spark] class Coarse

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/4129#discussion_r2332 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala --- @@ -78,7 +78,11 @@ private[spark] class SparkD

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/4129#discussion_r23300122 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala --- @@ -78,7 +78,11 @@ private[spark] class SparkDeployS

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/4129#discussion_r23300256 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -87,7 +89,12 @@ private[spark] class Coarse

[GitHub] spark pull request: [MLLib]SPARK-5027:add SVMWithLBFGS interface i...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3890#issuecomment-70843031 [Test build #25889 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25889/consoleFull) for PR 3890 at commit [`020a9e2`](https://gith

[GitHub] spark pull request: [MLLib]SPARK-5027:add SVMWithLBFGS interface i...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3890#issuecomment-70843039 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/4123#discussion_r23303748 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -95,6 +95,10 @@ private[spark] class ClientArguments(args: Array[St

[GitHub] spark pull request: [SPARK-5196][SQL] Support `comment` in Create ...

2015-01-21 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/3999#issuecomment-70850713 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request: [SPARK-5173]support python application running...

2015-01-21 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3976#issuecomment-70851442 Sorry its not clear to me, by pyspark I meant the interactive /bin/pyspark, is this intended to make the interactive python shell work or just batch mode specify the py

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-01-21 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r23305535 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java --- @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-01-21 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r23306196 --- Diff: launcher/src/main/java/org/apache/spark/launcher/AbstractLauncher.java --- @@ -0,0 +1,461 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70858064 @tgravescs Updated as you said. Perhaps we should do the same change in `ExecutorAllocationManager`, but I am not familiar with that future, so let's leave it wit

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70858047 [Test build #25890 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25890/consoleFull) for PR 4123 at commit [`6c9676a`](https://githu

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-01-21 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-70857354 At some point we should put some documentation in the .md files about the library and how to use it. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70859826 I agree, the text was that way in ExecutorAllocationManager so lets leave it for now. This looks good. Thanks @WangTaoTheTonic --- If your project is set up

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4123 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-5351][GraphX] Do not use Partitioner.de...

2015-01-21 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/4136 [SPARK-5351][GraphX] Do not use Partitioner.defaultPartitioner as a partitioner of EdgeRDDImp... If the value of 'spark.default.parallelism' does not match the number of partitoins in EdgePartition(

[GitHub] spark pull request: [SPARK-5351][GraphX] Do not use Partitioner.de...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4136#issuecomment-70867791 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: SPARK-1714. Take advantage of AMRMClient APIs ...

2015-01-21 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3765#issuecomment-70869778 looks good. Thanks @sryza --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5176] The thrift server does not suppor...

2015-01-21 Thread tpanningnextcen
GitHub user tpanningnextcen opened a pull request: https://github.com/apache/spark/pull/4137 [SPARK-5176] The thrift server does not support cluster mode Output an error message if the thrift server is started in cluster mode. You can merge this pull request into a Git repository by

[GitHub] spark pull request: SPARK-1714. Take advantage of AMRMClient APIs ...

2015-01-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3765 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-5176] The thrift server does not suppor...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4137#issuecomment-70870989 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-5158] [core] [security] Spark standalon...

2015-01-21 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/4106#discussion_r23312771 --- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala --- @@ -193,17 +193,21 @@ class HadoopRDD[K, V]( override def getPartitions: Array[

[GitHub] spark pull request: [SPARK-5158] [core] [security] Spark standalon...

2015-01-21 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/4106#discussion_r23313017 --- Diff: core/src/main/scala/org/apache/spark/deploy/StandaloneSparkHadoopUtil.scala --- @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software F

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70871880 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-5336][YARN]spark.executor.cores must no...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4123#issuecomment-70871868 [Test build #25890 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25890/consoleFull) for PR 4123 at commit [`6c9676a`](https://gith

[GitHub] spark pull request: [SPARK-5158] [core] [security] Spark standalon...

2015-01-21 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/4106#discussion_r23313220 --- Diff: core/src/main/scala/org/apache/spark/deploy/StandaloneSparkHadoopUtil.scala --- @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software F

[GitHub] spark pull request: [SPARK-5158] [core] [security] Spark standalon...

2015-01-21 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/4106#issuecomment-70872378 So are you trying to add security such that spark cluster would run as one superuser who would have to be configured as proxy user on the hadoop cluster and then each j

[GitHub] spark pull request: [SPARK-4749] [mllib]: Allow initializing KMean...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3610#issuecomment-70874289 [Test build #25891 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25891/consoleFull) for PR 3610 at commit [`a2ebbd3`](https://githu

[GitHub] spark pull request: Log failures in REPL class loading

2015-01-21 Thread gzm0
Github user gzm0 commented on a diff in the pull request: https://github.com/apache/spark/pull/4130#discussion_r23314675 --- Diff: repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala --- @@ -91,7 +91,14 @@ class ExecutorClassLoader(conf: SparkConf, classUri: String

[GitHub] spark pull request: [MLlib] [SPARK-5301] Missing conversions and o...

2015-01-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4089 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-5317]Set BoostingStrategy.defaultParams...

2015-01-21 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/4103#discussion_r23318708 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/BoostingStrategy.scala --- @@ -88,4 +88,25 @@ object BoostingStrategy {

[GitHub] spark pull request: [SPARK-5353] Log failures in REPL class loadin...

2015-01-21 Thread gzm0
Github user gzm0 commented on the pull request: https://github.com/apache/spark/pull/4130#issuecomment-70878284 Just added a JIRA --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-3974][MLlib] Distributed Block Matrix A...

2015-01-21 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/3200#discussion_r23319267 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala --- @@ -0,0 +1,215 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-5119] java.lang.ArrayIndexOutOfBoundsEx...

2015-01-21 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/3975#issuecomment-70897863 @mengxr I agree that setup is simplest, in that it makes users consciously handle all re-labeling explicitly. I'm OK with it if we add data validation at the start of

[GitHub] spark pull request: [SPARK-3974][MLlib] Distributed Block Matrix A...

2015-01-21 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/3200#discussion_r23318868 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala --- @@ -0,0 +1,215 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-5118][SQL] Fix: create table test store...

2015-01-21 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/3921#issuecomment-70881554 I think you can use `describe formatted ` to get the information of a table. You should be able to find `InputFormat` and `OutputFormat` in the output. We can just double c

[GitHub] spark pull request: [SPARK-4894][mllib] Added Bernoulli option to ...

2015-01-21 Thread rnowling
Github user rnowling commented on the pull request: https://github.com/apache/spark/pull/4087#issuecomment-70887128 @leahmcguire The updated patch looks great to me. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If y

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4129#issuecomment-70884543 [Test build #25893 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25893/consoleFull) for PR 4129 at commit [`9f2a8be`](https://githu

[GitHub] spark pull request: [SPARK-4959][SQL] Attributes are case sensitiv...

2015-01-21 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/4013#issuecomment-70879610 Yes, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enab

[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...

2015-01-21 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/4047#discussion_r23322992 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala --- @@ -0,0 +1,472 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-4894][mllib] Added Bernoulli option to ...

2015-01-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4087#issuecomment-70897566 [Test build #25894 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25894/consoleFull) for PR 4087 at commit [`0313c0c`](https://gith

[GitHub] spark pull request: [SPARK-5158] [core] [security] Spark standalon...

2015-01-21 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/4106#discussion_r23320905 --- Diff: core/src/main/scala/org/apache/spark/deploy/StandaloneSparkHadoopUtil.scala --- @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Fou

  1   2   3   4   5   6   >