[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...

2015-09-02 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-137267685 Yes, and if you look at that code, that secret is the return value of `SecurityManager.getSecretKey()`, which on YARN is stored in the `UserGroupInformation` object. Here

[GitHub] spark pull request: [SPARK-4449][Core]Specify port range in spark

2015-09-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5722#issuecomment-137266772 > Ok, I can somewhat see that argument, although I don't see why someone would require that Spark's UI bind to a specific port. If they're building some application o

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-09-02 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/8512#discussion_r38594881 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala --- @@ -249,6 +256,68 @@ class HadoopTableReader( } /** +

[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-137266073 > No, there's one kind of authentication. I don't know what's this "handshake" you talk about. What? I'm talking about the shuffle secret the executor contain

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-09-02 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/8512#discussion_r38594747 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala --- @@ -311,6 +311,19 @@ private[spark] object SQLConf { defaultValue = Some(false

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8567#issuecomment-137265373 [Test build #41950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41950/consoleFull) for PR 8567 at commit [`bf324c4`](https://gith

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8567#issuecomment-137264525 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8567#issuecomment-137264508 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread dbtsai
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/7884#issuecomment-137264169 I will prefer to change the `weights` in linear models to `coefficients` as well! --- If your project is set up for it, you can reply to this email and have your reply a

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38593540 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala --- @@ -781,4 +808,117 @@ class LogisticRegressionSuite extend

[GitHub] spark pull request: [SPARK-10311][Streaming]Reload appId and attem...

2015-09-02 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/8477#issuecomment-137263619 +1. This looks good. @vanzin - I assume your concern is with the reloading. Reloading here is to ignore the old values from the serlialized spark conf so

[GitHub] spark pull request: [SPARK-10421] [build] Exclude curator artifact...

2015-09-02 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8577#issuecomment-137263323 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled a

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8567#issuecomment-137263278 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8567#issuecomment-137263277 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8567#issuecomment-137263271 [Test build #41949 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41949/console) for PR 8567 at commit [`0235a6a`](https://github.

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/8389#issuecomment-137262771 Hi @marmbrus @cloud-fan all my comments are minor and this is mergeable as is, but I have two high level comments: (1) I think we should always put `with Test

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8567#issuecomment-137262398 [Test build #41949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41949/consoleFull) for PR 8567 at commit [`0235a6a`](https://gith

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38592397 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -25,16 +27,8 @@ import org.apache.spark.sql.{Row, SQLConf, QueryTe

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38592440 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveOperatorQueryableSuite.scala --- @@ -17,13 +17,14 @@ package org.apac

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38592416 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/QueryPartitionSuite.scala --- @@ -18,22 +18,20 @@ package org.apache.spark.sql.hive

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38592348 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala --- @@ -51,6 +51,11 @@ object TestHive // SPARK-8910

[GitHub] spark pull request: [SPARK-4449][Core]Specify port range in spark

2015-09-02 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5722#issuecomment-137261869 I think it's almost all about binding to port 80. It doesn't make sense to retry on 81, 82, etc. even if it makes sense for all the other daemons to retry 9000, 9001, etc

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38592233 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetHiveCompatibilitySuite.scala --- @@ -23,13 +23,12 @@ import java.util.{Locale, TimeZone

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38592181 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/ListTablesSuite.scala --- @@ -19,12 +19,12 @@ package org.apache.spark.sql.hive impo

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38592158 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala --- @@ -51,6 +51,11 @@ object TestHive // SPARK-8910

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38592048 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala --- @@ -39,10 +39,8 @@ import org.apache.spark.util.Utils

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8567#issuecomment-137261286 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38591968 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala --- @@ -39,10 +39,8 @@ import org.apache.spark.util.Utils

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8567#issuecomment-137261307 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38591900 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertIntoHiveTableSuite.scala --- @@ -35,7 +35,7 @@ case class TestData(key: Int, value: Stri

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38591851 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala --- @@ -273,6 +273,10 @@ object SparkSQLConfTest extends Logging {

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8579#issuecomment-137260752 [Test build #41948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41948/consoleFull) for PR 8579 at commit [`16756a6`](https://gith

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38591695 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSuite.scala --- @@ -17,15 +17,15 @@ package org.apache.spark.sql.hive

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38591674 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveMetastoreCatalogSuite.scala --- @@ -52,8 +52,10 @@ class HiveMetastoreCatalogSuite extends

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38591631 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDataFrameWindowSuite.scala --- @@ -17,13 +17,14 @@ package org.apache.spark.sql

[GitHub] spark pull request: [SPARK-10176][SQL] Show partially analyzed pla...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8389#discussion_r38591627 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDataFrameJoinSuite.scala --- @@ -17,11 +17,12 @@ package org.apache.spark.sql.h

[GitHub] spark pull request: [SPARK-4449][Core]Specify port range in spark

2015-09-02 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/5722#issuecomment-137260281 > What if you want to run your driver UI on exactly one port Ok, I can somewhat see that argument, although I don't see why someone would require that Spark's UI

[GitHub] spark pull request: [SPARK-9851] Support submitting map stages ind...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8180#issuecomment-137259774 [Test build #41947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41947/consoleFull) for PR 8180 at commit [`dac96b7`](https://gith

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8579#issuecomment-137259513 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8579#issuecomment-137259497 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-10247] [core] improve readability of a ...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8434#issuecomment-137259334 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-10353] [MLLIB] (1.3 backport) BLAS gemm...

2015-09-02 Thread srowen
Github user srowen closed the pull request at: https://github.com/apache/spark/pull/8572 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-9851] Support submitting map stages ind...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/8180#issuecomment-137259196 > For example, consider Andrew Or’s plea in SPARK-8987 which starts with [...] OK, my quote was taken out of its original context. It was intended to moti

[GitHub] spark pull request: [SPARK-10247] [core] improve readability of a ...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8434#issuecomment-137259337 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [SPARK-10247] [core] improve readability of a ...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8434#issuecomment-137259107 [Test build #41942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41942/console) for PR 8434 at commit [`94927c8`](https://github.

[GitHub] spark pull request: [SPARK-4449][Core]Specify port range in spark

2015-09-02 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5722#issuecomment-137258901 What if you want to run your driver UI on exactly one port, but need the executors to try many ports because they may collide with other executors on the cluster from oth

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38590661 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -803,13 +829,17 @@ private class LogisticAggregator(

[GitHub] spark pull request: [SPARK-9066][SQL] Improve cartesian performanc...

2015-09-02 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/7417#issuecomment-137258596 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark pull request: [SPARK-9851] Support submitting map stages ind...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8180#issuecomment-137258503 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-4449][Core]Specify port range in spark

2015-09-02 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/5722#issuecomment-137258443 > If you mean, let N vary per port, yes, that seems to me exactly what a range is. Correct, but my original question was the same as yours: is there a use case f

[GitHub] spark pull request: [SPARK-9066][SQL] Improve cartesian performanc...

2015-09-02 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/7417#discussion_r38590589 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/CartesianProduct.scala --- @@ -27,16 +27,27 @@ import org.apache.spark.sql.execution.{B

[GitHub] spark pull request: [SPARK-9851] Support submitting map stages ind...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8180#issuecomment-137258519 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38590409 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -218,31 +217,51 @@ class LogisticRegression(override val uid

[GitHub] spark pull request: [SPARK-10261][Documentation, ML] Fixed @Since ...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/8554#discussion_r38590198 --- Diff: mllib/src/main/scala/org/apache/spark/ml/evaluation/RegressionEvaluator.scala --- @@ -29,10 +29,12 @@ import org.apache.spark.sql.types.DoubleType

[GitHub] spark pull request: [SPARK-10261][Documentation, ML] Fixed @Since ...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/8554#discussion_r38590092 --- Diff: mllib/src/main/scala/org/apache/spark/ml/evaluation/RegressionEvaluator.scala --- @@ -29,10 +29,12 @@ import org.apache.spark.sql.types.DoubleType

[GitHub] spark pull request: [SPARK-4449][Core]Specify port range in spark

2015-09-02 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5722#issuecomment-137257579 @vanzin you're arguing that you either want to retry all ports N times, or not at all (N=0). That does describe a lot of use cases, but it wouldn't let you, say, let some

[GitHub] spark pull request: [SPARK-10261][Documentation, ML] Fixed @Since ...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/8554#discussion_r38590071 --- Diff: mllib/src/main/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.scala --- @@ -29,10 +29,12 @@ import org.apache.spark.sql.type

[GitHub] spark pull request: [SPARK-10261][Documentation, ML] Fixed @Since ...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/8554#discussion_r38589979 --- Diff: mllib/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala --- @@ -30,16 +30,19 @@ import org.apache.spark.sql.types.Do

[GitHub] spark pull request: [SPARK-10261][Documentation, ML] Fixed @Since ...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/8554#discussion_r38589981 --- Diff: mllib/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala --- @@ -30,16 +30,19 @@ import org.apache.spark.sql.types.Do

[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...

2015-09-02 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-137257022 > it seems then that there are two kinds of authentication No, there's one kind of authentication. I don't know what's this "handshake" you talk about. Whenever y

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8512#issuecomment-137256430 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8579#issuecomment-13725 [Test build #41946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41946/consoleFull) for PR 8579 at commit [`8a81df4`](https://gith

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8512#issuecomment-137256433 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8512#issuecomment-137255840 [Test build #41939 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41939/console) for PR 8512 at commit [`e2007ec`](https://github.

[GitHub] spark pull request: [SPARK-9730][SQL] Add Full Outer Join support ...

2015-09-02 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/8383#discussion_r38589660 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeOuterJoin.scala --- @@ -71,6 +77,7 @@ case class SortMergeOuterJoin(

[GitHub] spark pull request: [SPARK-9730][SQL] Add Full Outer Join support ...

2015-09-02 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/8383#issuecomment-137255108 @viirya I had created #8579. If you'd like to continue to work on this one, feel free to pull that one into this one. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8579#issuecomment-137254994 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-9723][ML] params getordefault should th...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/8567#discussion_r38589547 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/params.scala --- @@ -501,7 +501,13 @@ trait Params extends Identifiable with Serializable { */

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8512#issuecomment-137255071 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8579#issuecomment-137255024 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8512#issuecomment-137255066 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8512#issuecomment-137254810 [Test build #41941 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41941/console) for PR 8512 at commit [`de89710`](https://github.

[GitHub] spark pull request: [SPARK-4449][Core]Specify port range in spark

2015-09-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5722#issuecomment-137254438 Either way I agree with @vanzin that this is not the right fix. I've run Spark in a fire-walled environment where I had to assign a special port to each component tha

[GitHub] spark pull request: [SPARK-9730] [SQL] Add Full Outer Join support...

2015-09-02 Thread davies
GitHub user davies opened a pull request: https://github.com/apache/spark/pull/8579 [SPARK-9730] [SQL] Add Full Outer Join support for SortMergeJoin This PR is based on #8383 , thanks to @viirya JIRA: https://issues.apache.org/jira/browse/SPARK-9730 This patch adds

[GitHub] spark pull request: [SPARK-9730][SQL] Add Full Outer Join support ...

2015-09-02 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/8383#discussion_r38589087 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeOuterJoin.scala --- @@ -271,3 +298,224 @@ private class RightOuterIterator(

[GitHub] spark pull request: [SPARK-4449][Core]Specify port range in spark

2015-09-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5722#issuecomment-137253831 > More importantly imagine running the web UI on port 80; it doesn't make sense to march through 81, 82, 83 as those are different well-known ports. I would a

[GitHub] spark pull request: [SPARK-4229] Create hadoop configuration in a ...

2015-09-02 Thread koeninger
Github user koeninger commented on the pull request: https://github.com/apache/spark/pull/3543#issuecomment-137253615 @andrewor14 master has diverged sufficiently from this PR that I don't think it's useful to keep it merge-able. If we think someone's willing to accept the changes to

[GitHub] spark pull request: [SPARK-10422] [SQL] String column in InMemoryC...

2015-09-02 Thread bignoggins
Github user bignoggins commented on the pull request: https://github.com/apache/spark/pull/8578#issuecomment-137252725 :+1: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabl

[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-137252475 I see, so it seems then that there are two kinds of authentication, one during the handshake where we use some secure secret (i.e. the one used passed from `ExecutorR

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/7884#issuecomment-137251694 1. `weight` vs. `sampleWeight`. I don't think `sampleWeight` solves the problem. I would rather change `weights` in linear models to `coefficients` instead. 2. Using

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/7884#issuecomment-137251765 @dbtsai I made one pass:) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [SPARK-10421] [build] Exclude curator artifact...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8577#issuecomment-137251230 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-10421] [build] Exclude curator artifact...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8577#issuecomment-137251232 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [SPARK-10421] [build] Exclude curator artifact...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8577#issuecomment-137251173 [Test build #41938 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41938/console) for PR 8577 at commit [`8d8683c`](https://github.

[GitHub] spark pull request: [SPARK-9851] Support submitting map stages ind...

2015-09-02 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/8180#discussion_r38587515 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -746,6 +848,63 @@ class DAGScheduler( submitWaitingStages()

[GitHub] spark pull request: [SPARK-10421] [build] Exclude curator artifact...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8577#issuecomment-137250923 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41

[GitHub] spark pull request: [SPARK-10421] [build] Exclude curator artifact...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8577#issuecomment-137250921 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-10421] [build] Exclude curator artifact...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8577#issuecomment-137250800 [Test build #41940 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41940/console) for PR 8577 at commit [`a320d1b`](https://github.

[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...

2015-09-02 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8218#discussion_r38587368 --- Diff: network/common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java --- @@ -109,15 +111,34 @@ public void connectionTerminated(

[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...

2015-09-02 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8218#discussion_r38586992 --- Diff: network/common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java --- @@ -109,15 +111,34 @@ public void connectionTermina

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38586906 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -833,14 +863,13 @@ private class LogisticCostFun( v

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38586942 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala --- @@ -781,4 +808,117 @@ class LogisticRegressionSuite extend

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38586923 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala --- @@ -59,8 +64,7 @@ class LogisticRegressionSuite extends Sp

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38586934 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala --- @@ -216,43 +221,65 @@ class LogisticRegressionSuite extend

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38586917 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala --- @@ -22,10 +22,15 @@ import org.apache.spark.ml.param.Param

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38586928 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala --- @@ -216,43 +221,65 @@ class LogisticRegressionSuite extend

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38586864 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -833,14 +863,13 @@ private class LogisticCostFun( v

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38586801 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -803,13 +829,17 @@ private class LogisticAggregator(

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38586798 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -803,13 +829,17 @@ private class LogisticAggregator(

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

2015-09-02 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/7884#discussion_r38586731 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -218,31 +217,51 @@ class LogisticRegression(override val uid

<    1   2   3   4   5   6   7   >