[GitHub] spark pull request: [SPARK-2213] [SQL] sort merge join for spark s...

2015-04-13 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/5208#discussion_r28287120 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/Exchange.scala --- @@ -168,14 +190,16 @@ private[sql] case class AddExchange(sqlContext: SQ

[GitHub] spark pull request: [SPARK-2213] [SQL] sort merge join for spark s...

2015-04-13 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/5208#discussion_r28286721 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala --- @@ -72,6 +72,12 @@ abstract class SparkPlan extends QueryPlan[SparkPlan]

[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...

2015-04-13 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5354#issuecomment-92521916 In this case, `tachyon` includes `thrift` (shaded) and `httpclient` (unshaded). Adding a direct dependency on `httpclient` shouldn't do anything. Shading hasn't changed th

[GitHub] spark pull request: Export driver quirks

2015-04-13 Thread rtreffer
Github user rtreffer commented on the pull request: https://github.com/apache/spark/pull/5498#issuecomment-92521352 Added a ticket: https://issues.apache.org/jira/browse/SPARK-6888 Will add that to the commit after some sleep .zZzZzZ --- If your project is set up for it, you can r

[GitHub] spark pull request: [SPARK-2213] [SQL] sort merge join for spark s...

2015-04-13 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/5208#discussion_r28286620 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/Exchange.scala --- @@ -69,7 +72,16 @@ case class Exchange(newPartitioning: Partitioning, ch

[GitHub] spark pull request: [SPARK-6734] [SQL] Add UDTF.close support in G...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5383#issuecomment-92520820 [Test build #30202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30202/consoleFull) for PR 5383 at commit [`d719983`](https://githu

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/5497#discussion_r28286461 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer2.scala --- @@ -0,0 +1,378 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark pull request: SPARK-6698: where RandomForest input specifies...

2015-04-13 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/5351#issuecomment-92520525 @bien Did it help? I'm not sure if it will, but I could imagine it helping depending on whether the RDDs are getting materialized multiple times. Adjusting executor m

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/4688#issuecomment-92520458 I need a second look, but looks sane, other than RPC system weirdness which isn't really your fault. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28286363 --- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala --- @@ -74,6 +77,13 @@ private[spark] class CoarseGrainedExecutorBa

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28286340 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala --- @@ -0,0 +1,211 @@ +/* + * Licensed to the Apach

[GitHub] spark pull request: [SPARK-6065] [MLlib] Optimize word2vec.findSyn...

2015-04-13 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/5467#issuecomment-92519784 Yep, that's pretty much what I had in mind, except that I'd recommend: * using MLlib's local Matrix type (and its BLAS call in mllib.linalg.BLAS) * computing and

[GitHub] spark pull request: [SPARK-5972] [MLlib] Cache residuals and gradi...

2015-04-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5330 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-5972] [MLlib] Cache residuals and gradi...

2015-04-13 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/5330#issuecomment-92519859 Merged into master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/5497#discussion_r28286134 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer2.scala --- @@ -0,0 +1,378 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28286065 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala --- @@ -0,0 +1,211 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/5497#discussion_r28286063 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer2.scala --- @@ -0,0 +1,378 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28286027 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala --- @@ -0,0 +1,211 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28286012 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala --- @@ -0,0 +1,211 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark pull request: [WIP][HOTFIX][SPARK-4123]: Fix bug in PR depen...

2015-04-13 Thread brennonyork
Github user brennonyork commented on the pull request: https://github.com/apache/spark/pull/5443#issuecomment-92518053 @shaneknapp can you help me understand how Jenkins is doing the checkouts? I'm seeing the PR builder outputting: ``` Building remotely on amp-jenkins-work

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28285855 --- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala --- @@ -164,6 +174,9 @@ private[spark] object CoarseGraine

[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...

2015-04-13 Thread calvinjia
Github user calvinjia commented on the pull request: https://github.com/apache/spark/pull/5354#issuecomment-92518143 Sorry for the bad example, here is (hopefully) a better explanation: Previously: `tachyon-client` depends on `tachyon` `tachyon` depends on `hadoop-clien

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28285820 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala --- @@ -0,0 +1,211 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark pull request: [SPARK-6886] [PySpark] fix big closure with sh...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5496#issuecomment-92517801 [Test build #30192 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30192/consoleFull) for PR 5496 at commit [`9a0ea4c`](https://gith

[GitHub] spark pull request: [SPARK-6886] [PySpark] fix big closure with sh...

2015-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5496#issuecomment-92517832 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28285643 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala --- @@ -0,0 +1,211 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark pull request: [SPARK-4897] [PySpark] Python 3 support

2015-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5173#issuecomment-92517129 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28285532 --- Diff: core/src/main/scala/org/apache/spark/deploy/ExecutorDelegationTokenUpdater.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-4897] [PySpark] Python 3 support

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5173#issuecomment-92517118 [Test build #30193 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30193/consoleFull) for PR 5173 at commit [`5a55ab4`](https://gith

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28285458 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala --- @@ -0,0 +1,211 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28285427 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala --- @@ -0,0 +1,211 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28285231 --- Diff: core/src/main/scala/org/apache/spark/deploy/ExecutorDelegationTokenUpdater.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28285158 --- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala --- @@ -164,6 +174,9 @@ private[spark] object CoarseGrainedExecutor

[GitHub] spark pull request: [Minor][SparkR] Minor refactor and removes red...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5495#issuecomment-92515969 [Test build #30200 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30200/consoleFull) for PR 5495 at commit [`74ec303`](https://githu

[GitHub] spark pull request: [SPARK-6881][SparkR] Changes the checkpoint di...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5493#issuecomment-92515980 [Test build #30201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30201/consoleFull) for PR 5493 at commit [`e67fc40`](https://githu

[GitHub] spark pull request: [SPARK-6887][SQL] ColumnBuilder misses FloatTy...

2015-04-13 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/5499#issuecomment-92515941 @liancheng Where should I add the test? At first, I thought `NullableColumnBuilderSuite` is the place But, why `NullableColumnBuilderSuite` does not really use those real c

[GitHub] spark pull request: [SPARK-6887][SQL] ColumnBuilder misses FloatTy...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5499#issuecomment-92515928 [Test build #30199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30199/consoleFull) for PR 5499 at commit [`d169d33`](https://githu

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28285004 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala --- @@ -201,6 +206,49 @@ class SparkHadoopUtil extends Logging { val baseS

[GitHub] spark pull request: [SPARK-6887][SQL] ColumnBuilder misses FloatTy...

2015-04-13 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/5499 [SPARK-6887][SQL] ColumnBuilder misses FloatType https://issues.apache.org/jira/browse/SPARK-6887 You can merge this pull request into a Git repository by running: $ git pull https://github.com/

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28284885 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala --- @@ -201,6 +206,49 @@ class SparkHadoopUtil extends Logging { val baseS

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28284858 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala --- @@ -17,18 +17,22 @@ package org.apache.spark.deploy +impor

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28284852 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala --- @@ -17,18 +17,22 @@ package org.apache.spark.deploy +impor

[GitHub] spark pull request: [Minor][SparkR] Minor refactor and removes red...

2015-04-13 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/5495#issuecomment-92515198 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [SPARK-6881][SparkR] Changes the checkpoint di...

2015-04-13 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/5493#issuecomment-92515235 Jenkins, add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [SPARK-6881][SparkR] Changes the checkpoint di...

2015-04-13 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/5493#issuecomment-92515144 Jenkins, ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28284735 --- Diff: core/src/main/scala/org/apache/spark/deploy/ExecutorDelegationTokenUpdater.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apac

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28284671 --- Diff: core/src/main/scala/org/apache/spark/deploy/ExecutorDelegationTokenUpdater.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-3454] [WIP] separate json endpoints for...

2015-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4435#issuecomment-92514838 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30

[GitHub] spark pull request: [SPARK-3454] [WIP] separate json endpoints for...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4435#issuecomment-92514829 [Test build #30191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30191/consoleFull) for PR 4435 at commit [`cf86175`](https://gith

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28284358 --- Diff: core/src/main/scala/org/apache/spark/deploy/ExecutorDelegationTokenUpdater.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/4688#discussion_r28284272 --- Diff: core/src/main/scala/org/apache/spark/deploy/ExecutorDelegationTokenUpdater.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...

2015-04-13 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5236#issuecomment-92514243 Looking strong. I think that's all the comments addressed now. Going once, going twice? --- If your project is set up for it, you can reply to this email and have your re

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/5497#issuecomment-92514199 @yhuai this is a really cool improvement, definitely will improve the performance a lot. I have some of the comments about the future improvement(of course we can

[GitHub] spark pull request: [SPARK-6844][SQL] Clean up accumulators used i...

2015-04-13 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/5475#issuecomment-92513734 I see, thanks for clarifying. It seems bad that we are creating accumulators per copy anyway then. Can we just pass in the accumulator with the buffers and statistics

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/5497#discussion_r28283974 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala --- @@ -139,6 +141,8 @@ private[sql] class SQLConf extends Serializable { *

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/5431#issuecomment-92513221 @mengxr I added minor comment but don't have major ones. That's it for now! --- If your project is set up for it, you can reply to this email and have your reply appea

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/5431#discussion_r28283755 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/shared/SharedParamsCodeGen.scala --- @@ -0,0 +1,169 @@ +/* + * Licensed to the Apache Soft

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/5431#discussion_r28283689 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/params.scala --- @@ -325,7 +379,7 @@ class ParamMap private[ml] (private val map: mutable.Map[Param

[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...

2015-04-13 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5354#issuecomment-92512638 Yes, let's call the shaded B artifact B'. Are you saying that the published artifact of A shades B and C? yes, then it contains B' and C' and they are inseparable. It isn'

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/5497#discussion_r28283598 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer2.scala --- @@ -0,0 +1,378 @@ +/* + * Licensed to the Apach

[GitHub] spark pull request: [Spark-6839] BlockManger.dataDeserialize must ...

2015-04-13 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/5463#issuecomment-92512403 I talked to @squito offline and it doesn't seem like there's a less intrusive solution to this issue, so this looks OK to me. But I'm not that familiar with these code pat

[GitHub] spark pull request: [Spark-6839] BlockManger.dataDeserialize must ...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/5463#discussion_r28283453 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockStore.scala --- @@ -22,13 +22,18 @@ import java.nio.ByteBuffer import scala.collection.mutable.

[GitHub] spark pull request: [Spark-6839] BlockManger.dataDeserialize must ...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/5463#discussion_r28283431 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -755,104 +769,115 @@ private[spark] class BlockManager( case _ => n

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/5431#discussion_r28283381 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/params.scala --- @@ -179,52 +179,96 @@ trait Params extends Identifiable with Serializable {

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/5497#discussion_r28283271 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/Exchange.scala --- @@ -45,6 +46,27 @@ case class Exchange(newPartitioning: Partitionin

[GitHub] spark pull request: Export driver quirks

2015-04-13 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5498#issuecomment-92511690 This needs a JIRA -- see https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark --- If your project is set up for it, you can reply to this email and hav

[GitHub] spark pull request: Export driver quirks

2015-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5498#issuecomment-92511404 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: Export driver quirks

2015-04-13 Thread rtreffer
GitHub user rtreffer opened a pull request: https://github.com/apache/spark/pull/5498 Export driver quirks Make it possible to (temporary) overwrite the driver quirks. This can be used to overcome problems with specific schemas or to add new jdbc driver support on the fly.

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/5497#discussion_r28282909 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer2.scala --- @@ -0,0 +1,378 @@ +/* + * Licensed to the Apach

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/5497#discussion_r28282279 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer2.scala --- @@ -0,0 +1,378 @@ +/* + * Licensed to the Apach

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/5431#discussion_r28282819 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/params.scala --- @@ -179,52 +179,96 @@ trait Params extends Identifiable with Serializable {

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/5497#discussion_r28282630 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer2.scala --- @@ -0,0 +1,378 @@ +/* + * Licensed to the Apach

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/5431#discussion_r28282538 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/params.scala --- @@ -55,58 +49,42 @@ class Param[T] ( */ def ->(value: T): ParamPair[

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/5431#discussion_r28282519 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/Classifier.scala --- @@ -17,15 +17,16 @@ package org.apache.spark.ml.classifica

[GitHub] spark pull request: [SPARK-6229] Add SASL encryption to network li...

2015-04-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/5377#discussion_r28282594 --- Diff: network/common/src/main/java/org/apache/spark/network/sasl/SaslClientBootstrap.java --- @@ -62,13 +76,26 @@ public void doBootstrap(TransportClient

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/5497#discussion_r28282545 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer2.scala --- @@ -0,0 +1,378 @@ +/* + * Licensed to the Apach

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/5431#discussion_r28282554 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/params.scala --- @@ -179,52 +179,96 @@ trait Params extends Identifiable with Serializable {

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/5431#discussion_r28282550 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/params.scala --- @@ -179,52 +179,96 @@ trait Params extends Identifiable with Serializable {

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/5431#discussion_r28282562 --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/SchemaUtils.scala --- @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/5431#discussion_r28282559 --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/SchemaUtils.scala --- @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark pull request: [SQL][SPARK-6742]: Don't push down predicates ...

2015-04-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5390 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/5431#discussion_r28282508 --- Diff: mllib/src/test/scala/org/apache/spark/ml/param/ParamsSuite.scala --- @@ -78,23 +81,42 @@ class ParamsSuite extends FunSuite { }

[GitHub] spark pull request: WIP: re-enable flaky test to catch failure in ...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5459#issuecomment-92509801 [Test build #30198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30198/consoleFull) for PR 5459 at commit [`2ead85b`](https://githu

[GitHub] spark pull request: [SQL][SPARK-6742]: Don't push down predicates ...

2015-04-13 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/5390#issuecomment-92509822 Thanks! Merged to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5957][ML] better handling of parameters

2015-04-13 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/5431#issuecomment-92509721 Should this method in Params be made abstract? ``` def validate(paramMap: ParamMap): Unit = {} ``` I just realized we haven't been using it, and making it

[GitHub] spark pull request: [SPARK-6006][SQL]: Optimize count distinct for...

2015-04-13 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/4764#issuecomment-92509606 Here is the JIRA: SPARK-4366. Unless you think you will have something in the next day or two, would you mind closing this JIRA. I'd like to keep the PR queue to only

[GitHub] spark pull request: WIP: re-enable flaky test to catch failure in ...

2015-04-13 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/5459#issuecomment-92509413 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/5497#discussion_r28282234 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer2.scala --- @@ -0,0 +1,378 @@ +/* + * Licensed to the Apach

[GitHub] spark pull request: [SPARK-2669] [yarn] Distribute client configur...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4142#issuecomment-92508870 [Test build #30197 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30197/consoleFull) for PR 4142 at commit [`f5434b9`](https://githu

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5497#issuecomment-92507937 [Test build #30196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30196/consoleFull) for PR 5497 at commit [`2379eeb`](https://gith

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5497#issuecomment-92507955 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30

[GitHub] spark pull request: [SPARK-6130] [SQL] support if not exists for i...

2015-04-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4865 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-6130] [SQL] support if not exists for i...

2015-04-13 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/4865#issuecomment-92506360 Thanks! Merged to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5360] [SPARK-6606] Eliminate duplicate ...

2015-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4145#issuecomment-92505669 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30

[GitHub] spark pull request: [SPARK-5360] [SPARK-6606] Eliminate duplicate ...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4145#issuecomment-92505635 [Test build #30189 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30189/consoleFull) for PR 4145 at commit [`85156c3`](https://gith

[GitHub] spark pull request: WIP: re-enable flaky test to catch failure in ...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5459#issuecomment-92505634 [Test build #30188 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30188/consoleFull) for PR 5459 at commit [`2ead85b`](https://gith

[GitHub] spark pull request: WIP: re-enable flaky test to catch failure in ...

2015-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5459#issuecomment-92505667 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30

[GitHub] spark pull request: [SPARK-6368][SQL] Build a specialized serializ...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5497#issuecomment-92502644 [Test build #30196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30196/consoleFull) for PR 5497 at commit [`2379eeb`](https://githu

[GitHub] spark pull request: [SPARK-6734] [SQL] Add UDTF.close support in G...

2015-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5383#issuecomment-92501509 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30

[GitHub] spark pull request: [SPARK-6734] [SQL] Add UDTF.close support in G...

2015-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5383#issuecomment-92501504 [Test build #30195 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30195/consoleFull) for PR 5383 at commit [`63c88cc`](https://gith

<    1   2   3   4   5   6   7   >