[GitHub] spark pull request: [SPARK-6331] Load new master URL if present wh...

2015-03-14 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/5024#discussion_r26436080 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala --- @@ -43,10 +43,15 @@ class Checkpoint(@transient ssc: StreamingConte

[GitHub] spark pull request: [SPARK-5987] [MLlib] Save/load for GaussianMix...

2015-03-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4986#issuecomment-80037314 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-5987] [MLlib] Save/load for GaussianMix...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4986#issuecomment-80037247 [Test build #28607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28607/consoleFull) for PR 4986 at commit [`4321743`](https://gith

[GitHub] spark pull request: [SPARK-6331] Load new master URL if present wh...

2015-03-14 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/5024#issuecomment-80039109 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark pull request: [SPARK-6331] Load new master URL if present wh...

2015-03-14 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/5024#discussion_r26436131 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala --- @@ -43,10 +43,15 @@ class Checkpoint(@transient ssc: StreamingContext, val ch

[GitHub] spark pull request: [SPARK-6210] [SQL] use prettyString as column ...

2015-03-14 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/5006#issuecomment-80065079 cc @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enable

[GitHub] spark pull request: [SPARK-6210] [SQL] use prettyString as column ...

2015-03-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/5006#issuecomment-80084986 LGTM - merging in master & branch-1.3. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does n

[GitHub] spark pull request: [SPARK-6210] [SQL] use prettyString as column ...

2015-03-14 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5006 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-5922][GraphX]: Add diff(other: RDD[Vert...

2015-03-14 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/4733#discussion_r26436343 --- Diff: graphx/src/test/scala/org/apache/spark/graphx/VertexRDDSuite.scala --- @@ -17,6 +17,7 @@ package org.apache.spark.graphx +import

[GitHub] spark pull request: [SPARK-5922][GraphX]: Add diff(other: RDD[Vert...

2015-03-14 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/4733#discussion_r26436345 --- Diff: project/MimaExcludes.scala --- @@ -181,6 +181,9 @@ object MimaExcludes { ProblemFilters.exclude[MissingClassProblem]("org.apache.spa

[GitHub] spark pull request: [SPARK-6195] [SQL] Adds in-memory column type ...

2015-03-14 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/4938#issuecomment-8013 Added micro benchmark result. Notice that explicit casting is required because `ScalaReflection` can't capture precision information. This doesn't affect use cases wher

[GitHub] spark pull request: [SPARK-5790][GraphX]: VertexRDD's won't zip pr...

2015-03-14 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5023#issuecomment-80138149 OK to merge, since tests are good. Does this mean SPARK-5790 is essentially resolved by this? sounds like it was fixed separately, and this is about getting in additional

[GitHub] spark pull request: [SPARK-6195] [SQL] Adds in-memory column type ...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4938#issuecomment-80140405 [Test build #28608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28608/consoleFull) for PR 4938 at commit [`fef5338`](https://githu

[GitHub] spark pull request: [SQL]Delete some dupliate code in HiveThriftSe...

2015-03-14 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5007 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-6195] [SQL] Adds in-memory column type ...

2015-03-14 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/4938#issuecomment-80146169 /cc @yhuai, this should be helpful for the TPC-DS benchmark. Gonna merge this once Jenkins nods. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: SPARK-5134 [BUILD] Bump default Hadoop version...

2015-03-14 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/5027 SPARK-5134 [BUILD] Bump default Hadoop version to 2+ Bump default Hadoop version to 2.2.0. (This is already the dependency version reported by published Maven artifacts.) See JIRA for further discuss

[GitHub] spark pull request: SPARK-5134 [BUILD] Bump default Hadoop version...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5027#issuecomment-80157961 [Test build #28609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28609/consoleFull) for PR 5027 at commit [`acbee14`](https://githu

[GitHub] spark pull request: [SPARK-6195] [SQL] Adds in-memory column type ...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4938#issuecomment-80232632 [Test build #28608 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28608/consoleFull) for PR 4938 at commit [`fef5338`](https://gith

[GitHub] spark pull request: [SPARK-6195] [SQL] Adds in-memory column type ...

2015-03-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4938#issuecomment-80232679 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: SPARK-5134 [BUILD] Bump default Hadoop version...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5027#issuecomment-80253524 [Test build #28609 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28609/consoleFull) for PR 5027 at commit [`acbee14`](https://gith

[GitHub] spark pull request: SPARK-5134 [BUILD] Bump default Hadoop version...

2015-03-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5027#issuecomment-80253600 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6189][DataFrame][SQL] Replace "." in sc...

2015-03-14 Thread mgdadv
Github user mgdadv closed the pull request at: https://github.com/apache/spark/pull/4982 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-3957]: show broadcast variable resource...

2015-03-14 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/2851#issuecomment-80383022 Hi, @squito , thanks for the comments, I made some revision on the patch --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-4012] exit JVM process when the excepti...

2015-03-14 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/5004#issuecomment-80395411 @srowen thanks for the comments, the reason to change FsHistoryProvider is that the runner generated by this function is essentially executed by the threads w

[GitHub] spark pull request: [SPARK-6195] [SQL] Adds in-memory column type ...

2015-03-14 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4938 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-3957]: show broadcast variable resource...

2015-03-14 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/2851#issuecomment-80476703 thanks for the updates! just one small style comment. FWIW, after thinking about it more, I am in favor of your approach of sending the block updates, just want to get

[GitHub] spark pull request: [SPARK-5790][GraphX]: VertexRDD's won't zip pr...

2015-03-14 Thread brennonyork
Github user brennonyork commented on the pull request: https://github.com/apache/spark/pull/5023#issuecomment-80538110 Sorry @srowen I should've been more clear, but yes you're correct. SPARK-5790 ended up duplicating most effort from SPARK-1955, PR #4705, but @maropu had created test

[GitHub] spark pull request: [SPARK-6329][Docs]: Minor doc changes for Meso...

2015-03-14 Thread brennonyork
Github user brennonyork commented on the pull request: https://github.com/apache/spark/pull/5022#issuecomment-80538767 Agreed, wasn't exactly sure if this would work as I'm not sure what tool is parsing the `md` files, but figured it was the best option. --- If your project is set up

[GitHub] spark pull request: [ML] SPARK-2426: Breeze QuadraticMinimizer and...

2015-03-14 Thread debasish83
Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-80555943 For NNLS also it is applicable...Let me use lapack ssbmv basically to do symmetric matrix vector multiply for generating gradients --- If your project is set up for i

[GitHub] spark pull request: [SPARK-4894][mllib] Added Bernoulli option to ...

2015-03-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/4087#discussion_r26439272 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala --- @@ -35,26 +39,30 @@ import org.apache.spark.sql.{DataFrame, SQLCon

[GitHub] spark pull request: [SPARK-4894][mllib] Added Bernoulli option to ...

2015-03-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/4087#discussion_r26439275 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala --- @@ -262,4 +303,58 @@ object NaiveBayes { def train(input: R

[GitHub] spark pull request: [SPARK-4894][mllib] Added Bernoulli option to ...

2015-03-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/4087#discussion_r26439273 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala --- @@ -156,9 +181,14 @@ object NaiveBayesModel extends Loader[NaiveBa

[GitHub] spark pull request: [SPARK-4894][mllib] Added Bernoulli option to ...

2015-03-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/4087#discussion_r26439278 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala --- @@ -166,6 +196,14 @@ class NaiveBayes private (private var lambda:

[GitHub] spark pull request: [SPARK-4894][mllib] Added Bernoulli option to ...

2015-03-14 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/4087#issuecomment-80565896 Thanks for the updates again! I responded inline to those 2 comments above. I'll write up something for the model versioning and send a PR. --- If your projec

[GitHub] spark pull request: [SPARK-5922][GraphX]: Add diff(other: RDD[Vert...

2015-03-14 Thread brennonyork
Github user brennonyork commented on a diff in the pull request: https://github.com/apache/spark/pull/4733#discussion_r26439303 --- Diff: project/MimaExcludes.scala --- @@ -181,6 +181,9 @@ object MimaExcludes { ProblemFilters.exclude[MissingClassProblem]("org.apach

[GitHub] spark pull request: [SPARK-5922][GraphX]: Add diff(other: RDD[Vert...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4733#issuecomment-80568303 [Test build #28610 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28610/consoleFull) for PR 4733 at commit [`f86375c`](https://githu

[GitHub] spark pull request: [SPARK-5084][SQL]add if not exists after creat...

2015-03-14 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3895#issuecomment-80582987 This LGTM, merging to master. Thanks for working on this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-5753] [SQL] add JDBCRDD support for pos...

2015-03-14 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/4549#issuecomment-80592824 @marmbrus @lepfhty The integration tests were removed in #4872 because they cause dependency issue and are ignored from the very beginning anyway. I plan to but haven't

[GitHub] spark pull request: [SPARK-5922][GraphX]: Add diff(other: RDD[Vert...

2015-03-14 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/4733#discussion_r26439546 --- Diff: project/MimaExcludes.scala --- @@ -181,6 +181,9 @@ object MimaExcludes { ProblemFilters.exclude[MissingClassProblem]("org.apache.spa

[GitHub] spark pull request: [SPARK-6329][Docs]: Minor doc changes for Meso...

2015-03-14 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5022 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-6332] [MLlib] compute calibration curve...

2015-03-14 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/5025#issuecomment-80624475 @robert-dodier Thanks for the PR! I added a couple of clarification questions to the JIRA --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-3382][MLLIB] GradientDescent convergenc...

2015-03-14 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/3636#issuecomment-80626659 @Lewuathe Was a decision made about @mengxr 's suggestion about using relative convergence tolerance (instead of absolute, which is used by this PR currently)? LBFGS

[GitHub] spark pull request: [SPARK-5790][GraphX]: VertexRDD's won't zip pr...

2015-03-14 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5023 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/4906#issuecomment-80628130 @MechCoder > I have a query regarding the use of mapPartition vs map. I see that mapPartition is preferred when there is a huge overhead for each task, is that rig

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/4906#discussion_r26439846 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/loss/Loss.scala --- @@ -49,4 +49,14 @@ trait Loss extends Serializable { */ def co

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/4906#discussion_r26439845 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/loss/AbsoluteError.scala --- @@ -61,4 +61,18 @@ object AbsoluteError extends Loss { ma

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/4906#discussion_r26439847 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala --- @@ -108,6 +110,58 @@ class GradientBoostedTreesModel(

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/4906#discussion_r26439865 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/loss/Loss.scala --- @@ -47,6 +47,18 @@ trait Loss extends Serializable { * @param data Tr

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread MechCoder
Github user MechCoder commented on a diff in the pull request: https://github.com/apache/spark/pull/4906#discussion_r26439882 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/loss/AbsoluteError.scala --- @@ -61,4 +61,18 @@ object AbsoluteError extends Loss { ma

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/4906#issuecomment-80639905 That should cover it! It will still be a couple of days before my API PR is ready, so we should be able to finish and merge this before any conflicts arise. Thanks!

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/4906#discussion_r26439907 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/loss/AbsoluteError.scala --- @@ -61,4 +61,18 @@ object AbsoluteError extends Loss { ma

[GitHub] spark pull request: [SPARK-5084][SQL]add if not exists after creat...

2015-03-14 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3895#issuecomment-80642173 I actually question why we need this at all. As far as I understand, we only need this function because of a bug in the way we are initializing `TestHive` (the only pla

[GitHub] spark pull request: [SPARK-5922][GraphX]: Add diff(other: RDD[Vert...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4733#issuecomment-80646241 [Test build #28610 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28610/consoleFull) for PR 4733 at commit [`f86375c`](https://gith

[GitHub] spark pull request: [SPARK-5922][GraphX]: Add diff(other: RDD[Vert...

2015-03-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4733#issuecomment-80646292 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread MechCoder
Github user MechCoder commented on the pull request: https://github.com/apache/spark/pull/4906#issuecomment-80651819 @jkbradley should look ok now! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4906#issuecomment-80652118 [Test build #28611 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28611/consoleFull) for PR 4906 at commit [`c04a430`](https://githu

[GitHub] spark pull request: [SPARK-6326][SQL] Improve castStruct to be fas...

2015-03-14 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/5017#issuecomment-80659066 Please post benchmark results for PRs with performance improvements. --- If your project is set up for it, you can reply to this email and have your reply appear on GitH

[GitHub] spark pull request: [SPARK-4012] exit JVM process when the excepti...

2015-03-14 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/5004#issuecomment-80663465 Is it really OK to System.exit() the driver JVM? This may be user code that has an embedded SparkContext. The SparkUncaughtExceptionHandler is suitable for Executors, wh

[GitHub] spark pull request: SPARK-6044 [CORE] RDD.aggregate() should not u...

2015-03-14 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/5028 SPARK-6044 [CORE] RDD.aggregate() should not use the closure serializer on the zero value Use configured serializer in RDD.aggregate, to match PairRDDFunctions.aggregateByKey, instead of closure ser

[GitHub] spark pull request: SPARK-6044 [CORE] RDD.aggregate() should not u...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5028#issuecomment-80673656 [Test build #28612 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28612/consoleFull) for PR 5028 at commit [`a4040a7`](https://githu

[GitHub] spark pull request: [SPARK-6269] Use a different implementation of...

2015-03-14 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4972#issuecomment-80674937 @mccheah yeah are you able to go back and revise the PR title, JIRA title, and description? I can touch some of that up on merge but just boiling it down to a quick and up

[GitHub] spark pull request: [SPARK-5843] Allowing map-side combine to be s...

2015-03-14 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4634#issuecomment-80680853 Going once going twice for final comments as I'd like to resolve this one and move on. Right now it's two mild thumbs up, one thumbs down, and a neutral (?) from @rxin who

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4906#issuecomment-80689362 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4906#issuecomment-80689341 [Test build #28611 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28611/consoleFull) for PR 4906 at commit [`c04a430`](https://gith

[GitHub] spark pull request: [SPARK-4012] exit JVM process when the excepti...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5004#issuecomment-80697805 [Test build #28613 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28613/consoleFull) for PR 5004 at commit [`6087864`](https://githu

[GitHub] spark pull request: [SPARK-4012] exit JVM process when the excepti...

2015-03-14 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/5004#issuecomment-80698614 @aarondav , thanks for the insightful suggestion I just updated the patch the change becomes a bit bigger, as I need to create a new method in Utils as

[GitHub] spark pull request: SPARK-6044 [CORE] RDD.aggregate() should not u...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5028#issuecomment-80702753 [Test build #28612 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28612/consoleFull) for PR 5028 at commit [`a4040a7`](https://gith

[GitHub] spark pull request: SPARK-6044 [CORE] RDD.aggregate() should not u...

2015-03-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5028#issuecomment-80702776 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: SPARK-6044 [CORE] RDD.aggregate() should not u...

2015-03-14 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5028#issuecomment-80703329 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request: SPARK-6338 [CORE] Use standard temp dir mechan...

2015-03-14 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/5029 SPARK-6338 [CORE] Use standard temp dir mechanisms in tests to avoid orphaned temp files Use `Utils.createTempDir()` to replace other temp file mechanisms used in some tests, to further ensure they

[GitHub] spark pull request: SPARK-6338 [CORE] Use standard temp dir mechan...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5029#issuecomment-80725677 [Test build #28614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28614/consoleFull) for PR 5029 at commit [`1a12efa`](https://githu

[GitHub] spark pull request: [SPARK-4012] stop SparkContext when the except...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5004#issuecomment-80725697 [Test build #28613 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28613/consoleFull) for PR 5004 at commit [`6087864`](https://gith

[GitHub] spark pull request: [SPARK-4012] stop SparkContext when the except...

2015-03-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5004#issuecomment-80725704 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-4012] stop SparkContext when the except...

2015-03-14 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5004#discussion_r26441964 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1156,6 +1156,18 @@ private[spark] object Utils extends Logging { }

[GitHub] spark pull request: [SPARK-4012] stop SparkContext when the except...

2015-03-14 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5004#discussion_r26441965 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1156,6 +1156,18 @@ private[spark] object Utils extends Logging { }

[GitHub] spark pull request: [SPARK-4012] stop SparkContext when the except...

2015-03-14 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5004#discussion_r26441972 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1156,6 +1156,18 @@ private[spark] object Utils extends Logging { }

[GitHub] spark pull request: [SPARK-4012] stop SparkContext when the except...

2015-03-14 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5004#discussion_r26441974 --- Diff: core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala --- @@ -90,8 +93,9 @@ private[spark] abstract class AsynchronousListenerBus

[GitHub] spark pull request: [SPARK-4012] stop SparkContext when the except...

2015-03-14 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5004#discussion_r26441977 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -145,7 +145,7 @@ private[spark] class TaskSchedulerImpl( i

[GitHub] spark pull request: [SPARK-4012] stop SparkContext when the except...

2015-03-14 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5004#discussion_r26441980 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -93,7 +93,7 @@ private[history] class FsHistoryProvider(conf: Sp

[GitHub] spark pull request: SPARK-6333 [CORE] Added compression option to ...

2015-03-14 Thread deenar
GitHub user deenar opened a pull request: https://github.com/apache/spark/pull/5030 SPARK-6333 [CORE] Added compression option to RDD.saveAsObjectFile added to JavaAPI too You can merge this pull request into a Git repository by running: $ git pull https://github.com/deenar/spa

[GitHub] spark pull request: SPARK-6333 [CORE] Added compression option to ...

2015-03-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5030#issuecomment-80741476 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: SPARK-6333 [CORE] Added compression option to ...

2015-03-14 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/5030#discussion_r26442290 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -1318,7 +1318,7 @@ abstract class RDD[T: ClassTag]( /** * Save this RDD as a

[GitHub] spark pull request: SPARK-6333 [CORE] Added compression option to ...

2015-03-14 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5030#issuecomment-80742476 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabl

[GitHub] spark pull request: SPARK-6333 [CORE] Added compression option to ...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5030#issuecomment-80742910 [Test build #28615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28615/consoleFull) for PR 5030 at commit [`c90eee2`](https://githu

[GitHub] spark pull request: SPARK-6338 [CORE] Use standard temp dir mechan...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5029#issuecomment-80751314 [Test build #28614 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28614/consoleFull) for PR 5029 at commit [`1a12efa`](https://gith

[GitHub] spark pull request: SPARK-6338 [CORE] Use standard temp dir mechan...

2015-03-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5029#issuecomment-80751339 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: SPARK-6333 [CORE] Added compression option to ...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5030#issuecomment-80753614 [Test build #28615 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28615/consoleFull) for PR 5030 at commit [`c90eee2`](https://gith

[GitHub] spark pull request: SPARK-6333 [CORE] Added compression option to ...

2015-03-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5030#issuecomment-80753642 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-5843] Allowing map-side combine to be s...

2015-03-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/4634#issuecomment-80753791 Give me until Monday. Want to think a little bit more about this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as w

[GitHub] spark pull request: [SPARK-6269] Use a different implementation of...

2015-03-14 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/4972#discussion_r26442690 --- Diff: core/src/main/scala/org/apache/spark/util/SizeEstimator.scala --- @@ -184,14 +182,14 @@ private[spark] object SizeEstimator extends Logging { pri

[GitHub] spark pull request: [SPARK-6269] Use a different implementation of...

2015-03-14 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/4972#discussion_r26442700 --- Diff: core/src/main/scala/org/apache/spark/util/SizeEstimator.scala --- @@ -213,7 +211,7 @@ private[spark] object SizeEstimator extends Logging {

[GitHub] spark pull request: [SPARK-6269] Use a different implementation of...

2015-03-14 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/4972#discussion_r26442697 --- Diff: core/src/main/scala/org/apache/spark/util/SizeEstimator.scala --- @@ -200,7 +198,7 @@ private[spark] object SizeEstimator extends Logging {

[GitHub] spark pull request: [SPARK-6269] Use a different implementation of...

2015-03-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/4972#issuecomment-80755993 Other than the two minor things I pointed out, and the need to update title/pr description, this change lgtm. Thanks! --- If your project is set up for it, you can reply t

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/4906#discussion_r26442790 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala --- @@ -108,6 +110,58 @@ class GradientBoostedTreesModel(

[GitHub] spark pull request: [SPARK-6025] [MLlib] Add helper method evaluat...

2015-03-14 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/4906#issuecomment-80758187 I think that's the last to-do item. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-5922][GraphX]: Add diff(other: RDD[Vert...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4733#issuecomment-80762914 [Test build #28616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28616/consoleFull) for PR 4733 at commit [`b9274af`](https://githu

[GitHub] spark pull request: SPARK-6333 [CORE] Added compression option to ...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5030#issuecomment-80765129 [Test build #28617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28617/consoleFull) for PR 5030 at commit [`d9f8ba3`](https://githu

[GitHub] spark pull request: [SPARK-6222][STREAMING] Make sure batches are ...

2015-03-14 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/4964#issuecomment-80768238 When `concurrentJobs` is 1, we can add optimizations, but since both maps would be of size 1, I don't think the cost would be too high, but I am open to optimizin

[GitHub] spark pull request: [SPARK-6222][STREAMING] Make sure batches are ...

2015-03-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4964#issuecomment-80768268 [Test build #28618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28618/consoleFull) for PR 4964 at commit [`67a5a50`](https://githu

[GitHub] spark pull request: [SPARK-5922][GraphX]: Add diff(other: RDD[Vert...

2015-03-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4733#issuecomment-80780186 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

  1   2   >