[GitHub] incubator-spark pull request: [java8API] SPARK-964 Investigate the...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/539#issuecomment-35826380 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12817/ --- If you

[GitHub] incubator-spark pull request: [SPARK-1102] Create a saveAsNewAPIHa...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/636#issuecomment-35826394 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To

[GitHub] incubator-spark pull request: [java8API] SPARK-964 Investigate the...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/539#issuecomment-35826379 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-post y

Anyone wants to look at SPARK-1123?

2014-02-23 Thread Nan Zhu
Hi, all I found the weird thing on saveAsNewAPIHadoopFile in PairRDDFunctions.scala when working on the other issue, saveAsNewAPIHadoopFile throws java.lang.InstantiationException all the time I checked the commit history of the file, it seems that the API exists for a long time, no one e

Re: [DISCUSS] Extending public API

2014-02-23 Thread Cheng Lian
I think SPARK-1063 (PR-503) “Add .sortBy(f) method on RDD” would be a good example. Note that I’m not saying that this PR is already qualified to be accepted, just take it as an example: JIRA issue: https://spark-project.atlassian.net/browse/SPARK-1063 GitHub PR: https://github.com/apache/incubat

Re: [DISCUSS] Extending public API

2014-02-23 Thread Sean Owen
Thank you for bringing this up. I think the current committers are bravely facing down a flood of PRs, and this (among other things) is a step that needs to be taken to scale up and keep this fun. I'd love to have a separate discussion about more steps, but for here I offer two bits of advice from

[GitHub] incubator-spark pull request: SPARK-1071: Tidy logging strategy an...

2014-02-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/570#discussion_r9975073 --- Diff: bagel/pom.xml --- @@ -51,6 +51,11 @@ scalacheck_${scala.binary.version} test + + org.slf4j

[GitHub] incubator-spark pull request: SPARK-1071: Tidy logging strategy an...

2014-02-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/570#discussion_r9975070 --- Diff: project/SparkBuild.scala --- @@ -236,13 +236,15 @@ object SparkBuild extends Build { publishLocalBoth <<= Seq(publishLocal in MavenCo

[GitHub] incubator-spark pull request: SPARK-1071: Tidy logging strategy an...

2014-02-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/570#discussion_r9975071 --- Diff: project/SparkBuild.scala --- @@ -268,9 +272,9 @@ object SparkBuild extends Build { "it.unimi.dsi" % "fastutil"

[GitHub] incubator-spark pull request: SPARK-1084. Fix most build warnings

2014-02-23 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/incubator-spark/pull/586#issuecomment-35827729 @aarondav Sure, it's already split into commits, and one of them has the dependency changes: https://github.com/srowen/incubator-spark/commit/6f2f67974bfedd40bafc

Re: [DISCUSS] Extending public API

2014-02-23 Thread Mridul Muralidharan
Good point, and I was purposefully vague on that since that is something which our community should evolve imo : this was just an initial proposal :-) For example: there are multiple ways to do cartesian - and each has its own trade offs. Another candidate could be, as I mentioned, new methods wh

[GitHub] incubator-spark pull request: SPARK-1084. Fix most build warnings

2014-02-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/586#discussion_r9975099 --- Diff: project/SparkBuild.scala --- @@ -340,7 +336,8 @@ object SparkBuild extends Build { def streamingSettings = sharedSettings ++ Seq(

[GitHub] incubator-spark pull request: SPARK-1084. Fix most build warnings

2014-02-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/586#discussion_r9975107 --- Diff: project/SparkBuild.scala --- @@ -340,7 +336,8 @@ object SparkBuild extends Build { def streamingSettings = sharedSettings ++ Seq(

[GitHub] incubator-spark pull request: Add Security to Spark - Akka, Http, ...

2014-02-23 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/332#discussion_r9975936 --- Diff: core/src/main/scala/org/apache/spark/network/ConnectionManager.scala --- @@ -483,10 +496,131 @@ private[spark] class ConnectionManager(port:

Re: Anyone wants to look at SPARK-1123?

2014-02-23 Thread Nick Pentreath
Hi What KeyClass and ValueClass are you trying to save as the keys/values of your dataset? On Sun, Feb 23, 2014 at 10:48 AM, Nan Zhu wrote: > Hi, all > > I found the weird thing on saveAsNewAPIHadoopFile in > PairRDDFunctions.scala when working on the other issue, > > saveAsNewAPIHadoopFile

ask for receiving spark user mailing list

2014-02-23 Thread Lianhui Wang
hi i want to ask for receiving spark user mailing list -- thanks 王联辉(Lianhui Wang) blog; http://blog.csdn.net/lance_123 兴趣方向:数据库,分布式,数据挖掘,编程语言,互联网技术等

[GitHub] incubator-spark pull request: MLLIB-25: Implicit ALS runs out of m...

2014-02-23 Thread MLnick
Github user MLnick commented on the pull request: https://github.com/apache/incubator-spark/pull/629#issuecomment-35835626 @srowen good catch, thanks Sean. Didn't really think about this when I wrote it. Shows that testing on larger scale input data / params is always required! ---

[GitHub] incubator-spark pull request: SPARK-1078: Replace lift-json with j...

2014-02-23 Thread willb
Github user willb commented on the pull request: https://github.com/apache/incubator-spark/pull/582#issuecomment-35837078 Yes, I'll make the changes today. Thanks, Aaron! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. T

[GitHub] incubator-spark pull request: SPARK-1071: Tidy logging strategy an...

2014-02-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/570#discussion_r9976665 --- Diff: project/SparkBuild.scala --- @@ -236,13 +236,15 @@ object SparkBuild extends Build { publishLocalBoth <<= Seq(publishLocal in MavenCo

[GitHub] incubator-spark pull request: SPARK-1071: Tidy logging strategy an...

2014-02-23 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/incubator-spark/pull/570#issuecomment-35837259 @pwendell I addressed the last point about pulling up slf4j-over-log4j12 into core (non-test), and the indentation issue. Tests look good. --- If your project is

Re: Anyone wants to look at SPARK-1123?

2014-02-23 Thread Nan Zhu
String, it should be get the following helper function private[spark] def getKeyClass() = implicitly[ClassTag[K]].runtimeClass private[spark] def getValueClass() = implicitly[ClassTag[V]].runtimeClass and this is what I run scala> val a = sc.textFile("/Users/nanzhu/code/incubator-spark/LICENS

Re: Anyone wants to look at SPARK-1123?

2014-02-23 Thread Nan Zhu
OK, I know where I was wrong Best, -- Nan Zhu Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Sunday, February 23, 2014 at 12:50 PM, Nan Zhu wrote: > String, it should be get the following helper function > > private[spark] def getKeyClass() = implicitly[ClassTag[K]].runtimeCla

[GitHub] incubator-spark pull request: SPARK-1071: Tidy logging strategy an...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/570#issuecomment-35838298 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-

[GitHub] incubator-spark pull request: SPARK-1071: Tidy logging strategy an...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/570#issuecomment-35838296 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please t

[GitHub] incubator-spark pull request: SPARK-1084. Fix most build warnings

2014-02-23 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/incubator-spark/pull/586#issuecomment-35838441 Ah, great, that'll make it simple. We can only merge at the granularity of PRs, so it'd be great if you could split the dependency stuff into its own. --- If y

Re: [DISCUSS] Extending public API

2014-02-23 Thread Amandeep Khurana
This makes sense. Thanks for clarifying, Mridul. As Sean pointed out - a contrib module quickly turns into a legacy code base that becomes hard to maintain. From that perspective, I think the idea of a separate sparkbank github that is maintained by Spark contributors (along with users who wish to

[GitHub] incubator-spark pull request: [SPARK-1100] prevent Spark from over...

2014-02-23 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/incubator-spark/pull/626#issuecomment-35838665 OK, fixed some bugs and squashed the commits, I think it's ready for further review --- If your project is set up for it, you can reply to this email and have

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

2014-02-23 Thread fommil
Github user fommil commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-35839024 @mengxr looking through all the Apache authorised licenses, it would appear that the Mozilla license is a better fit with my goals since it would require distribu

[GitHub] incubator-spark pull request: SPARK-1071: Tidy logging strategy an...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/570#issuecomment-35839844 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12818/ --- If you

[GitHub] incubator-spark pull request: SPARK-1071: Tidy logging strategy an...

2014-02-23 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/incubator-spark/pull/570#issuecomment-35840761 @srowen thanks for this clean-up. I'm going to merge this into master. --- If your project is set up for it, you can reply to this email and have your reply app

[GitHub] incubator-spark pull request: [SPARK-1100] prevent Spark from over...

2014-02-23 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/incubator-spark/pull/626#issuecomment-35841445 Hey @CodingCat this approach has a few drawbacks. First, it will mean a pretty bad regression for some users. For instance, say that a user is calling saveAsHad

[GitHub] incubator-spark pull request: [SPARK-1100] prevent Spark from over...

2014-02-23 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/incubator-spark/pull/626#issuecomment-35841703 @pwendell Thanks for the comments, I also considered what you mentioned, but will that prevent other components like Spark Streaming from doing the right job?

[GitHub] incubator-spark pull request: SPARK-1122: allCollect functions for...

2014-02-23 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/incubator-spark/pull/635#issuecomment-35841766 Hey @coderxiang - this is interesting functionality but I'm -1 on including it in the standard API. The main reason is that this will perform poorly on most lar

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

2014-02-23 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-35842122 @fommil ASF is silent on the MPL: http://www.apache.org/legal/resolved.html#category-a But Mozilla says it's compatible with AL2: http://www.mozilla.org/MPL/l

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

2014-02-23 Thread dlwh
Github user dlwh commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-35842233 @srowen @fommil Breeze is flexible enough that we can swap out different back ends quickly (and let users decide at runtime). So if need be, I can do the work to ma

[GitHub] incubator-spark pull request: [SPARK-1100] prevent Spark from over...

2014-02-23 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/incubator-spark/pull/626#issuecomment-35842285 @pwendell the second situation can be avoided, sorry, just brain damaged..the only issue is if there is a component relies on the fact that Spark allows th

[GitHub] incubator-spark pull request: SPARK-1071: Tidy logging strategy an...

2014-02-23 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-spark/pull/570 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-post your response. If your project does not have this featur

[GitHub] incubator-spark pull request: SPARK-1084. Fix most build warnings

2014-02-23 Thread srowen
Github user srowen closed the pull request at: https://github.com/apache/incubator-spark/pull/586 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-post your response. If your project does not have this featur

[GitHub] incubator-spark pull request: SPARK-1084. Fix most build warnings

2014-02-23 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/incubator-spark/pull/586#issuecomment-35843240 OK I'm going to come back with two PRs. One will have the squashed final output of this PR, and the other will have the parts related to dependencies (which are n

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

2014-02-23 Thread fommil
Github user fommil commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-35844613 @srowen hehe, oh, I know. Actually I'm more interested in knowing exactly *why* they don't like LGPL. There have been so many discussions in the past between FSF

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

2014-02-23 Thread fommil
Github user fommil commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-35844934 Actually, if somebody creates a ticket for me on https://github.com/fommil/jniloader that's the best way to ensure that I'll actually update the license and relea

Re: standard way of running a compiled jar

2014-02-23 Thread Sandy Ryza
Is the client=driver mode still a supported option (outside of the REPLs), at least for the medium term? My impression from reading the docs is that it's the most common, if not recommended, way to submit jobs. If that's the case, I still think it's important, or at least helpful, to have somethi

[GitHub] incubator-spark pull request: SPARK-1078: Replace lift-json with j...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/582#issuecomment-35845761 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-

[GitHub] incubator-spark pull request: SPARK-1078: Replace lift-json with j...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/582#issuecomment-35845760 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please t

[GitHub] incubator-spark pull request: SPARK-1078: Replace lift-json with j...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/582#issuecomment-35847507 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12819/ ---

[GitHub] incubator-spark pull request: SPARK-1078: Replace lift-json with j...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/582#issuecomment-35847506 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top

[GitHub] incubator-spark pull request: SPARK-1078: Replace lift-json with j...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/582#issuecomment-35847835 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-

[GitHub] incubator-spark pull request: SPARK-1078: Replace lift-json with j...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/582#issuecomment-35847834 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please t

[GitHub] incubator-spark pull request: SPARK-1084 (part 1). Fix most build ...

2014-02-23 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/incubator-spark/pull/637 SPARK-1084 (part 1). Fix most build warnings. This is a redo of https://github.com/apache/incubator-spark/pull/586 This contains all the same changes, minus dependency changes. It al

[GitHub] incubator-spark pull request: SPARK-1122: allCollect functions for...

2014-02-23 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/incubator-spark/pull/635#issuecomment-35848534 @coderxiang btw - it might be something where we make it a private API so it can be used inside of Spark if other packages need this to do broadcast joins. It w

[GitHub] incubator-spark pull request: SPARK-1078: Replace lift-json with j...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/582#issuecomment-35849112 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top

[GitHub] incubator-spark pull request: SPARK-1078: Replace lift-json with j...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/582#issuecomment-35849113 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12820/ ---

[GitHub] incubator-spark pull request: SPARK-1084 (part 1). Fix most build ...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/637#issuecomment-35849130 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-

[GitHub] incubator-spark pull request: SPARK-1084 (part 1). Fix most build ...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/637#issuecomment-35849129 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please t

[GitHub] incubator-spark pull request: SPARK-1122: allCollect functions for...

2014-02-23 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/incubator-spark/pull/635#issuecomment-35849312 @markhamstra @pwendell For the use cases, this allCollect operation may be useful in the grid search for a good set of training parameters for machine learning pr

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread xoltar
GitHub user xoltar opened a pull request: https://github.com/apache/incubator-spark/pull/638 For outputformats that are Configurable, call setConf before sending data to them. This allows us to use, e.g. HBase's TableOutputFormat with PairRDDFunctions.saveAsNewAPIHadoopFile, which

[GitHub] incubator-spark pull request: SPARK-1084 (part 1). Fix most build ...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/637#issuecomment-35850735 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top

[GitHub] incubator-spark pull request: SPARK-1084 (part 1). Fix most build ...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/637#issuecomment-35850736 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12821/ --- If you

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/638#issuecomment-35850754 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To

[GitHub] incubator-spark pull request: Spark-615: make mapPartitionsWithInd...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/606#issuecomment-35850760 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-post yo

[GitHub] incubator-spark pull request: Spark-615: make mapPartitionsWithInd...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/606#issuecomment-35850759 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-post

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/incubator-spark/pull/638#issuecomment-35851045 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/638#discussion_r9979453 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -617,6 +617,10 @@ class PairRDDFunctions[K: ClassTag, V: ClassTag](

[GitHub] incubator-spark pull request: [SPARK-1100] prevent Spark from over...

2014-02-23 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/incubator-spark/pull/626#issuecomment-35851751 I just went through the Spark Streaming document, it seems that it's safe to follow your suggestion @pwendell --- If your project is set up for it, you can r

[GitHub] incubator-spark pull request: [SPARK-1100] prevent Spark from over...

2014-02-23 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/incubator-spark/pull/626#issuecomment-35851996 but why not just preventing users from overwriting the directory, no matter whether there is part-*? --- If your project is set up for it, you can reply to th

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/638#issuecomment-35852729 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please t

[GitHub] incubator-spark pull request: Spark-615: make mapPartitionsWithInd...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/606#issuecomment-35852724 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12822/ --- If you

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/638#issuecomment-35852730 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-

[GitHub] incubator-spark pull request: Spark-615: make mapPartitionsWithInd...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/606#issuecomment-35852723 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-post y

[GitHub] incubator-spark pull request: MLI-2: Start adding k-fold cross val...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/572#issuecomment-35852739 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-

[GitHub] incubator-spark pull request: MLI-2: Start adding k-fold cross val...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/572#issuecomment-35852737 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please t

Re: standard way of running a compiled jar

2014-02-23 Thread Matei Zaharia
Yes, it is a supported option. I’m just wondering whether we want to create a script for it specifically. Maybe the same script could also allow submitting to the cluster or something. Matei On Feb 23, 2014, at 1:55 PM, Sandy Ryza wrote: > Is the client=driver mode still a supported option (o

Re: [DISCUSS] Extending public API

2014-02-23 Thread Matei Zaharia
My sense on all this is that it should be done on a case-by-case basis. To add a new API, it needs to be general enough that a lot of users will want to use it. If adding that API confuses users, that’s a problem. However, on the flip side, if it’s not a super-popular function but it’s just 10-2

[GitHub] incubator-spark pull request: MLI-2: Start adding k-fold cross val...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/572#issuecomment-35854653 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12824/ --- If you

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/638#issuecomment-35854654 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12823/ --- If you

[GitHub] incubator-spark pull request: MLI-2: Start adding k-fold cross val...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/572#issuecomment-35854651 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/638#issuecomment-35854652 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/638#discussion_r9980719 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -617,6 +617,10 @@ class PairRDDFunctions[K: ClassTag, V: ClassTag](s

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/638#discussion_r9980723 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -617,6 +617,10 @@ class PairRDDFunctions[K: ClassTag, V: ClassTag](s

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/638#discussion_r9980734 --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala --- @@ -26,6 +26,11 @@ import com.google.common.io.Files im

[GitHub] incubator-spark pull request: fix building with maven on Mac OS X

2014-02-23 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/incubator-spark/pull/639 fix building with maven on Mac OS X fix building with maven on Mac OS X throw Failure to find org.eclipse.paho:mqtt-client:jar:0.4.0 in https://repository.apache.org/content/repositories/rel

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/638#discussion_r9980747 --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala --- @@ -330,4 +335,74 @@ class PairRDDFunctionsSuite extends FunSuite

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/638#discussion_r9980751 --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala --- @@ -330,4 +335,74 @@ class PairRDDFunctionsSuite extends FunSuite

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/incubator-spark/pull/638#issuecomment-35856419 Thanks a lot for tracking this down, fixing it, and adding tests! I added some minor style comments, modulo those comments LGTM. --- If your project is set up

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/incubator-spark/pull/638#issuecomment-35856428 We should put this fix in 0.9 as well once it's ready to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] incubator-spark pull request: fix building with maven on Mac OS X

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/639#issuecomment-35856579 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To

[GitHub] incubator-spark pull request: For outputformats that are Configura...

2014-02-23 Thread xoltar
Github user xoltar commented on the pull request: https://github.com/apache/incubator-spark/pull/638#issuecomment-35857946 Thanks, last change should address all code review comments. Also cleaned up some imports in PairRDDFunctionsSuite that weren't needed. --- If your project is se

[GitHub] incubator-spark pull request: [SPARK-1089] fix the regression prob...

2014-02-23 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/incubator-spark/pull/614#issuecomment-35859704 Nice catch ! and thanks for taking the time to dig this. I am okay with this way of doing it, however if you and others prefer we can move this code to create

[GitHub] incubator-spark pull request: [SPARK-1089] fix the regression prob...

2014-02-23 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/incubator-spark/pull/614#issuecomment-35862636 Also from this I just went ahead and tried fixing this problem in scala and it worked. https://github.com/ScrapCodes/scala/tree/si-6502-fix --- If your p

[GitHub] incubator-spark pull request: SPARK-1004: PySpark on YARN

2014-02-23 Thread sryza
GitHub user sryza opened a pull request: https://github.com/apache/incubator-spark/pull/640 SPARK-1004: PySpark on YARN Make pyspark work in yarn-client mode. This build's on Josh's work. I tested verified it works on a 5-node cluster. You can merge this pull request into a Git r

[GitHub] incubator-spark pull request: SPARK-1004: PySpark on YARN

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/640#issuecomment-35863413 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please t

[GitHub] incubator-spark pull request: SPARK-1004: PySpark on YARN

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/640#issuecomment-35863414 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-

[GitHub] incubator-spark pull request: [SPARK-1102] Create a saveAsNewAPIHa...

2014-02-23 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/incubator-spark/pull/636#issuecomment-35863485 Jenkins, this is OK to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-p

[GitHub] incubator-spark pull request: [SPARK-1102] Create a saveAsNewAPIHa...

2014-02-23 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/incubator-spark/pull/636#discussion_r9983025 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -686,6 +649,47 @@ class PairRDDFunctions[K: ClassTag, V: ClassTag](sel

[GitHub] incubator-spark pull request: add threadPool shutdown hook when ki...

2014-02-23 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/incubator-spark/pull/628#issuecomment-35863650 Given this, can you close the pull request? Or do you plan to try interrupt? That may also not fix the issue. --- If your project is set up for it, you can reply

[GitHub] incubator-spark pull request: add threadPool shutdown hook when ki...

2014-02-23 Thread wchswchs
Github user wchswchs closed the pull request at: https://github.com/apache/incubator-spark/pull/628 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-post your response. If your project does not have this feat

[GitHub] incubator-spark pull request: add threadPool shutdown hook when ki...

2014-02-23 Thread wchswchs
Github user wchswchs commented on the pull request: https://github.com/apache/incubator-spark/pull/628#issuecomment-35863734 ok,i have closed it! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-po

[GitHub] incubator-spark pull request: SPARK-1124: Fix infinite retries of ...

2014-02-23 Thread mateiz
GitHub user mateiz opened a pull request: https://github.com/apache/incubator-spark/pull/641 SPARK-1124: Fix infinite retries of reduce stage when a map stage failed In the previous code, if you had a failing map stage and then tried to run reduce stages on it repeatedly, the first

[GitHub] incubator-spark pull request: SPARK-1124: Fix infinite retries of ...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/641#issuecomment-35864121 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please top-

[GitHub] incubator-spark pull request: SPARK-1124: Fix infinite retries of ...

2014-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/641#issuecomment-35864120 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. To do so, please t

  1   2   >