Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

2014-02-25 Thread Evan chan
Actually you can control exactly how sbt assembly merges or resolves conflicts. I believe the default settings however lead to order which cannot be controlled. I do wish for a smarter fat jar plugin. -Evan To be free is not merely to cast off one's chains, but to live in a way that

Re: SPARK-942 patch review

2014-02-25 Thread Andrew Ash
I've always felt that the Spark team was extremely responsive to PRs and I've been very impressed over the past year with your output. As Matei said, probably the best thing to do here is to be more diligent about closing PRs that are old/abandoned so that every PR is active. Whenever I comment

Discussion on SPARK-1139

2014-02-26 Thread Nan Zhu
Hi, all I just created a JIRA https://spark-project.atlassian.net/browse/SPARK-1139 . The issue discusses that: the new Hadoop API based Spark APIs are actually a mixture of old and new Hadoop API. Spark APIs are still using JobConf (or Configuration) as one of the parameters, but actually

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

2014-02-26 Thread Koert Kuipers
We maintain in house spark build using sbt. We have no problem using sbt assembly. We did add a few exclude statements for transitive dependencies. The main enemy of assemblies are jars that include stuff they shouldn't (kryo comes to mind, I think they include logback?), new versions of jars

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

2014-02-26 Thread Patrick Wendell
@mridul - As far as I know both Maven and Sbt use fairly similar processes for building the assembly/uber jar. We actually used to package spark with sbt and there were no specific issues we encountered and AFAIK sbt respects versioning of transitive dependencies correctly. Do you have a specific

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

2014-02-26 Thread Sandy Ryza
@patrick - It seems like my point about being able to inherit the root pom was addressed and there's a way to handle this. The larger point I meant to make is that Maven is by far the most common build tool in projects that are likely to share contributors with Spark. I personally know 10 people

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

2014-02-26 Thread Koert Kuipers
i dont buy the argument that we should use it because its the most common. if all we would do is use what is most common then we should switch to java, svn and maven On Wed, Feb 26, 2014 at 1:38 PM, Mark Grover grover.markgro...@gmail.comwrote: Hi Patrick, And, to pile on what Sandy said.

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

2014-02-26 Thread Evan Chan
Mark, No, I haven't tried this myself yet :-p Also I would expect that sbt-pom-reader does not do assemblies at all because that is an SBT plugin, so we would still need code to include sbt-assembly. There is also the trick question of how to include the assembly stuff into sbt-pom-reader

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

2014-02-26 Thread Mark Hamstra
Yes, but the POM generated in that fashion is only sufficient for linking with Spark, not for building Spark or serving as a basis from which to build a customized Spark with Maven. So, starting from SparkBuild.scala and generating a POM with make-pom, those who wish to build a customized Spark

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

2014-02-26 Thread Evan Chan
Can't maven pom's include other ones? So what if we remove the artifact specs from the main pom, have them generated by sbt make-pom, and include the generated file in the main pom.xml?I guess, just trying to figure out how much this would help (it seems at least it would remove the issue of

Re: [IMPORTANT] Github/jenkins migration

2014-02-26 Thread Koert Kuipers
github is not aware of the new repo being a base-fork, so its not easy to re-point pull requests. i am guessing it didnt get cloned from the incubator spark one? On Wed, Feb 26, 2014 at 5:56 PM, Patrick Wendell pwend...@gmail.com wrote: Sorry if this wasn't clear - If you are in the middle of

[GitHub] spark pull request: SPARK-1121 Only add avro if the build is for H...

2014-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6#issuecomment-36212897 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

Re: How to run a single test suite?

2014-02-26 Thread Reynold Xin
You put your quotes in the wrong place. See https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools On Wed, Feb 26, 2014 at 10:04 PM, Bryn Keller xol...@xoltar.org wrote: Hi Folks, I've tried using sbt test-only '*PairRDDFunctionsSuite' to run only that test suite, which

Re: How to run a single test suite?

2014-02-26 Thread Bryn Keller
Thanks, that was it! On Wed, Feb 26, 2014 at 10:05 PM, Reynold Xin r...@databricks.com wrote: You put your quotes in the wrong place. See https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools On Wed, Feb 26, 2014 at 10:04 PM, Bryn Keller xol...@xoltar.org wrote: Hi

[GitHub] spark pull request: SPARK-1121 Only add avro if the build is for H...

2014-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6#issuecomment-36214461 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: Show Master status on UI page

2014-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/24#issuecomment-36215905 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Updated more links in documentation

2014-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/23#issuecomment-36216059 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1121 Only add avro if the build is for H...

2014-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6#issuecomment-36216313 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1121 Only add avro if the build is for H...

2014-02-26 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/6#issuecomment-36217654 Rebased !! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-1089] fix the regression problem on ADD...

2014-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13#issuecomment-36218102 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1089] fix the regression problem on ADD...

2014-02-26 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/13#issuecomment-36218095 Thanks I've merged this into master and 0.9. Actually didn't notice tests hadn't gone through. We can revert if there are any issues. Jenkins, test this please. --- If

[GitHub] spark pull request: SPARK-1125: When using a http proxy,the maven ...

2014-02-26 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/25#issuecomment-36218375 Perhaps no one encountered the same problem.Well, let me close this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: SPARK-1125: When using a http proxy,the maven ...

2014-02-26 Thread witgo
Github user witgo closed the pull request at: https://github.com/apache/spark/pull/25 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-1100] prevent Spark from overwriting di...

2014-02-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/11#discussion_r10114841 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -618,10 +619,6 @@ class PairRDDFunctions[K: ClassTag, V: ClassTag](self:

[GitHub] spark pull request: [SPARK-1100] prevent Spark from overwriting di...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11#issuecomment-36218942 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Vagrant support for Spark

2014-02-27 Thread ngbinh
GitHub user ngbinh opened a pull request: https://github.com/apache/spark/pull/26 Vagrant support for Spark This PR uses Vagrant to create a clusters of three VMs, one master and two workers. It allows running/testing Spark Cluster mode on one machine. My initial goal is

[GitHub] spark pull request: [SPARK-979] Randomize order of offers.

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/27#issuecomment-36224189 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1146] Vagrant support for Spark

2014-02-27 Thread jyotiska
Github user jyotiska commented on the pull request: https://github.com/apache/spark/pull/26#issuecomment-36225645 +1 It will be better if these projects were made separate from core-spark project and grown as independent projects. This keeps the core project lean and helps to grow

[GitHub] spark pull request: [SPARK-1146] Vagrant support for Spark

2014-02-27 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/26#issuecomment-36225852 FWIW I agree. The tendency is almost always to include a bunch of modules that are really separate, slightly-downstream projects. You could make similar arguments for even

[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36226279 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1004. PySpark on YARN

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/30#issuecomment-36226245 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1146] Vagrant support for Spark

2014-02-27 Thread ngbinh
Github user ngbinh commented on the pull request: https://github.com/apache/spark/pull/26#issuecomment-36226895 One of the main reason I work on this PR is I found many times working on Spark local mode doesn't expose problems when deployed on a cluster. This PR should allow Spark

[GitHub] spark pull request: SPARK 1084.1 (resubmitted)

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/31#issuecomment-36234496 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-1084.2 (resubmitted)

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/32#issuecomment-36234483 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: Added a unit test for PairRDDFunctions.lookup

2014-02-27 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/36#issuecomment-36278409 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Added a unit test for PairRDDFunctions.lookup

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/36#issuecomment-36278643 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK 1084.1 (resubmitted)

2014-02-27 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/31#issuecomment-36278982 Looks good, merging this into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10140917 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10141220 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10142402 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -135,6 +135,8 @@ class SparkContext( val isLocal = (master == local ||

[GitHub] spark pull request: [HOTFIX] Patching maven build after #6 (SPARK-...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/37#issuecomment-36294696 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10143842 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10144358 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/ui/WorkerWebUI.scala --- @@ -198,6 +199,6 @@ class WorkerWebUI(val worker: Worker, val workDir:

[GitHub] spark pull request: [HOTFIX] Patching maven build after #6 (SPARK-...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/37#issuecomment-36297711 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12921/ --- If your project

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10144869 --- Diff: docs/configuration.md --- @@ -477,6 +505,21 @@ Apart from these, the following properties are also available, and may be useful td

[GitHub] spark pull request: SPARK-1134

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/38#issuecomment-36301181 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: SPARK-1134

2014-02-27 Thread dianacarroll
GitHub user dianacarroll opened a pull request: https://github.com/apache/spark/pull/38 SPARK-1134 bug with ipython: startup scripts are called in both interactive and non-interactive mode. This prevents non-interactive use with spark because the spark startup script and user

[GitHub] spark pull request: SPARK-1134

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/38#issuecomment-36301198 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [HOTFIX] Patching maven build after #6 (SPARK-...

2014-02-27 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/37#issuecomment-36303359 Merged into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread kayousterhout
GitHub user kayousterhout opened a pull request: https://github.com/apache/spark/pull/39 Remote BlockFetchTracker trait This trait seems to have been created a while ago when there were multiple implementations; now that there's just one, I think it makes sense to merge it

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/39#issuecomment-36305800 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36306177 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-02-27 Thread ngbinh
GitHub user ngbinh opened a pull request: https://github.com/apache/spark/pull/41 Update io.netty from 4.0.13 Final to 4.0.17.Final This update contains a lot of bug fixes and some new perf improvements. It is also binary compatible with the current 4.0.13.Final For

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/41#issuecomment-36306872 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/39#issuecomment-36307824 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12922/ --- If your project

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/41#issuecomment-36308969 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12923/ --- If your project

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10150320 --- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala --- @@ -41,56 +46,103 @@ private[spark] object JettyUtils extends Logging { type

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10150334 --- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala --- @@ -41,56 +46,103 @@ private[spark] object JettyUtils extends Logging { type

[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...

2014-02-27 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36311392 Oops, actually pushed it this time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36313766 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36313763 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36313840 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12925/ --- If your

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36314536 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10152124 --- Diff: docs/security.md --- @@ -0,0 +1,14 @@ +--- +layout: global +title: Spark Security +--- + +Spark currently supports

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/33#issuecomment-36315155 Hey Tom, This patch is looking really great. I took a pretty thorough look through this and there were really only two high level things: 1.

Re: Discussion on SPARK-1139

2014-02-27 Thread ligq
You can make the patch so that everyone review On Wednesday, February 26, 2014 at 8:23 AM, Nan Zhu wrote: Hi, all I just created a JIRA https://spark-project.atlassian.net/browse/SPARK-1139 . The issue discusses that: the new Hadoop API based Spark APIs are actually a mixture of

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/39#discussion_r10152243 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -44,9 +44,14 @@ import org.apache.spark.util.Utils */

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/39#discussion_r10152493 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -233,7 +238,16 @@ object BlockFetcherIterator {

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36315929 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36315930 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12926/ --- If your

[GitHub] spark pull request: [SPARK-1102] Create a saveAsNewAPIHadoopDatase...

2014-02-27 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/12#issuecomment-36321849 this is a re-opened PR, in the old PR, https://github.com/apache/incubator-spark/pull/636, all test cases have passed Can anyone verify that and make further

[GitHub] spark pull request: Removed reference to incubation in Spark user ...

2014-02-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

How best to ensure partitioners behave?

2014-02-27 Thread Bryn Keller
Hi Folks, I just filed https://spark-project.atlassian.net/browse/SPARK-1149 - I'm happy to fix it, but I'd like input about how best to go about it. The problem is, if a partitioner misbehaves by, say, returning a negative partition number, Spark hangs. This is easier to do than it sounds. I'd

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/39#issuecomment-36323841 Thanks @pwendell -- fixed the style! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/39 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/43#issuecomment-36326074 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-27 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/43#discussion_r10156300 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -146,6 +146,12 @@ object BlockFetcherIterator { }

[GitHub] spark pull request: SPARK-1134 pyspark only uses ipython if starte...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/38#issuecomment-36326995 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12928/ --- If your project

[GitHub] spark pull request: SPARK-1134 pyspark only uses ipython if starte...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/38#issuecomment-36326993 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/43#issuecomment-36327168 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/43#issuecomment-36327170 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12929/ --- If your

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-02-28 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/44 fix #SPARK-1149 Bad partitioners can cause Spark to hang You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark SPARK-1149 Alternatively you

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/44#issuecomment-36332248 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: remove sbt/ prefix in $JAR variable make the...

2014-02-28 Thread coderh
GitHub user coderh opened a pull request: https://github.com/apache/spark/pull/46 remove sbt/ prefix in $JAR variable make the script pass if test There is no sbt/ dir under sbt/sbt. if test on line 29 can not pass. remove sbt/ prefix in $JAR variable make the script pass

[GitHub] spark pull request: remove sbt/ prefix in $JAR variable make the...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/46#issuecomment-36338520 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: remove sbt/ prefix in $JAR variable make the...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/46#issuecomment-36338504 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: SPARK-1121 Only add avro if the build is for H...

2014-02-28 Thread lfrancke
Github user lfrancke commented on the pull request: https://github.com/apache/spark/pull/6#issuecomment-36343187 The Avro dependency is still in the yarn/pom.xml file and causes Maven to fail early. I'm happy to submit a pull request/issue but maybe you want to do another Hotfix

YARN Maven build questions

2014-02-28 Thread Lars Francke
Hey, I'm trying to dig into Spark's code but am running into a couple of problems. 1) The yarn-common directory is not included in the Maven build causing things to fail because the dependency is missing. If I see the history correct it used to be a Maven module but is not anymore. 2) When I

[GitHub] spark pull request: remove sbt/ prefix in $JAR variable make the...

2014-02-28 Thread coderh
Github user coderh closed the pull request at: https://github.com/apache/spark/pull/46 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-02-28 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/44#discussion_r10164732 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -847,6 +847,8 @@ class SparkContext( partitions: Seq[Int],

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-02-28 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/44#discussion_r10166854 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -847,6 +847,8 @@ class SparkContext( partitions: Seq[Int],

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-02-28 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/44#discussion_r10166948 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -847,6 +847,8 @@ class SparkContext( partitions: Seq[Int],

[GitHub] spark pull request: SPARK-1032. If Yarn app fails before registeri...

2014-02-28 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/28#issuecomment-36362791 I committed this. Thanks Sandy! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Update dev merge script to use spark.git inste...

2014-02-28 Thread tgravescs
GitHub user tgravescs opened a pull request: https://github.com/apache/spark/pull/47 Update dev merge script to use spark.git instead of incubator-spark You can merge this pull request into a Git repository by running: $ git pull https://github.com/tgravescs/spark

[GitHub] spark pull request: Update dev merge script to use spark.git inste...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/47#issuecomment-36363038 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Update dev merge script to use spark.git inste...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/47#issuecomment-36363010 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: fix repo location in create script

2014-02-28 Thread CodingCat
GitHub user CodingCat opened a pull request: https://github.com/apache/spark/pull/48 fix repo location in create script fix the repo location in create_release script You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: [SPARK-1150] fix repo location in create scrip...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/48#issuecomment-36364625 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Update dev merge script to use spark.git inste...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/47#issuecomment-36366007 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12930/ --- If your project

[GitHub] spark pull request: Update dev merge script to use spark.git inste...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/47#issuecomment-36366006 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

  1   2   3   4   5   6   7   8   9   10   >