[GitHub] spark pull request: [SPARK-1100] prevent Spark from overwriting di...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11#issuecomment-36218942 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Vagrant support for Spark

2014-02-27 Thread ngbinh
GitHub user ngbinh opened a pull request: https://github.com/apache/spark/pull/26 Vagrant support for Spark This PR uses Vagrant to create a clusters of three VMs, one master and two workers. It allows running/testing Spark Cluster mode on one machine. My initial goal is

[GitHub] spark pull request: [SPARK-979] Randomize order of offers.

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/27#issuecomment-36224189 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1146] Vagrant support for Spark

2014-02-27 Thread jyotiska
Github user jyotiska commented on the pull request: https://github.com/apache/spark/pull/26#issuecomment-36225645 +1 It will be better if these projects were made separate from core-spark project and grown as independent projects. This keeps the core project lean and helps to grow

[GitHub] spark pull request: [SPARK-1146] Vagrant support for Spark

2014-02-27 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/26#issuecomment-36225852 FWIW I agree. The tendency is almost always to include a bunch of modules that are really separate, slightly-downstream projects. You could make similar arguments for even

[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36226279 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1004. PySpark on YARN

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/30#issuecomment-36226245 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1146] Vagrant support for Spark

2014-02-27 Thread ngbinh
Github user ngbinh commented on the pull request: https://github.com/apache/spark/pull/26#issuecomment-36226895 One of the main reason I work on this PR is I found many times working on Spark local mode doesn't expose problems when deployed on a cluster. This PR should allow Spark

[GitHub] spark pull request: SPARK 1084.1 (resubmitted)

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/31#issuecomment-36234496 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-1084.2 (resubmitted)

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/32#issuecomment-36234483 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: Added a unit test for PairRDDFunctions.lookup

2014-02-27 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/36#issuecomment-36278409 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Added a unit test for PairRDDFunctions.lookup

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/36#issuecomment-36278643 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK 1084.1 (resubmitted)

2014-02-27 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/31#issuecomment-36278982 Looks good, merging this into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10140917 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10141220 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10142402 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -135,6 +135,8 @@ class SparkContext( val isLocal = (master == local ||

[GitHub] spark pull request: [HOTFIX] Patching maven build after #6 (SPARK-...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/37#issuecomment-36294696 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10143842 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10144358 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/ui/WorkerWebUI.scala --- @@ -198,6 +199,6 @@ class WorkerWebUI(val worker: Worker, val workDir:

[GitHub] spark pull request: [HOTFIX] Patching maven build after #6 (SPARK-...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/37#issuecomment-36297711 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12921/ --- If your project

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10144869 --- Diff: docs/configuration.md --- @@ -477,6 +505,21 @@ Apart from these, the following properties are also available, and may be useful td

[GitHub] spark pull request: SPARK-1134

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/38#issuecomment-36301181 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: SPARK-1134

2014-02-27 Thread dianacarroll
GitHub user dianacarroll opened a pull request: https://github.com/apache/spark/pull/38 SPARK-1134 bug with ipython: startup scripts are called in both interactive and non-interactive mode. This prevents non-interactive use with spark because the spark startup script and user

[GitHub] spark pull request: SPARK-1134

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/38#issuecomment-36301198 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [HOTFIX] Patching maven build after #6 (SPARK-...

2014-02-27 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/37#issuecomment-36303359 Merged into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread kayousterhout
GitHub user kayousterhout opened a pull request: https://github.com/apache/spark/pull/39 Remote BlockFetchTracker trait This trait seems to have been created a while ago when there were multiple implementations; now that there's just one, I think it makes sense to merge it

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/39#issuecomment-36305800 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36306177 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-02-27 Thread ngbinh
GitHub user ngbinh opened a pull request: https://github.com/apache/spark/pull/41 Update io.netty from 4.0.13 Final to 4.0.17.Final This update contains a lot of bug fixes and some new perf improvements. It is also binary compatible with the current 4.0.13.Final For

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/41#issuecomment-36306872 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/39#issuecomment-36307824 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12922/ --- If your project

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/41#issuecomment-36308969 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12923/ --- If your project

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10150320 --- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala --- @@ -41,56 +46,103 @@ private[spark] object JettyUtils extends Logging { type

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10150334 --- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala --- @@ -41,56 +46,103 @@ private[spark] object JettyUtils extends Logging { type

[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...

2014-02-27 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36311392 Oops, actually pushed it this time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36313766 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36313763 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36313840 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12925/ --- If your

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36314536 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10152124 --- Diff: docs/security.md --- @@ -0,0 +1,14 @@ +--- +layout: global +title: Spark Security +--- + +Spark currently supports

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-02-27 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/33#issuecomment-36315155 Hey Tom, This patch is looking really great. I took a pretty thorough look through this and there were really only two high level things: 1.

Re: Discussion on SPARK-1139

2014-02-27 Thread ligq
You can make the patch so that everyone review On Wednesday, February 26, 2014 at 8:23 AM, Nan Zhu wrote: Hi, all I just created a JIRA https://spark-project.atlassian.net/browse/SPARK-1139 . The issue discusses that: the new Hadoop API based Spark APIs are actually a mixture of

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/39#discussion_r10152243 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -44,9 +44,14 @@ import org.apache.spark.util.Utils */

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/39#discussion_r10152493 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -233,7 +238,16 @@ object BlockFetcherIterator {

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36315929 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36315930 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12926/ --- If your

[GitHub] spark pull request: [SPARK-1102] Create a saveAsNewAPIHadoopDatase...

2014-02-27 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/12#issuecomment-36321849 this is a re-opened PR, in the old PR, https://github.com/apache/incubator-spark/pull/636, all test cases have passed Can anyone verify that and make further

[GitHub] spark pull request: Removed reference to incubation in Spark user ...

2014-02-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

How best to ensure partitioners behave?

2014-02-27 Thread Bryn Keller
Hi Folks, I just filed https://spark-project.atlassian.net/browse/SPARK-1149 - I'm happy to fix it, but I'd like input about how best to go about it. The problem is, if a partitioner misbehaves by, say, returning a negative partition number, Spark hangs. This is easier to do than it sounds. I'd

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/39#issuecomment-36323841 Thanks @pwendell -- fixed the style! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: Remote BlockFetchTracker trait

2014-02-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/39 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/43#issuecomment-36326074 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-27 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/43#discussion_r10156300 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -146,6 +146,12 @@ object BlockFetcherIterator { }

[GitHub] spark pull request: SPARK-1134 pyspark only uses ipython if starte...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/38#issuecomment-36326995 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12928/ --- If your project

[GitHub] spark pull request: SPARK-1134 pyspark only uses ipython if starte...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/38#issuecomment-36326993 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/43#issuecomment-36327168 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/43#issuecomment-36327170 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12929/ --- If your