[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-13 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10553709 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-13 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37502813 @mateiz maybe you could take a pass on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37506962 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37507039 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13155/ --- If your

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37507037 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-13 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37601581 Yeah, workin on it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37611181 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37611180 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37611363 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37611365 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13174/ --- If your

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-12 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37381283 Updated patch takes review comments form @mridulm and @pwendell into account. spark.max.cores is now correctly handled. Jars passed in with --more-jars are not

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37382160 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37382161 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37382223 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13127/ --- If your

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-3738 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-07 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10399046 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-07 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10405655 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-07 Thread hsaputra
Github user hsaputra commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10406142 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37088913 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37088937 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13063/ --- If your

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37088936 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-07 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-37088962 Newest patch includes tests and doc. @pwendell, do you have a link to the addJar patch? If it's definitely going to happen, I'll take out the classloader stuff here. ---

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10359483 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala --- @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-36939213 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-36939354 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-36939470 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13027/ --- If your

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-36939469 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-36945861 I see, regarding the memory part, it sounds like we could do it in bash, but it might be kind of painful. We could do the following: - Look for just the driver memory

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10372142 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala --- @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-36966736 I uploaded a new patch that doesn't start a new JVM and parses --driver-memory in bash. It wasn't as bad as I expected (thanks to some help from @umbrant and @atm).

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-36967209 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-36967210 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10373180 --- Diff: bin/spark-submit --- @@ -0,0 +1,38 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-36967269 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13036/ --- If your

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-36967267 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10373335 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10373471 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10374754 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-06 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10374821 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread sryza
GitHub user sryza opened a pull request: https://github.com/apache/spark/pull/86 SPARK-1126. spark-app preliminary This is a starting version of the spark-app script for running compiled binaries against Spark. It still needs tests and some polish. The only testing I've done so

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10330438 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkAppArguments.scala --- @@ -0,0 +1,155 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10330672 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkAppArguments.scala --- @@ -0,0 +1,155 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10330871 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala --- @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10331089 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala --- @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10331562 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkAppArguments.scala --- @@ -0,0 +1,155 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10332560 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala --- @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10332582 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala --- @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10332644 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala --- @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10332654 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala --- @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10332670 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApp.scala --- @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/86#discussion_r10332688 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkAppArguments.scala --- @@ -0,0 +1,155 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: SPARK-1126. spark-app preliminary

2014-03-05 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/86#issuecomment-36828689 Thanks for taking a look, Matei. If we use system properties instead of env variables, the remaining reason we'd want to start a second JVM is to be able to have a