GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/1845
[SPARK-2849 / 2914] Handle certain Spark configs in bash correctly
We currently rely on the Java properties file to parse
`spark-defaults.conf` file. However, certain Spark configs need to be pro
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-51556022
QA tests have started for PR 1845. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18169/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-51558467
QA results for PR 1845:- This patch PASSES unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-51565448
Given the complexity of the bash scripts, can we make sure we have some
tests for this?
---
If your project is set up for it, you can reply to this email and have your
repl
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-51567374
Tests sound like a great idea.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-51674448
QA tests have started for PR 1845. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18239/consoleFull
---
If
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-51679922
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fea
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-51680007
QA tests have started for PR 1845. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18247/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-51680769
QA results for PR 1845:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-51695612
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this f
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-51695688
QA tests have started for PR 1845. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18258/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-51696855
QA results for PR 1845:- This patch PASSES unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16280454
--- Diff: bin/utils.sh ---
@@ -0,0 +1,108 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16280819
--- Diff: bin/spark-class ---
@@ -101,11 +106,16 @@ fi
# Set JAVA_OPTS to be able to load native libraries and to set heap size
JAVA_OPTS="-XX:MaxPe
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16281080
--- Diff: bin/utils.sh ---
@@ -0,0 +1,108 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52276624
Hey @andrewor14 it's great to see this being fixed. I think things can be
simplified a lot by deferring to the `eval` from bash to deal with the argumnet
parsing. It's w
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52383086
test this please!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fe
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16322977
--- Diff: bin/utils.sh ---
@@ -0,0 +1,108 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52383274
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18656/consoleFull)
for PR 1845 at commit
[`7a4190a`](https://github.com/ap
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52383497
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18657/consoleFull)
for PR 1845 at commit
[`c886568`](https://github.com/ap
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52384351
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18656/consoleFull)
for PR 1845 at commit
[`7a4190a`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52384399
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18657/consoleFull)
for PR 1845 at commit
[`c886568`](https://github.com/a
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52384612
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fea
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52384652
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18661/consoleFull)
for PR 1845 at commit
[`c886568`](https://github.com/ap
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52385431
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18661/consoleFull)
for PR 1845 at commit
[`c886568`](https://github.com/a
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16323472
--- Diff: bin/utils.sh ---
@@ -17,9 +16,53 @@
# limitations under the License.
#
+# *
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16323564
--- Diff: bin/utils.sh ---
@@ -17,9 +16,53 @@
# limitations under the License.
#
+# * *
Github user kanzhang commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16330493
--- Diff: bin/utils.sh ---
@@ -17,9 +16,53 @@
# limitations under the License.
#
+# *
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52557810
Working on a new version of this that involves launching two JVMs. Hold off
reviewing for now and stay tuned for a revised version.
---
If your project is set up for
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52582819
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18805/consoleFull)
for PR 1845 at commit
[`b71f52b`](https://github.com/ap
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52583116
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18807/consoleFull)
for PR 1845 at commit
[`c84f5c8`](https://github.com/ap
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52585848
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18805/consoleFull)
for PR 1845 at commit
[`b71f52b`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52586241
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18807/consoleFull)
for PR 1845 at commit
[`c84f5c8`](https://github.com/a
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16395985
--- Diff: bin/spark-class ---
@@ -101,11 +103,12 @@ fi
# Set JAVA_OPTS to be able to load native libraries and to set heap size
JAVA_OPTS="-XX:MaxPe
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16396112
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkClassLauncher.scala ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundati
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16396219
--- Diff: bin/spark-submit ---
@@ -36,9 +38,27 @@ while (($#)); do
done
DEPLOY_MODE=${DEPLOY_MODE:-"client"}
+DEFAULT_PROPERTIES_FILE="$SP
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16396854
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkClassLauncher.scala ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundati
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16397290
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkClassLauncher.scala ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundati
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16397316
--- Diff: bin/spark-class ---
@@ -101,11 +103,12 @@ fi
# Set JAVA_OPTS to be able to load native libraries and to set heap size
JAVA_OPTS="-XX:Max
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16397299
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkClassLauncher.scala ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Founda
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16397360
--- Diff: bin/spark-class ---
@@ -146,10 +149,28 @@ if $cygwin; then
fi
export CLASSPATH
-if [ "$SPARK_PRINT_LAUNCH_COMMAND" == "1" ]; the
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16397452
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1421,3 +1421,24 @@ private[spark] object Utils extends Logging {
}
}
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16397499
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkClassLauncher.scala ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundati
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16397548
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkClassLauncher.scala ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundati
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16397578
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkClassLauncher.scala ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundati
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52591994
Hey Andrew I took a pass. Overall I think this is the best approach we can
do right now (using a bootstrap java process). I felt like there were several
attempts to over
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52592106
Also one other thing - if you wanted to name that script
`DriverBootstrapLauncher` or something it might be better than
`SparkClassLauncher`. It's quite specific to the
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16403594
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1421,3 +1421,24 @@ private[spark] object Utils extends Logging {
}
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52608216
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18836/consoleFull)
for PR 1845 at commit
[`d6488f9`](https://github.com/ap
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1845#issuecomment-52608223
@pwendell Thanks for your feedback. I think I have addressed all your
comments. In particular, I have fixed the precedence order of the various
properties that were re
50 matches
Mail list logo
