[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/819 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43564151 Great! Thanks! Merging it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43564080 @tdas sure, go for it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43562033 @mateiz Is this good to merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43550903 I tested this again on Windows building with and without hive and I can verify that it works as we expect. I think this is ready to go. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12808084 --- Diff: bin/compute-classpath.cmd --- @@ -20,6 +20,13 @@ rem rem This script computes Spark's classpath and prints it to stdout; it's used by both the "run" rem script and the ExecutorRunner in standalone cluster mode. +rem If we're called from spark-class2.cmd, it already set enabledelayedexpansion and setting +rem it here would stop us from affecting its copy of the CLASSPATH variable; otherwise we +rem need to set it here because we use !datanucleus_jars! below. +if "%DONT_PRINT_CLASSPATH%"=="1" goto skip_delayed_expansion +setlocal enabledelayedexpansion +:skip_delayed_expansion + --- End diff -- Also, if the person runs `spark-class` and then run `compute-classpath` directly, it will skip delayed expansion when it shouldn't, because windows env vars are global and persistent. Maybe we should do `set DONT_PRINT_CLASSPATH=` here --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12807150 --- Diff: bin/compute-classpath.cmd --- @@ -20,6 +20,13 @@ rem rem This script computes Spark's classpath and prints it to stdout; it's used by both the "run" rem script and the ExecutorRunner in standalone cluster mode. +rem If we're called from spark-class2.cmd, it already set enabledelayedexpansion and setting +rem it here would stop us from affecting its copy of the CLASSPATH variable; otherwise we +rem need to set it here because we use !datanucleus_jars! below. +if "%DONT_PRINT_CLASSPATH%"=="1" goto skip_delayed_expansion --- End diff -- Where do you set this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43478310 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43478312 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15078/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43477471 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43477472 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15077/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43475544 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43475552 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43474576 Thanks for the review, @andrewor14! I think I've dealt with all the comments (modulo a few I replied to above). The enabledelayedexpansion thing was very weird; if you call compute-classpath from spark-shell2, you should *not* set it again in compute-classpath, otherwise its changes to variables will not propagate out. But if you run compute-classpath by itself in a new process (as we do to launch executors), you should set it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43474514 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43474498 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12782348 --- Diff: README.md --- @@ -41,9 +42,9 @@ And run the following command, which should also return 1000: Spark also comes with several sample programs in the `examples` directory. To run one of them, use `./bin/run-example [params]`. For example: -./bin/run-example org.apache.spark.examples.SparkLR +./bin/run-example SparkPi --- End diff -- I think this is pretty self-evident, I'd rather leave it out --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/819#discussion_r12780712
--- Diff: bin/compute-classpath.cmd ---
@@ -42,6 +42,21 @@ if exist "%FWDIR%RELEASE" (
set CLASSPATH=%CLASSPATH%;%ASSEMBLY_JAR%
+rem When Hive support is needed, Datanucleus jars must be included on the
classpath.
+rem Datanucleus jars do not work if only included in the uber jar as
plugin.xml metadata is lost.
+rem Both sbt and maven will populate "lib_managed/jars/" with the
datanucleus jars when Spark is
+rem built with Hive, so look for them there.
+if exist "%FWDIR%RELEASE" (
+ set datanucleus_dir=%FWDIR%\lib
+) else (
+ set datanucleus_dir=%FWDIR%\lib_managed\jars
+)
+set "datanucleus_jars="
+for %%d in ("%datanucleus_dir%\datanucleus-*.jar") do (
+ set datanucleus_jars=!datanucleus_jars!;%%d
--- End diff --
Ah, how did you try it, you just ran compute-classpath? I set this in
spark-class.cmd but not compute-classpath but I guess both need to work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12780685 --- Diff: bin/run-example --- @@ -55,5 +54,5 @@ fi ./bin/spark-submit \ --master $EXAMPLE_MASTER \ --- End diff -- With run-example you can't pass args to spark-submit, only to the example. So I think it's fine as is. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12780359 --- Diff: bin/compute-classpath.cmd --- @@ -42,6 +42,21 @@ if exist "%FWDIR%RELEASE" ( set CLASSPATH=%CLASSPATH%;%ASSEMBLY_JAR% +rem When Hive support is needed, Datanucleus jars must be included on the classpath. +rem Datanucleus jars do not work if only included in the uber jar as plugin.xml metadata is lost. +rem Both sbt and maven will populate "lib_managed/jars/" with the datanucleus jars when Spark is +rem built with Hive, so look for them there. +if exist "%FWDIR%RELEASE" ( + set datanucleus_dir=%FWDIR%\lib +) else ( + set datanucleus_dir=%FWDIR%\lib_managed\jars --- End diff -- This leads to two backslashes because `%FWDIR%` already has one backslash. This should just be `%FWDIR%lib_managed\jars` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/819#discussion_r12780348
--- Diff: bin/compute-classpath.cmd ---
@@ -42,6 +42,21 @@ if exist "%FWDIR%RELEASE" (
set CLASSPATH=%CLASSPATH%;%ASSEMBLY_JAR%
+rem When Hive support is needed, Datanucleus jars must be included on the
classpath.
+rem Datanucleus jars do not work if only included in the uber jar as
plugin.xml metadata is lost.
+rem Both sbt and maven will populate "lib_managed/jars/" with the
datanucleus jars when Spark is
+rem built with Hive, so look for them there.
+if exist "%FWDIR%RELEASE" (
+ set datanucleus_dir=%FWDIR%\lib
+) else (
+ set datanucleus_dir=%FWDIR%\lib_managed\jars
+)
+set "datanucleus_jars="
+for %%d in ("%datanucleus_dir%\datanucleus-*.jar") do (
+ set datanucleus_jars=!datanucleus_jars!;%%d
--- End diff --
Hey @mateiz I just tried this on windows 7 and my class path includes the
string "!datanucleus_jars!". It should probably be "%datanucleus_jars%" instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43468276 My worry with Windows is people downloading pre-built Spark and getting a bizarre behavior. I'm assuming most people will work with pre-built Spark (since you'd mostly use Windows for local development) so those who build it by hand can handle a bit more complexity. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43468247 Yes, I don't want to rely on "jar" being installed. It's not installed by default when you grab a JRE (as far as I can tell). I'd like to eventually do that on Unix too but it's okay to do it in a later release. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43467187 Did you mean to also remove the hive check here https://github.com/apache/spark/blob/master/bin/compute-classpath.sh#L93? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12779005 --- Diff: README.md --- @@ -41,9 +42,9 @@ And run the following command, which should also return 1000: Spark also comes with several sample programs in the `examples` directory. To run one of them, use `./bin/run-example [params]`. For example: -./bin/run-example org.apache.spark.examples.SparkLR +./bin/run-example SparkPi --- End diff -- Maybe add a param (e.g. `10`) to show the `[params]` part? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12778956 --- Diff: bin/run-example2.cmd --- @@ -30,7 +30,9 @@ if exist "%FWDIR%conf\spark-env.cmd" call "%FWDIR%conf\spark-env.cmd" rem Test that an argument was given if not "x%1"=="x" goto arg_given - echo Usage: run-example ^ [^] + echo Usage: run-example ^ [^] + echo - set MASTER=XX to use a specific master --- End diff -- Same here --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12778951 --- Diff: bin/run-example2.cmd --- @@ -47,15 +55,34 @@ if "x%SPARK_EXAMPLES_JAR%"=="x" ( goto exit ) -rem Compute Spark classpath using external script -set DONT_PRINT_CLASSPATH=1 -call "%FWDIR%bin\compute-classpath.cmd" -set DONT_PRINT_CLASSPATH=0 -set CLASSPATH=%SPARK_EXAMPLES_JAR%;%CLASSPATH% +rem Set master from MASTER environment variable if given +if "x%MASTER%"=="x" ( + set EXAMPLE_MASTER=local[*] +) else ( + set EXAMPLE_MASTER=%MASTER% +) + +rem If the EXAMPLE_CLASS does not start with org.apache.spark.examples, add that +set EXAMPLE_CLASS=%1 +set PREFIX=%EXAMPLE_CLASS:~0,25% +if not %PREFIX%==org.apache.spark.examples ( + set EXAMPLE_CLASS=org.apache.spark.examples.%EXAMPLE_CLASS% +) + +rem Get the tail of the argument list, to skip the first one. This is surprisingly +rem complicated on Windows. +set "ARGS=" +:top +shift +if "%~1" neq "" ( + set ARGS=%ARGS% "%~1" + goto :top +) +if defined ARGS set ARGS=%ARGS:~1% -rem Figure out where java is. -set RUNNER=java -if not "x%JAVA_HOME%"=="x" set RUNNER=%JAVA_HOME%\bin\java +call "%FWDIR%bin\spark-submit.cmd" ^ + --master %EXAMPLE_MASTER% ^ --- End diff -- Same here --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12778945 --- Diff: bin/run-example --- @@ -55,5 +54,5 @@ fi ./bin/spark-submit \ --master $EXAMPLE_MASTER \ --- End diff -- Not your change, but I think we should drop this; right now if the user passes in `--master` it doesn't take effect. Also, relying on `MASTER` is inconsistent with the other scripts (pyspark, spark-shell, spark-submit) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12778839 --- Diff: bin/run-example2.cmd --- @@ -30,7 +30,9 @@ if exist "%FWDIR%conf\spark-env.cmd" call "%FWDIR%conf\spark-env.cmd" rem Test that an argument was given if not "x%1"=="x" goto arg_given - echo Usage: run-example ^ [^] + echo Usage: run-example ^ [^] --- End diff -- small nit: it's not clear what `[]` means. How about just `[example-args]` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12778836 --- Diff: bin/run-example2.cmd --- @@ -30,7 +30,9 @@ if exist "%FWDIR%conf\spark-env.cmd" call "%FWDIR%conf\spark-env.cmd" rem Test that an argument was given if not "x%1"=="x" goto arg_given - echo Usage: run-example ^ [^] + echo Usage: run-example ^ [^] + echo - set MASTER=XX to use a specific master --- End diff -- Same here --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12778833 --- Diff: bin/run-example --- @@ -23,6 +23,16 @@ FWDIR="$(cd `dirname $0`/..; pwd)" export SPARK_HOME="$FWDIR" EXAMPLES_DIR="$FWDIR"/examples +if [ -n "$1" ]; then + EXAMPLE_CLASS="$1" + shift +else + echo "Usage: ./bin/run-example [example-args]" + echo " - set MASTER=XX to use a specific master" --- End diff -- Ah, I guess that was already there before --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/819#discussion_r12778830 --- Diff: bin/run-example --- @@ -23,6 +23,16 @@ FWDIR="$(cd `dirname $0`/..; pwd)" export SPARK_HOME="$FWDIR" EXAMPLES_DIR="$FWDIR"/examples +if [ -n "$1" ]; then + EXAMPLE_CLASS="$1" + shift +else + echo "Usage: ./bin/run-example [example-args]" + echo " - set MASTER=XX to use a specific master" --- End diff -- Isn't this deprecated? I thought in general we want people to use `--master`, since this goes through Spark submit --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43459319 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43459321 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15072/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43458284 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/819#issuecomment-43458281 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1876] Windows fixes to deal with latest...
GitHub user mateiz opened a pull request: https://github.com/apache/spark/pull/819 [SPARK-1876] Windows fixes to deal with latest distribution layout changes - Look for JARs in the right place - Launch examples the same way as on Unix - Load datanucleus JARs if they exist - Don't attempt to parse local paths as URIs in SparkSubmit, since paths with C:\ are not valid URIs - Also fixed POM exclusion rules for datanucleus (it wasn't properly excluding it, whereas SBT was) You can merge this pull request into a Git repository by running: $ git pull https://github.com/mateiz/spark win-fixes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/819.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #819 commit 144af84f727cb11befc11723152e65e5e84cbb16 Author: Matei Zaharia Date: 2014-05-18T23:44:26Z Update Windows scripts to match latest binary package layout Also fixed an issues where SparkSubmit was trying to parse local files as URLs, which fails on Windows because they contain backslashes. We didn't need to treat those as URLs to check if a file exists. commit d3b71c7611e5d52519fecb4e9268b0ec362ad65d Author: Matei Zaharia Date: 2014-05-19T00:16:40Z Properly exclude datanucleus files in Maven assembly They are excluded in SBT, but the rule added in Maven didn't actually remove the files from the JAR. The JARs built still worked despite this, but it's better to remove them than have 2 copies on the classpath. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
