Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17861#discussion_r114934079
--- Diff: bin/spark-class2.cmd ---
@@ -64,7 +64,7 @@ if not "x%JAVA_HOME%"=="x" (
rem The launcher library prints the command to be executed in a single
line suitable for being
rem executed by the batch interpreter. So read all the output of the
launcher into a variable.
set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt
-"%RUNNER%" -Xmx128m -cp "%LAUNCH_CLASSPATH%"
org.apache.spark.launcher.Main %* > %LAUNCHER_OUTPUT%
--- End diff --
I found this problem when I tested some cases on Windows before but I just
thought it was my wrong environmental setup.
I think we should change `set RUNNER="%JAVA_HOME%\bin\java"` to `set
RUNNER=%JAVA_HOME%\bin\java`. cc @felixcheung and @shivaram, I just realised I
was cc'ed in the PR (https://github.com/apache/spark/pull/16596) but it looks I
missed this as well ...
I can reproduce this problem as below:
```cmd
C:\...\spark>set JAVA_HOME
JAVA_HOME=C:\Program Files\Java\jdk1.8.0_121
C:\...\spark>.\bin\spark-shell
'""C:\Program' is not recognized as an internal or external command,
operable program or batch file.
```
```cmd
echo "%RUNNER%"
```
prints
```cmd
""C:\Program Files\Java\jdk1.8.0_121\bin\java""
```
It looks cmd does not recognise the space.
To double check, I copied the jdk into `C:\Java` and then ran the commands
as below:
```cmd
C:\...\spark>set JAVA_HOME=C:\Java\jdk1.8.0_121
C:\...\spark>.\bin\spark-shell
...
Spark context Web UI available at http://10.0.2.15:4040
Spark context available as 'sc' (master = local[*], app id =
local-1493961061248).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.0-SNAPSHOT
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) Client VM, Java 1.8.0_121)
Type in expressions to have them evaluated.
Type :help for more information.
...
```
```cmd
echo "%RUNNER%"
```
prints
```cmd
""C:\Java\jdk1.8.0_121\bin\java""
```
**After fixing the line I suggested**
```diff
if not "x%JAVA_HOME%"=="x" (
- set RUNNER="%JAVA_HOME%\bin\java"
+ set RUNNER=%JAVA_HOME%\bin\java
) else (
```
```cmd
C:\...\spark>set JAVA_HOME
JAVA_HOME=C:\Program Files\Java\jdk1.8.0_121
C:\...\spark>.\bin\spark-shell
...
Spark context Web UI available at http://10.0.2.15:4040
Spark context available as 'sc' (master = local[*], app id =
local-1493962115332).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.0-SNAPSHOT
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) Client VM, Java 1.8.0_121)
Type in expressions to have them evaluated.
Type :help for more information.
...
```
```cmd
echo "%RUNNER%"
```
prints
```cmd
"C:\Program Files\Java\jdk1.8.0_121\bin\java"
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
