Thanks Vanzin, spark-submit.cmd works
Thanks
Proust
From: Marcelo Vanzin <van...@cloudera.com>
To: Proust GZ Feng/China/IBM@IBMCN
Cc: Sean Owen <so...@cloudera.com>, user <user@spark.apache.org>
Date: 07/29/2015 10:35 AM
Subject: Re: NO Cygwin Support in bin/spark-class in Spark 1.4.0
Can you run the windows batch files (e.g. spark-submit.cmd) from the
cygwin shell?
On Tue, Jul 28, 2015 at 7:26 PM, Proust GZ Feng <pf...@cn.ibm.com> wrote:
Hi, Owen
Add back the cygwin classpath detection can pass the issue mentioned
before, but there seems lack of further support in the launch lib, see
below stacktrace
LAUNCH_CLASSPATH:
C:\spark-1.4.0-bin-hadoop2.3\lib\spark-assembly-1.4.0-hadoop2.3.0.jar
java -cp
C:\spark-1.4.0-bin-hadoop2.3\lib\spark-assembly-1.4.0-hadoop2.3.0.jar
org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit
--driver-class-path ../thirdparty/lib/db2-jdbc4-95fp6a/db2jcc4.jar
--properties-file conf/spark.properties
target/scala-2.10/price-scala-assembly-15.4.0-SNAPSHOT.jar
Exception in thread "main" java.lang.IllegalStateException: Library
directory 'C:\c\spark-1.4.0-bin-hadoop2.3\lib_managed\jars' does not
exist.
at
org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:229)
at
org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:215)
at
org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:115)
at
org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:192)
at
org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:117)
at org.apache.spark.launcher.Main.main(Main.java:74)
Thanks
Proust
From: Sean Owen <so...@cloudera.com>
To: Proust GZ Feng/China/IBM@IBMCN
Cc: user <user@spark.apache.org>
Date: 07/28/2015 06:54 PM
Subject: Re: NO Cygwin Support in bin/spark-class in Spark 1.4.0
Does adding back the cygwin detection and this clause make it work?
if $cygwin; then
CLASSPATH="`cygpath -wp "$CLASSPATH"`"
fi
If so I imagine that's fine to bring back, if that's still needed.
On Tue, Jul 28, 2015 at 9:49 AM, Proust GZ Feng <pf...@cn.ibm.com> wrote:
> Thanks Owen, the problem under Cygwin is while run spark-submit under
1.4.0,
> it simply report
>
> Error: Could not find or load main class org.apache.spark.launcher.Main
>
> This is because under Cygwin spark-class make the LAUNCH_CLASSPATH as
> "/c/spark-1.4.0-bin-hadoop2.3/lib/spark-assembly-1.4.0-hadoop2.3.0.jar"
> But under Cygwin java in Windows cannot recognize the classpath, so
below
> command simply error out
>
> java -cp
> /c/spark-1.4.0-bin-hadoop2.3/lib/spark-assembly-1.4.0-hadoop2.3.0.jar
> org.apache.spark.launcher.Main
> Error: Could not find or load main class org.apache.spark.launcher.Main
>
> Thanks
> Proust
>
>
>
> From: Sean Owen <so...@cloudera.com>
> To: Proust GZ Feng/China/IBM@IBMCN
> Cc: user <user@spark.apache.org>
> Date: 07/28/2015 02:20 PM
> Subject: Re: NO Cygwin Support in bin/spark-class in Spark 1.4.0
> ________________________________
>
>
>
> It wasn't removed, but rewritten. Cygwin is just a distribution of
> POSIX-related utilities so you should be able to use the normal .sh
> scripts. In any event, you didn't say what the problem is?
>
> On Tue, Jul 28, 2015 at 5:19 AM, Proust GZ Feng <pf...@cn.ibm.com>
wrote:
>> Hi, Spark Users
>>
>> Looks like Spark 1.4.0 cannot work with Cygwin due to the removing of
>> Cygwin
>> support in bin/spark-class
>>
>> The changeset is
>>
>>
https://github.com/apache/spark/commit/517975d89d40a77c7186f488547eed11f79c1e97#diff-fdf4d3e600042c63ffa17b692c4372a3
>>
>> The changeset said "Add a library for launching Spark jobs
>> programmatically", but how to use it in Cygwin?
>> I'm wondering any solutions available to make it work in Windows?
>>
>>
>> Thanks
>> Proust
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
--
Marcelo
