[
https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757178#comment-15757178
]
Hyukjin Kwon commented on SPARK-18648:
--------------------------------------
Thank you for cc'ing me.
Yes, I think {{textFile}} seems irrelevant. Also, yes, I think the safe choice
is always to use a URI format on Windows where applicable.
I just manually tested and I can reproduce this on Windows although I see these
seem parsed fine as below with {{--jars}}.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:\Users\IEUser\Downloads\spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
In those cases, It seems I can't import the contents in the jar on Windows
whereas I can on my Mac if I haven't done something wrong.
Personally, in my case, I always use
{{--driver-library-path XXX.jar}} and {{--conf
spark.executor.extraLibraryPath=XXX.jar}} in `spark-shell` or `spark-shell.cmd`
for sure.
or
{code}
spark.driver.extraClassPath
spark.executor.extraClassPath
{code}
in the configuration.
Anyway, I believe the case below {{C:/a/b/c}} is discouraged because I have
seen this is failed to be parsed properly as a URI ending up with recognising
C: as the scheme.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
Actually, I noticed such suspicious cases across Spark so I wanted to fix and
and fix them after making the tests passed on Windows all.
So, to cut this short,
- I can reproduce this on Windows
- Workaround seems setting {{spark.driver.extraClassPath}} and
{{spark.executor.extraClassPath}} too (or equivalent options).
- The right paths should be URI forms as suggested above.
- I think we should investigate this.
> spark-shell --jars option does not add jars to classpath on windows
> -------------------------------------------------------------------
>
> Key: SPARK-18648
> URL: https://issues.apache.org/jira/browse/SPARK-18648
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell, Windows
> Affects Versions: 2.0.2
> Environment: Windows 7 x64
> Reporter: Michel Lemay
> Labels: windows
>
> I can't import symbols from command line jars when in the shell:
> Adding jars via --jars:
> {code}
> spark-shell --master local[*] --jars path\to\deeplearning4j-core-0.7.0.jar
> {code}
> Same result if I add it through maven coordinates:
> {code}spark-shell --master local[*] --packages
> org.deeplearning4j:deeplearning4j-core:0.7.0
> {code}
> I end up with:
> {code}
> scala> import org.deeplearning4j
> <console>:23: error: object deeplearning4j is not a member of package org
> import org.deeplearning4j
> {code}
> NOTE: It is working as expected when running on linux.
> Sample output with --verbose:
> {code}
> Using properties file: null
> Parsed arguments:
> master local[*]
> deployMode null
> executorMemory null
> executorCores null
> totalExecutorCores null
> propertiesFile null
> driverMemory null
> driverCores null
> driverExtraClassPath null
> driverExtraLibraryPath null
> driverExtraJavaOptions null
> supervise false
> queue null
> numExecutors null
> files null
> pyFiles null
> archives null
> mainClass org.apache.spark.repl.Main
> primaryResource spark-shell
> name Spark shell
> childArgs []
> jars
> file:/C:/Apps/Spark/spark-2.0.2-bin-hadoop2.4/bin/../deeplearning4j-core-0.7.0.jar
> packages null
> packagesExclusions null
> repositories null
> verbose true
> Spark properties used, including those specified through
> --conf and those from the properties file null:
> Main class:
> org.apache.spark.repl.Main
> Arguments:
> System properties:
> SPARK_SUBMIT -> true
> spark.app.name -> Spark shell
> spark.jars ->
> file:/C:/Apps/Spark/spark-2.0.2-bin-hadoop2.4/bin/../deeplearning4j-core-0.7.0.jar
> spark.submit.deployMode -> client
> spark.master -> local[*]
> Classpath elements:
> file:/C:/Apps/Spark/spark-2.0.2-bin-hadoop2.4/bin/../deeplearning4j-core-0.7.0.jar
> 16/11/30 08:30:49 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 16/11/30 08:30:51 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> Spark context Web UI available at http://192.168.70.164:4040
> Spark context available as 'sc' (master = local[*], app id =
> local-1480512651325).
> Spark session available as 'spark'.
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 2.0.2
> /_/
> Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> import org.deeplearning4j
> <console>:23: error: object deeplearning4j is not a member of package org
> import org.deeplearning4j
> ^
> scala>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
