[
https://issues.apache.org/jira/browse/SPARK-11249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14987482#comment-14987482
]
Mark Grover commented on SPARK-11249:
-------------------------------------
Thanks for filing this, Hari. I am a little torn about this and would
appreciate your and other folks' (cc [~vanzin]) input. I poked around at
SparkSubmit* code and [the relevant
docs|https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management].
[The
code|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L239
currently throws an exception if a primary resource is not defined. Primary
resource being, say a jar, when submitting a scala/java spark app using
spark-submit. Also, it seems that the original intended purpose of
{{--packages}} was related to fetching the dependencies from maven, not
necessarily the actual app jar. In other words, it's meant to simplify usage of
{{--jars}} when using spark-submit.
Technically, users can download their application jar using {{--packages}}
similar to how they can supply their application jar using {{--jars}} but even
in the latter case, they'd still have to specify the app jar separately.
So, I think we have 2 options:
1. Make specifying the primary resource optional. In such a case, this would
apply only if {{--jars}} or {{--packages}} is being used in the same command
line and when the primary resource is a jar.
2. Leave things the way they are, and perhaps, clarify the documentation to say
that the intended purpose of {{--packages}} is to use it for dependency
management, not necessarily for fetching the app jar. And, if you are using it
for fetching the app jar, then you'd still need to specify an existing dummy
jar for primary resource.
What do you think? Any other option that I missed? Thanks in advance for your
help!
> [Launcher] Launcher library fails is app resource is not added
> --------------------------------------------------------------
>
> Key: SPARK-11249
> URL: https://issues.apache.org/jira/browse/SPARK-11249
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 1.5.1
> Reporter: Hari Shreedharan
>
> If the resource is downloaded via --packages, the app resource is not
> required. But the launcher library gets confused and assumes the first arg
> is the resource to be passed and all args don't get passed to the app itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
