[
https://issues.apache.org/jira/browse/SPARK-19630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15869884#comment-15869884
]
Mirko Kaempf commented on SPARK-19630:
--------------------------------------
Spark docu is already updated, but this version is only visible in the source
code:
<td><code>spark.jars.ivy</code></td>
<td></td>
<td>
Path to specify the Ivy user directory, used for the local Ivy cache and
package files from
<code>spark.jars.packages</code>. This will override the Ivy property
<code>ivy.default.ivy.user.dir</code>
which defaults to ~/.ivy2.
</td>
https://github.com/apache/spark/edit/master/docs/configuration.md
> spark.jars.ivy explanation is incorrect and missleading.
> --------------------------------------------------------
>
> Key: SPARK-19630
> URL: https://issues.apache.org/jira/browse/SPARK-19630
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
> Affects Versions: 1.6.2, 1.6.3, 2.0.0, 2.0.1, 2.0.2, 2.1.0
> Reporter: Mirko Kaempf
>
> Following the spark documentation
> (http://spark.apache.org/docs/latest/configuration.html) I hoped that this
> property allows me to configure some custom repositories to be used by
> Spark-shell:
> _spark.jars.ivy Comma-separated list of additional remote repositories
> to search for the coordinates given with spark.jars.packages._
> Using e.g., this configuration:
> _spark.jars.ivy
> https://oss.sonatype.org/content/repositories/snapshots/,http://bits.netbeans.org/maven2/,https://maven.atlassian.com/repository/public/_
> leads to an error which initially makes not much sense to me:
> Ivy Default Cache set to:
> */sparkws/bin/*https:/oss.sonatype.org/content/repositories/snapshots/,http:/bits.netbeans.org/maven2/,https:/maven.atlassian.com/repository/public/cache
> The jars for the packages stored in:
> https:/oss.sonatype.org/content/repositories/snapshots/,http:/bits.netbeans.org/maven2/,https:/maven.atlassian.com/repository/public/jars
> Exception in thread "main" java.lang.IllegalArgumentException: basedir must
> be absolute:
> https:/oss.sonatype.org/content/repositories/snapshots/,http:/bits.netbeans.org/maven2/,https:/maven.atlassian.com/repository/public/local
> at org.apache.ivy.util.Checks.checkAbsolute(Checks.java:48)
> I run the spark-shell from folder /sparkws/bin => to me it looks like that
> the property points to a path in which Ivy caches the data locally, but the
> docs say something different: „Comma-separated list of additional remote
> repositories to search for the coordinates given with spark.jars.packages“.
> Could this be a „docs-bug“ eventually?
> Some more tests showed that in fact the setting:
> _spark.jars.ivy /sparkws/bin/jars _
> is used by Ivy to store the dynamically loaded JAR files, and not to provide
> the URL of remote repositories.
> Locally we have found this solution working:
> We added this CLI option to the spark-shell start command:
> --properties (COMMA SEPARATED LIST OF REPOS)
> This approach worked well.
> _Ivy Default Cache set to: /sparkws/bin/jars/cache
> The jars for the packages stored in: /sparkws/bin/jars/jars
> http://bits.netbeans.org/maven2/ added as a remote repository with the name:
> repo-1
> https://oss.sonatype.org/content/repositories/snapshots/ added as a remote
> repository with the name: repo-2_
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
