[
https://issues.apache.org/jira/browse/SPARK-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14602993#comment-14602993
]
Josiah Samuel Sathiadass commented on SPARK-8410:
-------------------------------------------------
Captured some logs from 2 servers ( server 1 - with the issue, server 2 - w/o
the issue )
The below text is collected from file ::
~/.ivy2/cache/org.apache.spark-spark-submit-parent-default.xml
Server 1: ( looks for the respective jars in the "local-m2-cache")
>>>> entries from a machine where the problem is <<<<<<<<
<module organisation="org.codehaus.groovy" name="groovy-all">
<revision name="2.1.6" status="release"
pubdate="20150618023837" resolver="local-m2-cache" artresolver="local-m2-cache"
homepage="http://groovy.codehaus.org/"; downloaded="false" searched="false"
default="false" conf="compile, master(*), runtime, compile(*), runtime(*),
master" position="52">
<license name="The Apache Software License,
Version 2.0" url="http://www.apache.org/licenses/LICENSE-2.0.txt"/>
<metadata-artifact status="no" details=""
size="5591" time="0"
location="/home/joe/.ivy2/cache/org.codehaus.groovy/groovy-all/ivy-2.1.6.xml"
searched="false"
original-local-location="/home/joe/.m2/repository/org/codehaus/groovy/groovy-all/2.1.6/groovy-all-2.1.6.pom"
origin-is-local="true"
origin-location="file:/home/joe/.m2/repository/org/codehaus/groovy/groovy-all/2.1.6/groovy-all-2.1.6.pom"/>
<caller organisation="org.apache.hive"
name="hive-exec" conf="default, compile, runtime, master" rev="2.1.6"
rev-constraint-default="2.1.6" rev-constraint-dynamic="2.1.6"
callerrev="0.13.1"/>
<artifacts>
<artifact name="groovy-all" type="jar"
ext="jar" status="failed" details="missing artifact" size="0" time="0"/>
</artifacts>
</revision>
</module>
>>>> entries from a machine where the problem is <<<<<<<<
Server 2: ( looks for the respective jars in the "central")
>>>> entries from a machine where it works <<<<<<<<
<module organisation="org.codehaus.groovy" name="groovy-all">
<revision name="2.1.6" status="release"
pubdate="20130709121712" resolver="central" artresolver="central"
homepage="http://groovy.codehaus.org/"; downloaded="false" searched="false"
default="false" conf="compile, master(*), runtime, compile(*), runtime(*),
master" position="52">
<license name="The Apache Software License,
Version 2.0" url="http://www.apache.org/licenses/LICENSE-2.0.txt"/>
<metadata-artifact status="no" details=""
size="5591" time="0"
location="/home/joe/.ivy2/cache/org.codehaus.groovy/groovy-all/ivy-2.1.6.xml"
searched="false"
original-local-location="/home/joe/.ivy2/cache/org.codehaus.groovy/groovy-all/ivy-2.1.6.xml.original"
origin-is-local="false"
origin-location="https://repo1.maven.org/maven2/org/codehaus/groovy/groovy-all/2.1.6/groovy-all-2.1.6.pom"/>
<caller organisation="org.apache.hive"
name="hive-exec" conf="default, compile, runtime, master" rev="2.1.6"
rev-constraint-default="2.1.6" rev-constraint-dynamic="2.1.6"
callerrev="0.13.1"/>
<artifacts>
<artifact name="groovy-all" type="jar"
ext="jar" status="no" details="" size="6377448" time="0"
location="/home/joe/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.1.6.jar">
<origin-location
is-local="false"
location="https://repo1.maven.org/maven2/org/codehaus/groovy/groovy-all/2.1.6/groovy-all-2.1.6.jar"/>
</artifact>
</artifacts>
</revision>
</module>
>>>> entries from a machine where it works <<<<<<<<
Need some help from the community to identify from where ivy is picking up the
settings to populate this file
~/.ivy2/cache/org.apache.spark-spark-submit-parent-default.xml so that I can
narrow down the problem.
Thanks,
Joe.
> Hive VersionsSuite RuntimeException
> -----------------------------------
>
> Key: SPARK-8410
> URL: https://issues.apache.org/jira/browse/SPARK-8410
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.3.1, 1.4.0
> Environment: IBM Power system - P7
> running Ubuntu 14.04LE
> Reporter: Josiah Samuel Sathiadass
> Assignee: Burak Yavuz
> Priority: Minor
>
> While testing Spark Project Hive, there are RuntimeExceptions as follows,
> VersionsSuite:
> - success sanity check *** FAILED ***
> java.lang.RuntimeException: [download failed:
> org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle), download failed:
> org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar, download failed:
> asm#asm;3.2!asm.jar]
> at
> org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:978)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$3.apply(IsolatedClientLoader.scala:62)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$3.apply(IsolatedClientLoader.scala:62)
> at org.apache.spark.sql.catalyst.util.package$.quietly(package.scala:38)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$.org$apache$spark$sql$hive$client$IsolatedClientLoader$$downloadVersion(IsolatedClientLoader.scala:61)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$1.apply(IsolatedClientLoader.scala:44)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$1.apply(IsolatedClientLoader.scala:44)
> at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189)
> at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$.forVersion(IsolatedClientLoader.scala:44)
> ...
> The tests are executed with the following set of options,
> build/mvn --pl sql/hive --fail-never -Pyarn -Phadoop-2.4
> -Dhadoop.version=2.6.0 test
> Adding the following dependencies in the "spark/sql/hive/pom.xml" file
> solves this issue,
> < <dependency>
> < <groupId>org.jboss.netty</groupId>
> < <artifactId>netty</artifactId>
> < <version>3.2.2.Final</version>
> < <scope>test</scope>
> < </dependency>
> < <dependency>
> < <groupId>org.codehaus.groovy</groupId>
> < <artifactId>groovy-all</artifactId>
> < <version>2.1.6</version>
> < <scope>test</scope>
> < </dependency>
> <
> < <dependency>
> < <groupId>asm</groupId>
> < <artifactId>asm</artifactId>
> < <version>3.2</version>
> < <scope>test</scope>
> < </dependency>
> <
> The question is, Is this the correct way to fix this runtimeException ?
> If yes, Can a pull request fix this issue permanently ?
> If not, suggestions please.
> Updates:
> The above mentioned quick fix is not working with the latest 1.4 because of
> this pull commits :
> [SPARK-8095] Resolve dependencies of --packages in local ivy cache #6788
> https://github.com/apache/spark/pull/6788
> Due to this above commit, now the lookup directories during testing phase
> has changed as follows,
> :: problems summary ::
> :::: WARNINGS
> [NOT FOUND ]
> org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle) (2ms)
> ==== local-m2-cache: tried
>
> file:/home/joe/sparkibmsoe/spark/sql/hive/dummy/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar
> [NOT FOUND ]
> org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar (0ms)
> ==== local-m2-cache: tried
>
> file:/home/joe/sparkibmsoe/spark/sql/hive/dummy/.m2/repository/org/codehaus/groovy/groovy-all/2.1.6/groovy-all-2.1.6.jar
> [NOT FOUND ] asm#asm;3.2!asm.jar (0ms)
> ==== local-m2-cache: tried
>
> file:/home/joe/sparkibmsoe/spark/sql/hive/dummy/.m2/repository/asm/asm/3.2/asm-3.2.jar
> ::::::::::::::::::::::::::::::::::::::::::::::
> :: FAILED DOWNLOADS ::
> :: ^ see resolution messages for details ^ ::
> ::::::::::::::::::::::::::::::::::::::::::::::
> :: org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle)
> :: org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar
> :: asm#asm;3.2!asm.jar
> ::::::::::::::::::::::::::::::::::::::::::::::
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
