[ https://issues.apache.org/jira/browse/SPARK-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593453#comment-14593453 ]
Josiah Samuel Sathiadass commented on SPARK-8410: ------------------------------------------------- My findings so far, 1. In most of our machines where it runs successfully, these jars(netty-3.2.2.Final.jar, groovy-all-2.1.6.jar, asm-3.2.jar) are already found in the following locations, ~/.ivy2/jars ~/.ivy2/cache ~/.m2/repository/ In a fresh installation, these jars are not found and need to be downloaded explicitly from the maven repo. But its not triggered when spark is built/package/install/test since these dependencies seems to be missed out in one of the pom.xml(Please correct me if I'm wrong ) 2. To confirm it, clear up the jars from the above listed directories, it will consistently fail to retrieve the jars from the repositories in all the machines. 3. The following code throws the exception while trying to resolve the MavenCoordinates by the "SparkSubmitUtils.resolveMavenCoordinates" call, >>> // resolve dependencies val rr: ResolveReport = ivy.resolve(md, resolveOptions) if (rr.hasError) { throw new RuntimeException(rr.getAllProblemMessages.toString) } <<< Here is a verbose output of the call, Ivy Default Cache set to: /home/joe/.ivy2/cache The jars for the packages stored in: /home/joe/.ivy2/jars http://www.datanucleus.org/downloads/maven2 added as a remote repository with the name: repo-1 org.apache.hive#hive-metastore added as a dependency org.apache.hive#hive-exec added as a dependency org.apache.hive#hive-common added as a dependency org.apache.hive#hive-serde added as a dependency com.google.guava#guava added as a dependency org.apache.hadoop#hadoop-client added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found org.apache.hive#hive-metastore;0.13.1 in central found org.apache.hive#hive-serde;0.13.1 in central found org.apache.hive#hive-common;0.13.1 in central found org.apache.hive#hive-shims;0.13.1 in central found org.apache.hive.shims#hive-shims-common;0.13.1 in central ... .. .. . :: resolution report :: resolve 4149ms :: artifacts dl 100ms :: modules in use: antlr#antlr;2.7.7 from local-m2-cache in [default] aopalliance#aopalliance;1.0 from local-m2-cache in [default] asm#asm;3.2 from local-m2-cache in [default] com.google.code.findbugs#jsr305;1.3.9 from local-m2-cache in [default] com.google.guava#guava;14.0.1 from local-m2-cache in [default] com.google.inject#guice;3.0 from local-m2-cache in [default] ... .. .. . :: evicted modules: log4j#log4j;1.2.16 by [log4j#log4j;1.2.17] in [default] com.google.guava#guava;11.0.2 by [com.google.guava#guava;14.0.1] in [default] commons-lang#commons-lang;2.4 by [commons-lang#commons-lang;2.6] in [default] commons-collections#commons-collections;3.1 by [commons-collections#commons-collections;3.2.1] in [default] commons-httpclient#commons-httpclient;3.0.1 by [commons-httpclient#commons-httpclient;3.1] in [default] org.codehaus.jackson#jackson-core-asl;1.8.8 by [org.codehaus.jackson#jackson-core-asl;1.9.2] in [default] org.codehaus.jackson#jackson-mapper-asl;1.8.8 by [org.codehaus.jackson#jackson-mapper-asl;1.9.2] in [default] org.apache.avro#avro;1.7.4 by [org.apache.avro#avro;1.7.5] in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 108 | 0 | 0 | 8 || 100 | 0 | --------------------------------------------------------------------- :: problems summary :: :::: WARNINGS [NOT FOUND ] org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle) (2ms) ==== local-m2-cache: tried file:/home/joe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar [NOT FOUND ] org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar (0ms) ==== local-m2-cache: tried file:/home/joe/.m2/repository/org/codehaus/groovy/groovy-all/2.1.6/groovy-all-2.1.6.jar [NOT FOUND ] asm#asm;3.2!asm.jar (0ms) ==== local-m2-cache: tried file:/home/joe/.m2/repository/asm/asm/3.2/asm-3.2.jar :::::::::::::::::::::::::::::::::::::::::::::: :: FAILED DOWNLOADS :: :: ^ see resolution messages for details ^ :: :::::::::::::::::::::::::::::::::::::::::::::: :: org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle) :: org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar :: asm#asm;3.2!asm.jar :::::::::::::::::::::::::::::::::::::::::::::: Kindly help me find a solution. > Hive VersionsSuite RuntimeException > ----------------------------------- > > Key: SPARK-8410 > URL: https://issues.apache.org/jira/browse/SPARK-8410 > Project: Spark > Issue Type: Question > Components: SQL > Affects Versions: 1.3.1, 1.4.0 > Environment: IBM Power system - P7 > running Ubuntu 14.04LE > with IBM JDK version 1.7.0 > Reporter: Josiah Samuel Sathiadass > Priority: Minor > > While testing Spark Project Hive, there are RuntimeExceptions as follows, > VersionsSuite: > - success sanity check *** FAILED *** > java.lang.RuntimeException: [download failed: > org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle), download failed: > org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar, download failed: > asm#asm;3.2!asm.jar] > at > org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:978) > at > org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$3.apply(IsolatedClientLoader.scala:62) > at > org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$3.apply(IsolatedClientLoader.scala:62) > at org.apache.spark.sql.catalyst.util.package$.quietly(package.scala:38) > at > org.apache.spark.sql.hive.client.IsolatedClientLoader$.org$apache$spark$sql$hive$client$IsolatedClientLoader$$downloadVersion(IsolatedClientLoader.scala:61) > at > org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$1.apply(IsolatedClientLoader.scala:44) > at > org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$1.apply(IsolatedClientLoader.scala:44) > at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189) > at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91) > at > org.apache.spark.sql.hive.client.IsolatedClientLoader$.forVersion(IsolatedClientLoader.scala:44) > ... > The tests are executed with the following set of options, > build/mvn --pl sql/hive --fail-never -Pyarn -Phadoop-2.4 > -Dhadoop.version=2.6.0 test > Adding the following dependencies in the "spark/sql/hive/pom.xml" file > solves this issue, > < <dependency> > < <groupId>org.jboss.netty</groupId> > < <artifactId>netty</artifactId> > < <version>3.2.2.Final</version> > < <scope>test</scope> > < </dependency> > < <dependency> > < <groupId>org.codehaus.groovy</groupId> > < <artifactId>groovy-all</artifactId> > < <version>2.1.6</version> > < <scope>test</scope> > < </dependency> > < > < <dependency> > < <groupId>asm</groupId> > < <artifactId>asm</artifactId> > < <version>3.2</version> > < <scope>test</scope> > < </dependency> > < > The question is, Is this the correct way to fix this runtimeException ? > If yes, Can a pull request fix this issue permanently ? > If not, suggestions please. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org