spark-hive is excluded when using --packages, because it can be included in the spark-assembly by adding -Phive during mvn package or sbt assembly.
Best, Burak On Tue, Jul 7, 2015 at 8:06 AM, Hao Ren <inv...@gmail.com> wrote: > I want to add spark-hive as a dependence to submit my job, but it seems > that > spark-submit can not resolve it. > > $ ./bin/spark-submit \ > → --packages > > org.apache.spark:spark-hive_2.10:1.4.0,org.postgresql:postgresql:9.3-1103-jdbc3,joda-time:joda-time:2.8.1 > \ > → --class fr.leboncoin.etl.jobs.dwh.AdStateTraceDWHTransform \ > → --master spark://localhost:7077 \ > > Ivy Default Cache set to: /home/invkrh/.ivy2/cache > The jars for the packages stored in: /home/invkrh/.ivy2/jars > https://repository.jboss.org/nexus/content/repositories/releases/ added > as a > remote repository with the name: repo-1 > :: loading settings :: url = > > jar:file:/home/invkrh/workspace/scala/spark/assembly/target/scala-2.10/spark-assembly-1.4.0-SNAPSHOT-hadoop2.2.0.jar!/org/apache/ivy/core/settings/ivysettings.xml > org.apache.spark#spark-hive_2.10 added as a dependency > org.postgresql#postgresql added as a dependency > joda-time#joda-time added as a dependency > :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 > confs: [default] > found org.postgresql#postgresql;9.3-1103-jdbc3 in local-m2-cache > found joda-time#joda-time;2.8.1 in central > :: resolution report :: resolve 139ms :: artifacts dl 3ms > :: modules in use: > joda-time#joda-time;2.8.1 from central in [default] > org.postgresql#postgresql;9.3-1103-jdbc3 from local-m2-cache in > [default] > > --------------------------------------------------------------------- > | | modules || artifacts > | > | conf | number| search|dwnlded|evicted|| > number|dwnlded| > > --------------------------------------------------------------------- > | default | 2 | 0 | 0 | 0 || 2 | 0 > | > > --------------------------------------------------------------------- > :: retrieving :: org.apache.spark#spark-submit-parent > confs: [default] > 0 artifacts copied, 2 already retrieved (0kB/6ms) > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/spark/sql/hive/HiveContext > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:633) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) > at > org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > org.apache.spark.sql.hive.HiveContext > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 7 more > Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > 15/07/07 16:57:59 INFO Utils: Shutdown hook called > > Any help is appreciated. Thank you. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/spark-submit-can-not-resolve-spark-hive-2-10-tp23695.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
