Classpath problem trying to use DataFrames

[Christopher Brady]

I'm trying to run a basic "Hello world" type example using DataFrames 
with Hive in yarn-client mode. My code is:

JavaSparkContext sc = new JavaSparkContext("yarn-client", "Test app"))
HiveContext sqlContext = new HiveContext(sc.sc());
sqlContext.sql("SELECT * FROM my_table").count();
The exception I get on the driver is:
java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.plan.TableDesc

There are no exceptions on the executors.

That class is definitely on the classpath of the driver, and it runs 
without errors in local mode. I haven't been able to find any similar 
errors on google. Does anyone know what I'm doing wrong?

The full stack trace is included below:

java.lang.NoClassDefFoundError: Lorg/apache/hadoop/hive/ql/plan/TableDesc;
    at java.lang.Class.getDeclaredFields0(Native Method)
    at java.lang.Class.privateGetDeclaredFields(Class.java:2436)
    at java.lang.Class.getDeclaredField(Class.java:1946)
    at 
java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)
    at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
    at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)
    at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:602)
    at 
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at 
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at 
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at 
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at 
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    at 
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:72)
    at 
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:98)
    at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:64)
    at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at 
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.hive.ql.plan.TableDesc
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 109 more

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org