Have you tried adding path to hbase-protocol jar to
spark.driver.extraClassPath and spark.executor.extraClassPath ?
Cheers
On Mon, Aug 17, 2015 at 7:51 PM, stark_summer wrote:
> spark vesion:1.4.1
> java version:1.7
> hadoop version:
> Hadoop 2.3.0-cdh5.1.0
>
> submit spark job to yarn cluster that read hbase data,after job running, it
> comes below error :
>
> 15/08/17 19:28:33 ERROR yarn.ApplicationMaster: User class threw exception:
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.IllegalAccessError:
> class com.google.protobuf.HBaseZeroCopyByteString ca
> nnot access its superclass com.google.protobuf.LiteralByteString
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.IllegalAccessError:
> class com.google.protobuf.HBaseZeroCopyByteString cannot access its
> superclass com.google.protobuf.LiteralByteString
> at
>
> org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:210)
> at
>
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:121)
> at
>
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
> at
>
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
> at
>
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:164)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:107)
> at
> org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:736)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:178)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:82)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isTableAvailable(HConnectionManager.java:962)
> at
>
> org.apache.hadoop.hbase.client.HBaseAdmin.isTableAvailable(HBaseAdmin.java:1081)
> at
>
> org.apache.hadoop.hbase.client.HBaseAdmin.isTableAvailable(HBaseAdmin.java:1089)
> at com.umeng.dp.yuliang.play.HBaseToES$.main(HBaseToES.scala:28)
> at com.umeng.dp.yuliang.play.HBaseToES.main(HBaseToES.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
>
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:483)
> Caused by: java.lang.IllegalAccessError: class
> com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass
> com.google.protobuf.LiteralByteString
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
>
> org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:930)
> at
>
> org.apache.hadoop.hbase.protobuf.RequestConverter.buildScanRequest(RequestConverter.java:434)
> at
>
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:297)
> at
>
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
> at
>
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)
> at
>
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
> ... 18 more
>
> PS:
> running hadoop mr on yarn that read hbase data, also have this error,
> https://issues.apache.org/jira/browse/HBASE-10304,that is hbase issues ,
>
> when submit hadoop mr, add export
>
> HADOOP_CLASSPATH="./hbase/hbase-protocol/target/hbase-protocol-0.99.0-SNAPSHOT.jar"
> to shell comand
> or add export
>
> HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/cluster/apps/hbase/lib/hbase-protocol-0.98.1-cdh5.1.0.jar
> to linux /etc/basrc file,it can work well,
>
> but submit spark job ,it can not work
>
>
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-IllegalAccessError-class-com-goog
