Re: java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass com.google.protobuf.LiteralByteString

2015-08-17 Thread Ted Yu 
Have you tried adding path to hbase-protocol jar to
spark.driver.extraClassPath and spark.executor.extraClassPath ?

Cheers

On Mon, Aug 17, 2015 at 7:51 PM, stark_summer stark_sum...@qq.com wrote:

 spark vesion:1.4.1
 java version:1.7
 hadoop version:
 Hadoop 2.3.0-cdh5.1.0

 submit spark job to yarn cluster that read hbase data,after job running, it
 comes  below error :

 15/08/17 19:28:33 ERROR yarn.ApplicationMaster: User class threw exception:
 org.apache.hadoop.hbase.DoNotRetryIOException:
 java.lang.IllegalAccessError:
 class com.google.protobuf.HBaseZeroCopyByteString ca
 nnot access its superclass com.google.protobuf.LiteralByteString
 org.apache.hadoop.hbase.DoNotRetryIOException:
 java.lang.IllegalAccessError:
 class com.google.protobuf.HBaseZeroCopyByteString cannot access its
 superclass com.google.protobuf.LiteralByteString
 at

 org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:210)
 at

 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:121)
 at

 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
 at

 org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
 at

 org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
 at
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
 at
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
 at
 org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:736)
 at
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:178)
 at
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:82)
 at

 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isTableAvailable(HConnectionManager.java:962)
 at

 org.apache.hadoop.hbase.client.HBaseAdmin.isTableAvailable(HBaseAdmin.java:1081)
 at

 org.apache.hadoop.hbase.client.HBaseAdmin.isTableAvailable(HBaseAdmin.java:1089)
 at com.umeng.dp.yuliang.play.HBaseToES$.main(HBaseToES.scala:28)
 at com.umeng.dp.yuliang.play.HBaseToES.main(HBaseToES.scala)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at

 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at

 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at

 org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:483)
 Caused by: java.lang.IllegalAccessError: class
 com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass
 com.google.protobuf.LiteralByteString
 at java.lang.ClassLoader.defineClass1(Native Method)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
 at
 java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
 at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
 at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
 at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
 at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
 at

 org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:930)
 at

 org.apache.hadoop.hbase.protobuf.RequestConverter.buildScanRequest(RequestConverter.java:434)
 at

 org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:297)
 at

 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
 at

 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)
 at

 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
 ... 18 more

 PS:
 running hadoop mr on yarn that read hbase data, also have this error,
 https://issues.apache.org/jira/browse/HBASE-10304,that is hbase  issues ,

 when submit hadoop mr, add  export

 HADOOP_CLASSPATH=./hbase/hbase-protocol/target/hbase-protocol-0.99.0-SNAPSHOT.jar
 to shell comand
 or add export

 HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/cluster/apps/hbase/lib/hbase-protocol-0.98.1-cdh5.1.0.jar
 to linux /etc/basrc file,it can work well,

 but  submit spark job ,it can not work










 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-IllegalAccessError-class-com-google-protobuf-HBaseZeroCopyByteString-cannot-access-its-supg-tp24303.html
 Sent from the Apache Spark

Re: java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass com.google.protobuf.LiteralByteString

2015-08-17 Thread stark_summer 
approach1:
submit spark job add bolow:
 --conf
spark.driver.extraClassPath=/home/cluster/apps/hbase/lib/hbase-protocol-0.98.1-cdh5.1.0.jar
--conf
spark.executor.extraClassPath=/home/cluster/apps/hbase/lib/hbase-protocol-0.98.1-cdh5.1.0.jar
such as:

/home/dp/spark/spark-1.4/spark-1.4.1/bin/spark-submit --class
com.umeng.dp.yuliang.play.HBaseToES --master yarn-cluster --conf
spark.driver.extraClassPath=/home/cluster/apps/hbase/lib/hbase-protocol-0.98.1-cdh5.1.0.jar
--conf
spark.executor.extraClassPath=/home/cluster/apps/hbase/lib/hbase-protocol-0.98.1-cdh5.1.0.jar
  
--jars /home/cluster/apps/hbase/lib/hbase-protocol-0.98.1-cdh5.1.0.jar
ScalaMR-0.0.1-jar-with-dependencies.jar 


approach2:
add below config to $SPARK_HOME/conf/spark-deafults.conf
spark.driver.extraClassPath
/home/cluster/apps/hbase/lib/hbase-protocol-0.98.1-cdh5.1.0.jar
spark.executor.extraClassPath
/home/cluster/apps/hbase/lib/hbase-protocol-0.98.1-cdh5.1.0.jar




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-IllegalAccessError-class-com-google-protobuf-HBaseZeroCopyByteString-cannot-access-its-supg-tp24303p24306.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass com.google.protobuf.LiteralByteString

2015-08-17 Thread stark_summer 
spark vesion:1.4.1
java version:1.7
hadoop version:
Hadoop 2.3.0-cdh5.1.0

submit spark job to yarn cluster that read hbase data,after job running, it
comes  below error :

15/08/17 19:28:33 ERROR yarn.ApplicationMaster: User class threw exception:
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError:
class com.google.protobuf.HBaseZeroCopyByteString ca
nnot access its superclass com.google.protobuf.LiteralByteString
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError:
class com.google.protobuf.HBaseZeroCopyByteString cannot access its
superclass com.google.protobuf.LiteralByteString
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:210)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:121)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:736)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:178)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:82)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isTableAvailable(HConnectionManager.java:962)
at
org.apache.hadoop.hbase.client.HBaseAdmin.isTableAvailable(HBaseAdmin.java:1081)
at
org.apache.hadoop.hbase.client.HBaseAdmin.isTableAvailable(HBaseAdmin.java:1089)
at com.umeng.dp.yuliang.play.HBaseToES$.main(HBaseToES.scala:28)
at com.umeng.dp.yuliang.play.HBaseToES.main(HBaseToES.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:483)
Caused by: java.lang.IllegalAccessError: class
com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass
com.google.protobuf.LiteralByteString
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:930)
at
org.apache.hadoop.hbase.protobuf.RequestConverter.buildScanRequest(RequestConverter.java:434)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:297)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
... 18 more

PS:
running hadoop mr on yarn that read hbase data, also have this error,
https://issues.apache.org/jira/browse/HBASE-10304,that is hbase  issues ,

when submit hadoop mr, add  export
HADOOP_CLASSPATH=./hbase/hbase-protocol/target/hbase-protocol-0.99.0-SNAPSHOT.jar
 
to shell comand 
or add export
HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/cluster/apps/hbase/lib/hbase-protocol-0.98.1-cdh5.1.0.jar
 
to linux /etc/basrc file,it can work well,

but  submit spark job ,it can not work










--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-IllegalAccessError-class-com-google-protobuf-HBaseZeroCopyByteString-cannot-access-its-supg-tp24303.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org