Interesting. I will watching your PR.
On Wed, Nov 18, 2015 at 7:51 AM, 임정택 <kabh...@gmail.com> wrote: > Ted, > > I suspect I hit the issue > https://issues.apache.org/jira/browse/SPARK-11818 > Could you refer the issue and verify that it makes sense? > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 2015-11-18 20:32 GMT+09:00 Ted Yu <yuzhih...@gmail.com>: > >> Here is related code: >> >> private static void checkDefaultsVersion(Configuration conf) { >> >> if (conf.getBoolean("hbase.defaults.for.version.skip", Boolean.FALSE)) >> return; >> >> String defaultsVersion = conf.get("hbase.defaults.for.version"); >> >> String thisVersion = VersionInfo.getVersion(); >> >> if (!thisVersion.equals(defaultsVersion)) { >> >> throw new RuntimeException( >> >> "hbase-default.xml file seems to be for an older version of >> HBase (" + >> >> defaultsVersion + "), this version is " + thisVersion); >> >> null means that "hbase.defaults.for.version" was not set in the other >> hbase-default.xml >> >> Can you retrieve the classpath of Spark task so that we can have more >> clue ? >> >> >> Cheers >> >> On Tue, Nov 17, 2015 at 10:06 PM, 임정택 <kabh...@gmail.com> wrote: >> >>> Ted, >>> >>> Thanks for the reply. >>> >>> My fat jar has dependency with spark related library to only spark-core >>> as "provided". >>> Seems like Spark only adds 0.98.7-hadoop2 of hbase-common in >>> spark-example module. >>> >>> And if there're two hbase-default.xml in the classpath, should one of >>> them be loaded, instead of showing (null)? >>> >>> Best, >>> Jungtaek Lim (HeartSaVioR) >>> >>> >>> >>> 2015-11-18 13:50 GMT+09:00 Ted Yu <yuzhih...@gmail.com>: >>> >>>> Looks like there're two hbase-default.xml in the classpath: one for 0.98.6 >>>> and another for 0.98.7-hadoop2 (used by Spark) >>>> >>>> You can specify hbase.defaults.for.version.skip as true in your >>>> hbase-site.xml >>>> >>>> Cheers >>>> >>>> On Tue, Nov 17, 2015 at 1:01 AM, 임정택 <kabh...@gmail.com> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I'm evaluating zeppelin to run driver which interacts with HBase. >>>>> I use fat jar to include HBase dependencies, and see failures on >>>>> executor level. >>>>> I thought it is zeppelin's issue, but it fails on spark-shell, too. >>>>> >>>>> I loaded fat jar via --jars option, >>>>> >>>>> > ./bin/spark-shell --jars hbase-included-assembled.jar >>>>> >>>>> and run driver code using provided SparkContext instance, and see >>>>> failures from spark-shell console and executor logs. >>>>> >>>>> below is stack traces, >>>>> >>>>> org.apache.spark.SparkException: Job aborted due to stage failure: Task >>>>> 55 in stage 0.0 failed 4 times, most recent failure: Lost task 55.3 in >>>>> stage 0.0 (TID 281, <svr hostname>): java.lang.NoClassDefFoundError: >>>>> Could not initialize class >>>>> org.apache.hadoop.hbase.client.HConnectionManager >>>>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:197) >>>>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:159) >>>>> at >>>>> org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:101) >>>>> at >>>>> org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:128) >>>>> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104) >>>>> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66) >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >>>>> at >>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >>>>> at >>>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70) >>>>> at >>>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:70) >>>>> at >>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> >>>>> Driver stacktrace: >>>>> at >>>>> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1273) >>>>> at >>>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1264) >>>>> at >>>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1263) >>>>> at >>>>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) >>>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) >>>>> at >>>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1263) >>>>> at >>>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730) >>>>> at >>>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730) >>>>> at scala.Option.foreach(Option.scala:236) >>>>> at >>>>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730) >>>>> at >>>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1457) >>>>> at >>>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1418) >>>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) >>>>> >>>>> >>>>> 15/11/16 18:59:57 ERROR Executor: Exception in task 14.0 in stage 0.0 >>>>> (TID 14) >>>>> java.lang.ExceptionInInitializerError >>>>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:197) >>>>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:159) >>>>> at >>>>> org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:101) >>>>> at >>>>> org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:128) >>>>> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104) >>>>> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66) >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >>>>> at >>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >>>>> at >>>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70) >>>>> at >>>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:70) >>>>> at >>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Caused by: java.lang.RuntimeException: hbase-default.xml file seems to be >>>>> for and old version of HBase (null), this version is 0.98.6-cdh5.2.0 >>>>> at >>>>> org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:73) >>>>> at >>>>> org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:105) >>>>> at >>>>> org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:116) >>>>> at >>>>> org.apache.hadoop.hbase.client.HConnectionManager.<clinit>(HConnectionManager.java:222) >>>>> ... 18 more >>>>> >>>>> >>>>> Please note that it runs smoothly on spark-submit. >>>>> >>>>> Btw, if issue is that hbase-default.xml is not properly loaded (maybe >>>>> because of classloader), it seems to run properly on driver level. >>>>> >>>>> import org.apache.hadoop.hbase.HBaseConfiguration >>>>> val conf = HBaseConfiguration.create() >>>>> println(conf.get("hbase.defaults.for.version")) >>>>> >>>>> It prints "0.98.6-cdh5.2.0". >>>>> >>>>> I'm using Spark-1.4.1-hadoop-2.4-bin, and zeppelin 0.5.5, and HBase >>>>> 0.98.6-CDH5.2.0. >>>>> >>>>> Thanks in advance! >>>>> >>>>> Best, >>>>> Jungtaek Lim (HeartSaVioR) >>>>> >>>> >>>> >>> >>> >>> -- >>> Name : 임 정택 >>> Blog : http://www.heartsavior.net / http://dev.heartsavior.net >>> Twitter : http://twitter.com/heartsavior >>> LinkedIn : http://www.linkedin.com/in/heartsavior >>> >> >> > > > -- > Name : 임 정택 > Blog : http://www.heartsavior.net / http://dev.heartsavior.net > Twitter : http://twitter.com/heartsavior > LinkedIn : http://www.linkedin.com/in/heartsavior >
