Spark couldn't connect with the HBase cluster for metadata. Some possible causes I can think of: 1) the correct hbase-site.xml wasn't in the HADOOP_CONF_DIR 2) some HBase dependent jar wasn't submitted with the application
You may need check more logs, including the Spark executor logs. Start Spark history sever and then check that job's detail log. 2017-06-03 18:36 GMT+08:00 qiumingming <csurj...@gmail.com>: > Hi All: > I got a error when cubing by spark which make me confused. When I start > kylin I got these log: > 2017-06-03 18:09:19,647 INFO [Thread-8-EventThread] > state.ConnectionStateManager:228 : State > change: CONNECTED > 2017-06-03 18:09:19,680 INFO [Thread-8] execution.ExecutableManager:53 > : Using metadata url: > kylin_v2_metadata@hbase > 2017-06-03 18:09:19,682 INFO [Thread-8] dao.ExecutableDao:75 : Using > metadata url: > kylin_v2_metadata@hbase > 2017-06-03 18:09:19,687 INFO [Thread-8] persistence.ResourceStore:88 : > Using metadata url > kylin_v2_metadata@hbase for resource store > 2017-06-03 18:09:19,708 DEBUG [Thread-8] hbase.HBaseConnection:274 : > HTable > 'kylin_v2_metadata' already exists > 2017-06-03 18:09:19,710 DEBUG [Thread-8] metadata.MetadataManager:424 : > Reloading SourceTable > from folder kylin_v2_metadata(key='/table')@kylin_v2_metadata@hbase > > It seems the metadata store works well. But when I start cubing by spark > , the log file shows these info: > 17/06/03 18:14:16 WARN scheduler.TaskSetManager: Lost task 0.0 in stage > 0.0 (TID 0, n6-129-021.byted.org): java.lang.IllegalArgumentException: > Failed to find metadata store by url: kylin_v2_metadata@hbase > at > org.apache.kylin.common.persistence.ResourceStore.createResourceStore( > ResourceStore.java:99) > at > org.apache.kylin.common.persistence.ResourceStore. > getStore(ResourceStore.java:110) > at > org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370) > at > org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc( > CubeDescManager.java:298) > at > org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109) > at > org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81) > at > org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109) > at > org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119) > at > org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467) > at > org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48) > at > org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance( > AbstractRowKeyEncoder.java:48) > at > org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:209) > at > org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:197) > at > org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply( > JavaPairRDD.scala:1025) > at > org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply( > JavaPairRDD.scala:1025) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter. > scala:191) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write( > SortShuffleWriter.scala:64) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:218) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > I don't know why this happened, I need your help. Thanks. > > -- > View this message in context: http://apache-kylin.74782.x6. > nabble.com/Failed-to-find-metadata-store-when-cubing-by-spark-tp8167.html > Sent from the Apache Kylin mailing list archive at Nabble.com. > -- Best regards, Shaofeng Shi 史少锋
