Hi ShaoFeng , no more other error msg before or after this in kylin.log, but I try to execute cmd by spart-submit directly like this: -------------------------------------------------------------------- ./spark-submit --class org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=1 --conf spark.yarn.queue=default --conf spark.history.fs.logDirectory=hdfs://xxx/user/user1/kylin_2_0_0_test/spark-history --conf spark.master=yarn --conf spark.executor.memory=1G --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs://xxx/user/user1/kylin_2_0_0_test/spark-history --conf spark.executor.cores=2 --conf spark.submit.deployMode=cluster --files /xxx/hbase-0.98.8-hadoop2/conf/hbase-site.xml --jars /xxx/kylin-deploy/kylin-2.0.0/lib/kylin-job-2.0.0.jar,/user/user1/ext_lib/htrace-core-2.04.jar,/user/user1/ext_lib/hbase-client-0.98.8-hadoop2.jar,/user/user1/ext_lib/hbase-common-0.98.8-hadoop2.jar,/user/user1/ext_lib/hbase-protocol-0.98.8-hadoop2.jar,/user/user1/ext_lib/metrics-core-2.2.0.jar,/user/user1/ext_lib/guava-12.0.1.jar, /xxx/kylin-deploy/kylin-2.0.0/lib/kylin-job-2.0.0.jar -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable default.kylin_intermediate_kylin_sales_test_cube_a0cd9950_cddc_4c3b_aaa5_fddf87d1fdaa -segmentId a0cd9950-cddc-4c3b-aaa5-fddf87d1fdaa -confPath /xxx/kylin-deploy/kylin-2.0.0/conf -output hdfs:///user/user1/kylin_2_0_0_test/kylin_metadata_2_0_0/kylin-7a376cb7-7ee7-43fd-95dd-79c2c1999f40/kylin_sales_test_cube/cuboid/ -cubename kylin_sales_test_cube -------------------------------------------------------------------- Then, I got some other msg from spark, here is the full error msg: -------------------------------------------------------------------- 17/06/22 11:35:19 ERROR HBaseConnection: Error when open connection hbase java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:413) at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:306) at org.apache.kylin.storage.hbase.HBaseConnection.get(HBaseConnection.java:229) at org.apache.kylin.storage.hbase.HBaseResourceStore.getConnection(HBaseResourceStore.java:72) at org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:89) at org.apache.kylin.storage.hbase.HBaseResourceStore.<init>(HBaseResourceStore.java:85) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:91) at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) at org.apache.kylin.cube.CubeManager.getStore(CubeManager.java:820) at org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:740) at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:145) at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:109) at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:160) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:411) ... 27 more Caused by: java.lang.NoClassDefFoundError: org/cloudera/htrace/Trace at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:479) at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65) at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:839) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:642) ... 32 more Caused by: java.lang.ClassNotFoundException: org.cloudera.htrace.Trace at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 38 more 17/06/22 11:35:19 ERROR ResourceStore: Create new store instance failed java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:91) at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) at org.apache.kylin.cube.CubeManager.getStore(CubeManager.java:820) at org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:740) at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:145) at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:109) at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:160) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.IllegalArgumentException: File not exist by 'kylin_metadata_2_0_0@hbase': /xxx/spark-1.6.2-bin-hadoop2.7/kylin_metadata_2_0_0@hbase at org.apache.kylin.common.persistence.FileResourceStore.<init>(FileResourceStore.java:49) ... 22 more 17/06/22 11:35:19 ERROR ResourceStore: Create new store instance failed java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:91) at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) at org.apache.kylin.cube.CubeManager.getStore(CubeManager.java:820) at org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:740) at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:145) at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:109) at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:160) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.apache.kylin.common.persistence.StorageException: Error when open connection hbase at org.apache.kylin.storage.hbase.HBaseConnection.get(HBaseConnection.java:242) at org.apache.kylin.storage.hbase.HBaseResourceStore.getConnection(HBaseResourceStore.java:72) at org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:89) at org.apache.kylin.storage.hbase.HBaseResourceStore.<init>(HBaseResourceStore.java:85) ... 22 more Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:413) at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:306) at org.apache.kylin.storage.hbase.HBaseConnection.get(HBaseConnection.java:229) ... 25 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:411) ... 27 more Caused by: java.lang.NoClassDefFoundError: org/cloudera/htrace/Trace at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:479) at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65) at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:839) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:642) ... 32 more Caused by: java.lang.ClassNotFoundException: org.cloudera.htrace.Trace at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 38 more 17/06/22 11:35:19 INFO ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) Exception in thread "main" java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata_2_0_0@hbase at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99) at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) at org.apache.kylin.cube.CubeManager.getStore(CubeManager.java:820) at org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:740) at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:145) at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:109) at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:160) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) ... 10 more 17/06/22 11:35:19 INFO ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session 17/06/22 11:35:19 INFO SparkContext: Invoking stop() from shutdown hook -------------------------------------------------------------------- Thanks for your attention! 2017-06-22
skyyws 发件人：ShaoFeng Shi <[email protected]> 发送时间：2017-06-22 11:42 主题：Re: Re: Re: Build sample error with spark on kylin 2.0.0 收件人："dev"<[email protected]> 抄送： hi Sky, glad to see it moves forward. The "Failed to find metadata store by url: kylin_metadata_2_0 _0@hbase" is not root cause. Could you check more with the log files, is there any other error before or after this? 2017-06-21 20:43 GMT+08:00 skyyws <[email protected]>: > Thank you for your suggestion, Shaofeng Shi, I try to use hadoop client > 2.7.3, it worked. But I met another probelm: > -------------------------- -------------------------- > -------------------------- -------------------------- > 17/06/21 20:20:39 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, > hadoop645.lt.163.org): java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata_2_0 > _0@hbase > at org.apache.kylin.common.persistence.ResourceStore. > createResourceStore(ResourceStore.java:99) > at org.apache.kylin.common.persistence.ResourceStore. > getStore(ResourceStore.java:110) > at org.apache.kylin.cube.CubeDescManager.getStore( > CubeDescManager.java:370) > at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc( > CubeDescManager.java:298) > at org.apache.kylin.cube.CubeDescManager.<init>( > CubeDescManager.java:109) > at org.apache.kylin.cube.CubeDescManager.getInstance( > CubeDescManager.java:81) > at org.apache.kylin.cube.CubeInstance.getDescriptor( > CubeInstance.java:114) > at org.apache.kylin.cube.CubeSegment.getCubeDesc( > CubeSegment.java:119) > at org.apache.kylin.cube.kv.RowKeyEncoder.<init>( > RowKeyEncoder.java:50) > at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance( > AbstractRowKeyEncoder.java:48) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:205) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:193) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at org.apache.spark.util.collection.ExternalSorter. > insertAll(ExternalSorter.scala:191) > at org.apache.spark.shuffle.sort.SortShuffleWriter.write( > SortShuffleWriter.scala:64) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:73) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 17/06/21 20:20:39 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID > 2, hadoop645.lt.163.org, partition 0,RACK_LOCAL, 3276 bytes) > 17/06/21 20:21:14 WARN TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, > hadoop645.lt.163.org): java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata_2_0 > _0@hbase > at org.apache.kylin.common.persistence.ResourceStore. > createResourceStore(ResourceStore.java:99) > at org.apache.kylin.common.persistence.ResourceStore. > getStore(ResourceStore.java:110) > at org.apache.kylin.cube.CubeDescManager.getStore( > CubeDescManager.java:370) > at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc( > CubeDescManager.java:298) > at org.apache.kylin.cube.CubeDescManager.<init>( > CubeDescManager.java:109) > at org.apache.kylin.cube.CubeDescManager.getInstance( > CubeDescManager.java:81) > at org.apache.kylin.cube.CubeInstance.getDescriptor( > CubeInstance.java:114) > at org.apache.kylin.cube.CubeSegment.getCubeDesc( > CubeSegment.java:119) > at org.apache.kylin.cube.kv.RowKeyEncoder.<init>( > RowKeyEncoder.java:50) > at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance( > AbstractRowKeyEncoder.java:48) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:205) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:193) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at org.apache.spark.util.collection.ExternalSorter. > insertAll(ExternalSorter.scala:191) > at org.apache.spark.shuffle.sort.SortShuffleWriter.write( > SortShuffleWriter.scala:64) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:73) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > -------------------------- -------------------------- > -------------------------- -------------------------- > But I can use the kylin in-build spark-shell to get data from hive and > hbase successfully, just like this: > -------------------------- -------------------------- > -------------------------- -------------------------- > sqlContext.sql("show tables").take(1) > -------------------------- -------------------------- > -------------------------- -------------------------- > import org.apache.spark._ > import org.apache.spark.rdd.NewHadoopRDD > import org.apache.hadoop.fs.Path > import org.apache.hadoop.hbase.util.Bytes > import org.apache.hadoop.hbase.HColumnDescriptor > import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor} > import org.apache.hadoop.hbase.client.{HBaseAdmin, Put, HTable, Result} > import org.apache.hadoop.hbase.mapreduce.TableInputFormat > import org.apache.hadoop.hbase.io.ImmutableBytesWritable > val conf = HBaseConfiguration.create() > conf.set("hbase.zookeeper.quorum", "localhost") > conf.set(TableInputFormat.INPUT_TABLE, "test_table") > val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], > classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], > classOf[org.apache.hadoop.hbase.client.Result]) > > val res = hBaseRDD.take(1) > val rs = res(0)._2 > val kv = rs.raw > for(keyvalue <- kv) println("rowkey:"+ new String(keyvalue.getRow)+ " > cf:"+new String(keyvalue.getFamily()) + " column:" + new > String(keyvalue.getQualifier) + " " + "value:"+new > String(keyvalue.getValue())) > -------------------------- -------------------------- > -------------------------- -------------------------- > By the way, I've already put hive-site.xml and hbase-site.xml into the > HADOOP_CONF_DIR and $SPARK_HOME/conf(which is acually > $KYLIN_HOME/spark/conf), and I also set spark.driver.extraClassPath in > spark-defaults.conf to attachs some related jars(hbase-client.jar, > hbase-common.jar and so on). > I don't know why, anyone could give me some advice? > 2017-06-21 > > skyyws > > > > 发件人：ShaoFeng Shi <[email protected]> > 发送时间：2017-06-20 15:13 > 主题：Re: Re: Build sample error with spark on kylin 2.0.0 > 收件人："dev"<[email protected]> > 抄送： > > Or you can check whether there is old hadoop jars on your cluster, > according to https://issues.apache.org/jira/browse/HADOOP-11064 > > > 2017-06-20 9:33 GMT+08:00 skyyws <[email protected]>: > > > No, I deploy kylin on linux, this is my machine info: > > -------------------------- > > 3.2.0-4-amd64 #1 SMP Debian 3.2.82-1 x86_64 GNU/Linux > > ------------------------- > > > > 2017-06-20 > > > > skyyws > > > > > > > > 发件人：ShaoFeng Shi <[email protected]> > > 发送时间：2017-06-20 00:10 > > 主题：Re: Build sample error with spark on kylin 2.0.0 > > 收件人："dev"<[email protected]> > > 抄送： > > > > Are you running Kylin on windows? If yes, check: > > https://stackoverflow.com/questions/33211599/hadoop- > > error-on-windows-java-lang-unsatisfiedlinkerror > > > > 2017-06-19 21:55 GMT+08:00 skyyws <[email protected]>: > > > > > Hi all, > > > I met an error when using spark engine build kylin sample on step > "Build > > > Cube with Spark", here is the exception log: > > > ------------------------------------------------------------ > > > ----------------------------- > > > Exception in thread "main" java.lang.UnsatisfiedLinkError: > > > org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteAr > > > ray(II[BI[BIILjava/lang/String;JZ)V > > > at org.apache.hadoop.util.NativeCrc32. > > > nativeComputeChunkedSumsByteArray(Native Method) > > > at org.apache.hadoop.util.NativeCrc32. > > > calculateChunkedSumsByteArray(NativeCrc32.java:86) > > > at org.apache.hadoop.util.DataChecksum.calculateChunkedSums( > > > DataChecksum.java:430) > > > at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks( > > > FSOutputSummer.java:202) > > > at org.apache.hadoop.fs.FSOutputSummer.write1( > > > FSOutputSummer.java:124) > > > at org.apache.hadoop.fs.FSOutputSummer.write( > > > FSOutputSummer.java:110) > > > at org.apache.hadoop.fs.FSDataOutputStream$ > PositionCache.write( > > > FSDataOutputStream.java:58) > > > at java.io.DataOutputStream.write(DataOutputStream.java:107) > > > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:80) > > > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52) > > > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112) > > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366) > > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338) > > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289) > > > at org.apache.spark.deploy.yarn.Client.copyFileToRemote( > > > Client.scala:317) > > > at org.apache.spark.deploy.yarn.Client.org$apache$spark$ > > > deploy$yarn$Client$$distribute$1(Client.scala:407) > > > at org.apache.spark.deploy.yarn.Client$$anonfun$ > > > prepareLocalResources$5.apply(Client.scala:446) > > > at org.apache.spark.deploy.yarn.Client$$anonfun$ > > > prepareLocalResources$5.apply(Client.scala:444) > > > at scala.collection.immutable.List.foreach(List.scala:318) > > > at org.apache.spark.deploy.yarn.Client.prepareLocalResources( > > > Client.scala:444) > > > at org.apache.spark.deploy.yarn.Client. > > > createContainerLaunchContext(Client.scala:727) > > > at org.apache.spark.deploy.yarn.Client.submitApplication( > > > Client.scala:142) > > > at org.apache.spark.scheduler.cluster. > > YarnClientSchedulerBackend. > > > start(YarnClientSchedulerBackend.scala:57) > > > at org.apache.spark.scheduler.TaskSchedulerImpl.start( > > > TaskSchedulerImpl.scala:144) > > > at org.apache.spark.SparkContext. > <init>(SparkContext.scala:530) > > > at org.apache.spark.api.java.JavaSparkContext.<init>( > > > JavaSparkContext.scala:59) > > > at org.apache.kylin.engine.spark.SparkCubingByLayer.execute( > > > SparkCubingByLayer.java:150) > > > at org.apache.kylin.common.util.AbstractApplication.execute( > > > AbstractApplication.java:37) > > > at org.apache.kylin.common.util.SparkEntry.main(SparkEntry. > > > java:44) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at sun.reflect.NativeMethodAccessorImpl.invoke( > > > NativeMethodAccessorImpl.java:57) > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:606) > > > at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ > > > deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) > > > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1( > > > SparkSubmit.scala:181) > > > at org.apache.spark.deploy.SparkSubmit$.submit( > > > SparkSubmit.scala:206) > > > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit. > > > scala:121) > > > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > > 17/06/19 21:22:06 INFO storage.DiskBlockManager: Shutdown hook called > > > 17/06/19 21:22:06 INFO util.ShutdownHookManager: Shutdown hook called > > > 17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory > > > /tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28 > > > 17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory > > > /tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28/httpd- > > > 9bcb9a5d-569f-4f28-ad89-038a9020eda8 > > > 17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory > > > /tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28/userFiles- > > > 2e9ff265-3d37-40e0-8894-6fd4d1a3ad8b > > > > > > at org.apache.kylin.common.util.CliCommandExecutor.execute( > > > CliCommandExecutor.java:92) > > > at org.apache.kylin.engine.spark.SparkExecutable.doWork( > > > SparkExecutable.java:124) > > > at org.apache.kylin.job.execution.AbstractExecutable. > > > execute(AbstractExecutable.java:124) > > > at org.apache.kylin.job.execution.DefaultChainedExecutable. > > doWork( > > > DefaultChainedExecutable.java:64) > > > at org.apache.kylin.job.execution.AbstractExecutable. > > > execute(AbstractExecutable.java:124) > > > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$ > > > JobRunner.run(DefaultScheduler.java:142) > > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > > ThreadPoolExecutor.java:1145) > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > ThreadPoolExecutor.java:615) > > > at java.lang.Thread.run(Thread.java:745) > > > > > > ------------------------------------------------------------ > > > ----------------------------- > > > I can use the kylin in-build spark-shell to do some operations like: > > > ------------------------------------------------------------ > > > ----------------------------- > > > var textFile = sc.textFile("hdfs://xxxx/xxxx/README.md") > > > textFile.count() > > > textFile.first() > > > textFile.filter(line => line.contains("hello")).count() > > > ------------------------------------------------------------ > > > ----------------------------- > > > Here is the env info: > > > kylin version is 2.0.0 > > > hadoop version is 2.7.* > > > spark version is 1.6.* > > > ------------------------------------------------------------ > > > ----------------------------- > > > Anyone can help me?THX > > > > > > > > > 2017-06-19 > > > skyyws > > > > > > > > > > -- > > Best regards, > > > > Shaofeng Shi 史少锋 > > > > > > -- > Best regards, > > Shaofeng Shi 史少锋 > -- Best regards, Shaofeng Shi 史少锋
