No, I deploy kylin on linux, this is my machine info: -------------------------- 3.2.0-4-amd64 #1 SMP Debian 3.2.82-1 x86_64 GNU/Linux -------------------------
2017-06-20 skyyws 发件人：ShaoFeng Shi <[email protected]> 发送时间：2017-06-20 00:10 主题：Re: Build sample error with spark on kylin 2.0.0 收件人："dev"<[email protected]> 抄送： Are you running Kylin on windows? If yes, check: https://stackoverflow.com/questions/33211599/hadoop-error-on-windows-java-lang-unsatisfiedlinkerror 2017-06-19 21:55 GMT+08:00 skyyws <[email protected]>: > Hi all, > I met an error when using spark engine build kylin sample on step "Build > Cube with Spark", here is the exception log: > ------------------------------------------------------------ > ----------------------------- > Exception in thread "main" java.lang.UnsatisfiedLinkError: > org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteAr > ray(II[BI[BIILjava/lang/String;JZ)V > at org.apache.hadoop.util.NativeCrc32. > nativeComputeChunkedSumsByteArray(Native Method) > at org.apache.hadoop.util.NativeCrc32. > calculateChunkedSumsByteArray(NativeCrc32.java:86) > at org.apache.hadoop.util.DataChecksum.calculateChunkedSums( > DataChecksum.java:430) > at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks( > FSOutputSummer.java:202) > at org.apache.hadoop.fs.FSOutputSummer.write1( > FSOutputSummer.java:124) > at org.apache.hadoop.fs.FSOutputSummer.write( > FSOutputSummer.java:110) > at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write( > FSDataOutputStream.java:58) > at java.io.DataOutputStream.write(DataOutputStream.java:107) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:80) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289) > at org.apache.spark.deploy.yarn.Client.copyFileToRemote( > Client.scala:317) > at org.apache.spark.deploy.yarn.Client.org$apache$spark$ > deploy$yarn$Client$$distribute$1(Client.scala:407) > at org.apache.spark.deploy.yarn.Client$$anonfun$ > prepareLocalResources$5.apply(Client.scala:446) > at org.apache.spark.deploy.yarn.Client$$anonfun$ > prepareLocalResources$5.apply(Client.scala:444) > at scala.collection.immutable.List.foreach(List.scala:318) > at org.apache.spark.deploy.yarn.Client.prepareLocalResources( > Client.scala:444) > at org.apache.spark.deploy.yarn.Client. > createContainerLaunchContext(Client.scala:727) > at org.apache.spark.deploy.yarn.Client.submitApplication( > Client.scala:142) > at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend. > start(YarnClientSchedulerBackend.scala:57) > at org.apache.spark.scheduler.TaskSchedulerImpl.start( > TaskSchedulerImpl.scala:144) > at org.apache.spark.SparkContext.<init>(SparkContext.scala:530) > at org.apache.spark.api.java.JavaSparkContext.<init>( > JavaSparkContext.scala:59) > at org.apache.kylin.engine.spark.SparkCubingByLayer.execute( > SparkCubingByLayer.java:150) > at org.apache.kylin.common.util.AbstractApplication.execute( > AbstractApplication.java:37) > at org.apache.kylin.common.util.SparkEntry.main(SparkEntry. > java:44) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ > deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1( > SparkSubmit.scala:181) > at org.apache.spark.deploy.SparkSubmit$.submit( > SparkSubmit.scala:206) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit. > scala:121) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > 17/06/19 21:22:06 INFO storage.DiskBlockManager: Shutdown hook called > 17/06/19 21:22:06 INFO util.ShutdownHookManager: Shutdown hook called > 17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory > /tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28 > 17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory > /tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28/httpd- > 9bcb9a5d-569f-4f28-ad89-038a9020eda8 > 17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory > /tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28/userFiles- > 2e9ff265-3d37-40e0-8894-6fd4d1a3ad8b > > at org.apache.kylin.common.util.CliCommandExecutor.execute( > CliCommandExecutor.java:92) > at org.apache.kylin.engine.spark.SparkExecutable.doWork( > SparkExecutable.java:124) > at org.apache.kylin.job.execution.AbstractExecutable. > execute(AbstractExecutable.java:124) > at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork( > DefaultChainedExecutable.java:64) > at org.apache.kylin.job.execution.AbstractExecutable. > execute(AbstractExecutable.java:124) > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$ > JobRunner.run(DefaultScheduler.java:142) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > ------------------------------------------------------------ > ----------------------------- > I can use the kylin in-build spark-shell to do some operations like: > ------------------------------------------------------------ > ----------------------------- > var textFile = sc.textFile("hdfs://xxxx/xxxx/README.md") > textFile.count() > textFile.first() > textFile.filter(line => line.contains("hello")).count() > ------------------------------------------------------------ > ----------------------------- > Here is the env info: > kylin version is 2.0.0 > hadoop version is 2.7.* > spark version is 1.6.* > ------------------------------------------------------------ > ----------------------------- > Anyone can help me?THX > > > 2017-06-19 > skyyws -- Best regards, Shaofeng Shi 史少锋
