Are you running Kylin on windows? If yes, check:
https://stackoverflow.com/questions/33211599/hadoop-error-on-windows-java-lang-unsatisfiedlinkerror
2017-06-19 21:55 GMT+08:00 skyyws <sky...@163.com>:
> Hi all,
> I met an error when using spark engine build kylin sample on step "Build
> Cube with Spark", here is the exception log:
> ------------------------------------------------------------
> -----------------------------
> Exception in thread "main" java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteAr
> ray(II[BI[BIILjava/lang/String;JZ)V
> at org.apache.hadoop.util.NativeCrc32.
> nativeComputeChunkedSumsByteArray(Native Method)
> at org.apache.hadoop.util.NativeCrc32.
> calculateChunkedSumsByteArray(NativeCrc32.java:86)
> at org.apache.hadoop.util.DataChecksum.calculateChunkedSums(
> DataChecksum.java:430)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(
> FSOutputSummer.java:202)
> at org.apache.hadoop.fs.FSOutputSummer.write1(
> FSOutputSummer.java:124)
> at org.apache.hadoop.fs.FSOutputSummer.write(
> FSOutputSummer.java:110)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(
> FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:80)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
> at org.apache.spark.deploy.yarn.Client.copyFileToRemote(
> Client.scala:317)
> at org.apache.spark.deploy.yarn.Client.org$apache$spark$
> deploy$yarn$Client$$distribute$1(Client.scala:407)
> at org.apache.spark.deploy.yarn.Client$$anonfun$
> prepareLocalResources$5.apply(Client.scala:446)
> at org.apache.spark.deploy.yarn.Client$$anonfun$
> prepareLocalResources$5.apply(Client.scala:444)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at org.apache.spark.deploy.yarn.Client.prepareLocalResources(
> Client.scala:444)
> at org.apache.spark.deploy.yarn.Client.
> createContainerLaunchContext(Client.scala:727)
> at org.apache.spark.deploy.yarn.Client.submitApplication(
> Client.scala:142)
> at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.
> start(YarnClientSchedulerBackend.scala:57)
> at org.apache.spark.scheduler.TaskSchedulerImpl.start(
> TaskSchedulerImpl.scala:144)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:530)
> at org.apache.spark.api.java.JavaSparkContext.<init>(
> JavaSparkContext.scala:59)
> at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(
> SparkCubingByLayer.java:150)
> at org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:37)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.
> java:44)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
> SparkSubmit.scala:181)
> at org.apache.spark.deploy.SparkSubmit$.submit(
> SparkSubmit.scala:206)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.
> scala:121)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> 17/06/19 21:22:06 INFO storage.DiskBlockManager: Shutdown hook called
> 17/06/19 21:22:06 INFO util.ShutdownHookManager: Shutdown hook called
> 17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory
> /tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28
> 17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory
> /tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28/httpd-
> 9bcb9a5d-569f-4f28-ad89-038a9020eda8
> 17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory
> /tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28/userFiles-
> 2e9ff265-3d37-40e0-8894-6fd4d1a3ad8b
>
> at org.apache.kylin.common.util.CliCommandExecutor.execute(
> CliCommandExecutor.java:92)
> at org.apache.kylin.engine.spark.SparkExecutable.doWork(
> SparkExecutable.java:124)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:64)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> JobRunner.run(DefaultScheduler.java:142)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> ------------------------------------------------------------
> -----------------------------
> I can use the kylin in-build spark-shell to do some operations like:
> ------------------------------------------------------------
> -----------------------------
> var textFile = sc.textFile("hdfs://xxxx/xxxx/README.md")
> textFile.count()
> textFile.first()
> textFile.filter(line => line.contains("hello")).count()
> ------------------------------------------------------------
> -----------------------------
> Here is the env info:
> kylin version is 2.0.0
> hadoop version is 2.7.*
> spark version is 1.6.*
> ------------------------------------------------------------
> -----------------------------
> Anyone can help me?THX
>
>
> 2017-06-19
> skyyws
--
Best regards,
Shaofeng Shi 史少锋
