Hi all,
I met an error when using spark engine build kylin sample on step "Build Cube
with Spark", here is the exception log:
-----------------------------------------------------------------------------------------
Exception in thread "main" java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V
at
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native
Method)
at
org.apache.hadoop.util.NativeCrc32.calculateChunkedSumsByteArray(NativeCrc32.java:86)
at
org.apache.hadoop.util.DataChecksum.calculateChunkedSums(DataChecksum.java:430)
at
org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:202)
at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:124)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:110)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:80)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at
org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:317)
at
org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:407)
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$5.apply(Client.scala:446)
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$5.apply(Client.scala:444)
at scala.collection.immutable.List.foreach(List.scala:318)
at
org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:444)
at
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:727)
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:142)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:530)
at
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
at
org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:150)
at
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/06/19 21:22:06 INFO storage.DiskBlockManager: Shutdown hook called
17/06/19 21:22:06 INFO util.ShutdownHookManager: Shutdown hook called
17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28
17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28/httpd-9bcb9a5d-569f-4f28-ad89-038a9020eda8
17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28/userFiles-2e9ff265-3d37-40e0-8894-6fd4d1a3ad8b
at
org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:92)
at
org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:124)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
-----------------------------------------------------------------------------------------
I can use the kylin in-build spark-shell to do some operations like:
-----------------------------------------------------------------------------------------
var textFile = sc.textFile("hdfs://xxxx/xxxx/README.md")
textFile.count()
textFile.first()
textFile.filter(line => line.contains("hello")).count()
-----------------------------------------------------------------------------------------
Here is the env info:
kylin version is 2.0.0
hadoop version is 2.7.*
spark version is 1.6.*
-----------------------------------------------------------------------------------------
Anyone can help me?THX
2017-06-19
skyyws
