I'm trying to build a cube in Kylin. It has around 10 steps among which Spark
code runs on step 7. At this step, it's throwing the following error:
Running org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable
default.kylin_intermediate_table -output
gs://bucket/kylin/kylin_metadata/kylin-b52a579f-3084-1ced-3544-fdf5b3eccb8a/model/cuboid/
-input
gs://bucket/kylin/kylin_metadata/kylin-b52a579f-3084-1ced-3544-fdf5b3eccb8a/kylin_intermediate_table
-segmentId 9047b119-14ad-ecb8-bb87-c76d255907df -metaUrl
kylin_metadata@hdfs,path=gs://bucket/kylin/kylin_metadata/kylin-b52a579f-3084-1ced-3544-fdf5b3eccb8a/model/metadata
-cubename cube
*Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.kylin.common.persistence.ResourceStore.getResource(Ljava/lang/String;Ljava/lang/Class;Lorg/apache/kylin/common/persistence/Serializer;)Lorg/apache/kylin/common/persistence/RootPersistentEntity;*
at
org.apache.kylin.cube.CubeManager.loadCubeInstance(CubeManager.java:682)
at
org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:671)
at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:128)
at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:97)
at
org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:137)
at
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
The command is:
export HADOOP_CONF_DIR=/etc/hadoop/conf && /usr/lib/spark/bin/spark-submit
--class org.apache.kylin.common.util.SparkEntry --conf
spark.executor.instances=40 --conf spark.yarn.queue=default --conf
spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf
spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
--conf spark.master=yarn --conf
spark.hadoop.yarn.timeline-service.enabled=false --conf
spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf
spark.eventLog.dir=hdfs:///kylin/spark-history --conf
spark.yarn.executor.memoryOverhead=1024 --conf spark.driver.memory=2G
--conf spark.shuffle.service.enabled=true --jars
/etc/kylin/lib/kylin-job-2.6.4.jar /etc/kylin/lib/kylin-job-2.6.4.jar
-className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable
default.kylin_intermediate_sol_v3_new_9047b119_14ad_ecb8_bb87_c76d255907df
-output
gs://bucket/kylin/kylin_metadata/kylin-b52a579f-3084-1ced-3544-fdf5b3eccb8a/model/cuboid/
-input
gs://bucket/kylin/kylin_metadata/kylin-b52a579f-3084-1ced-3544-fdf5b3eccb8a/kylin_intermediate_table
-segmentId 9047b119-14ad-ecb8-bb87-c76d255907df -metaUrl
kylin_metadata@hdfs,path=gs://bucket/kylin/kylin_metadata/kylin-b52a579f-3084-1ced-3544-fdf5b3eccb8a/model/metadata
-cubename cube
at
org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:347)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I checked the *kylin-core-common-2.6.4.jar* file that I placed in the
/usr/lib/spark/jars path. The source code has the getResource() method
implementation in the ResourceStore class:
public final <T extends RootPersistentEntity> T getResource(final String
resPath, final Serializer<T> serializer) throws IOException {
return this.getResource(resPath,
(org.apache.kylin.common.persistence.ContentReader<T>)new
ContentReader((Serializer)serializer));
}
public final <T extends RootPersistentEntity> T getResource(String resPath,
final ContentReader<T> reader) throws IOException {
resPath = this.norm(resPath);
final RawResource res = this.getResourceWithRetry(resPath);
if (res == null) {
return null;
}
return (T)reader.readContent(res);
}
public final RawResource getResource(final String resPath) throws
IOException {
return this.getResourceWithRetry(this.norm(resPath));
}
I'm not sure why Kylin is unable to find this method. Please let me know if
I'm missing something here. Thanks
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
