weibin0516 created KYLIN-4339:
---------------------------------
Summary: Extract Fact Table Distinct Columns fail due to no kylin
installed on worker node
Key: KYLIN-4339
URL: https://issues.apache.org/jira/browse/KYLIN-4339
Project: Kylin
Issue Type: Bug
Affects Versions: v3.0.0
Reporter: weibin0516
Assignee: weibin0516
After set kylin.engine.spark-fact-distinct to true, using spark engine to build
cube will fail, error message as follow
{code:java}
2020-01-13 22:19:23 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2020-01-13 22:19:23 INFO
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 -
OutputCommitCoordinator stopped!
2020-01-13 22:19:23 INFO SparkContext:54 - Successfully stopped SparkContext
Exception in thread "main" java.lang.RuntimeException: error execute
org.apache.kylin.engine.spark.SparkFactDistinct. Root cause: Job aborted due to
stage failure: Task 9 in stage 1.0 failed 4 times, most recent failure: Lost
task 9.3 in stage 1.0 (TID 32, sql-gateway-eu95-17.gz00c.test.alipay.net,
executor 7): org.apache.kylin.common.KylinConfigCannotInitException: Didn't
find KYLIN_CONF or KYLIN_HOME, please set one of them
at
org.apache.kylin.common.KylinConfig.getSitePropertiesFile(KylinConfig.java:336)
at
org.apache.kylin.common.KylinConfig.buildSiteOrderedProps(KylinConfig.java:378)
at
org.apache.kylin.common.KylinConfig.buildSiteProperties(KylinConfig.java:358)
at
org.apache.kylin.common.KylinConfig.getInstanceFromEnv(KylinConfig.java:137)
at
org.apache.kylin.dict.CacheDictionary.enableCache(CacheDictionary.java:105)
at
org.apache.kylin.dict.TrieDictionaryForest.initForestCache(TrieDictionaryForest.java:394)
at
org.apache.kylin.dict.TrieDictionaryForest.init(TrieDictionaryForest.java:77)
at
org.apache.kylin.dict.TrieDictionaryForest.<init>(TrieDictionaryForest.java:67)
at
org.apache.kylin.dict.TrieDictionaryForestBuilder.build(TrieDictionaryForestBuilder.java:114)
at
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder.build(DictionaryGenerator.java:312)
at
org.apache.kylin.engine.spark.SparkFactDistinct$MultiOutputFunction.call(SparkFactDistinct.java:774)
at
org.apache.kylin.engine.spark.SparkFactDistinct$MultiOutputFunction.call(SparkFactDistinct.java:650)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:800)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:800)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:756)
{code}
we should put kylin.properties in the execution environment of spark
application (via --files) to fix this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
