[
https://issues.apache.org/jira/browse/KYLIN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691769#comment-16691769
]
ASF GitHub Bot commented on KYLIN-3666:
---------------------------------------
shaofengshi commented on a change in pull request #347: KYLIN-3666 HDFS
metadata url not be recognized
URL: https://github.com/apache/kylin/pull/347#discussion_r234639228
##########
File path:
engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
##########
@@ -494,6 +494,17 @@ public static KylinConfig
loadKylinConfigFromHdfs(SerializableConfiguration conf
}
public static KylinConfig loadKylinConfigFromHdfs(String uri) {
+ KylinConfig config = loadKylinConfigFromHdfsNotThreadLocal(uri);
+
+ // This is a bad example where the thread local KylinConfig cannot be
auto-closed due to
+ // limitation of MR API. It works because MR task runs its own
process. Do not copy.
+ @SuppressWarnings("unused")
+ SetAndUnsetThreadLocalConfig shouldAutoClose =
KylinConfig.setAndUnsetThreadLocalConfig(config);
+
+ return config;
+ }
+
+ public static KylinConfig loadKylinConfigFromHdfsNotThreadLocal(String
uri) {
Review comment:
The method name is confusing. I think the root cause is the bad sample in
AbstractHadoopJob.loadKylinConfigFromHdfs(), we should fix that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MERGE CUBE Step 2 Update Dictionary Data Throws IllegalStateException
> ---------------------------------------------------------------------
>
> Key: KYLIN-3666
> URL: https://issues.apache.org/jira/browse/KYLIN-3666
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v2.5.0
> Environment: cdh5.11.0
> Reporter: Xi Chen
> Assignee: Chao Long
> Priority: Major
> Fix For: v2.5.3
>
>
> MERGE CUBE job failed in Step 2, throwing such exception:
>
> {code:java}
> java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> is not recognized
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:172)
> at
> org.apache.kylin.common.KylinConfig.createInstanceFromUri(KylinConfig.java:182)
> at
> org.apache.kylin.engine.mr.steps.UpdateDictionaryStep.doWork(UpdateDictionaryStep.java:70)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> looks like a file but it's neither a file nor a directory
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:162)
> ... 9 more
> {code}
> The metadataUrl is on hdfs, but decideUriType() method does not recognize it.
> Thus throwing such exception.
> Both MR and Spark engine are affected.
> Would [~Wayne0101] please have a look? I see it's related to KYLIN-3471
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
