[
https://issues.apache.org/jira/browse/KYLIN-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16895768#comment-16895768
]
wangrupeng commented on KYLIN-4060:
-----------------------------------
Path oldPath = Path.getPathWithoutSchemeAndAuthority(new Path(path));
if (fileSystem.exists(oldPath)) {
fileSystem.delete(oldPath, true);
logger.debug("HDFS path " + oldPath + " is dropped.");
output.append("HDFS path " + oldPath + " is dropped.\n");
} else {
logger.debug("HDFS path " + oldPath + " not exists.");
output.append("HDFS path " + oldPath + " not exists.\n");
}
> "Garbage Collection on HDFS" step failed because of hdfs path not exists
> ------------------------------------------------------------------------
>
> Key: KYLIN-4060
> URL: https://issues.apache.org/jira/browse/KYLIN-4060
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v2.4.1
> Reporter: WangSheng
> Priority: Minor
>
> We found a bug recently when we used streaming cube on last job step "Garbage
> Collection on HDFS", the proplem is as blow:
>
> {code:java}
> Drop HDFS path on FileSystem: "hdfs://kylin-cluster"
> HDFS path
> /user/kylin/kylin_home/kylin_metadata/kylin-03c04b31-5d40-441a-a0df-289f5977b733/cube_test/fact_distinct_columns
> not exists.
> File
> /user/kylin/kylin_home/kylin_metadata/kylin-03c04b31-5d40-441a-a0df-289f5977b733/cube_test
> does not exist.
> {code}
> When I check the code and log, I found that the main reason is:
>
> # A build job first submitted, and on step "Update Cube Info", segment
> became "READY";
> # Then a merge job submitted automatically by kylin, include segment on
> step1. The merge job finished quickly, and deleted input segments hdfs path;
> # After merge job finished, the build job continue build, "Hive Cleanup" and
> "Garbage Collection on HBase", failed at last step because the hdfs path is
> deleted on step2.
> Our version is 2.4.x, I'm not sure this if this bug fixed on latest 2.6.x
> version. If not, please assign this Jira to me, thanks!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
