[
https://issues.apache.org/jira/browse/KYLIN-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385404#comment-15385404
]
hongbin ma commented on KYLIN-1883:
-----------------------------------
You're right, TIME_THREADSHOLD is for the time the DICT_A created.
I'm okay with the reordering, as it aligns with other cleanup jobs like storage
cleanup job.
Though it might reduce the error chance, we should still be aware that this
update won't fix the issue totally.
Maybe we should dig into the reason why DICT_A suddenly becomes "useful"?
The patch looks good, only one concern:
"toDeleteResource.add(ResourceStore.EXECUTE_RESOURCE_ROOT + "/" +
task.getUuid());"
is deleted, why?
> Consensus Problem when running the tool, MetadataCleanupJob
> -----------------------------------------------------------
>
> Key: KYLIN-1883
> URL: https://issues.apache.org/jira/browse/KYLIN-1883
> Project: Kylin
> Issue Type: Bug
> Reporter: Zhong Yanghong
> Assignee: Zhong Yanghong
> Attachments:
> better_solution_for_consensus_issue_of_MetadataCleanupJob.patch
>
>
> When do the cleanup, current strategy is as follows:
> 1. firstly create an referenceSet
> 2. then add items not belonging to the referenceSet to the toDeleteSet
> 3. finally delete those items in the toDeleteSet
> Consensus issue will occur since we cannot make sure that all of the items in
> toDeleteSet are not referenced in case that referenceSet changes during the
> process.
> For example, before the cleanup, SEGMENT_A is deleted and leave a DICT_A
> created at the building step. Then the referenceSet will not include DICT_A.
> After creating the reference set, SEGMENT_B is starting to build. Since
> DICT_A still exists, it can still be referenced by SEGMENT_B. Then DICT_A
> will still be included in the toDeleteSet and will be deleted later. Finally
> SEGMENT_B only owns a reference with no data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
