[jira] [Comment Edited] (KYLIN-4162) After drop the build task on the monitor page, subsequent segments cannot be constructed.
[
https://issues.apache.org/jira/browse/KYLIN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925772#comment-16925772
]
Xiaoxiang Yu edited comment on KYLIN-4162 at 9/9/19 3:26 PM:
-
If a discard job found in Coordinator, it maybe be detected but dropped .
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1186
{code:java}
else if (ExecutableState.DISCARDED.equals(jobState)) {
// if the job has been discard manually, just think that the segment is not
in building
logger.info("job:{} is discard, reset the job state in metaStore", jobId);
SegmentBuildState.BuildState state = new SegmentBuildState.BuildState();
state.setBuildStartTime(0);
state.setState(SegmentBuildState.BuildState.State.WAIT);
state.setJobId(cubingJob.getId());
streamMetadataStore.updateSegmentBuildState(cubeName,
segmentState.getSegmentName(), state);
segmentState.setState(state);
}
{code}
In above code, if coordinator find the state of job is *DISCARDED*, coordinator
will change the state of current segment in stream metadata from *building* to
*wait*, and try to add to *result* to mark it need to be rebuilt.
But, before add the segment the *result*, coordinator will first check it has
enough quota. If not, coordinator won't add it into *result*, thus that segment
will be lost.
When check in next time, it will be ignore because the state is *wait*.
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1160
{code:java}
if (segmentState.isInBuilding()) {
{code}
Did I understand it in right way? Happy to hear your reply.
was (Author: hit_lacus):
If a discard job found in Coordinator, it will be detected at
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1186
.
{code:java}
else if (ExecutableState.DISCARDED.equals(jobState)) {
// if the job has been discard manually, just think that the segment is not
in building
logger.info("job:{} is discard, reset the job state in metaStore", jobId);
SegmentBuildState.BuildState state = new SegmentBuildState.BuildState();
state.setBuildStartTime(0);
state.setState(SegmentBuildState.BuildState.State.WAIT);
state.setJobId(cubingJob.getId());
streamMetadataStore.updateSegmentBuildState(cubeName,
segmentState.getSegmentName(), state);
segmentState.setState(state);
}
{code}
In above code, if coordinator find the state of job is *DISCARDED*, coordinator
will change the entry in stream metadata from building to wait, and try to add
to *result* to mark it need to be rebuilt.
But, before add the segment the *result*, coordinator will first check it has
enough quota. If not, coordinator won't add it into *result*, thus that segment
will be lost.
When check in next time, it will be ignore because the state is *wait*.
{code:java}
if (segmentState.isInBuilding()) {
{code}
> After drop the build task on the monitor page, subsequent segments cannot be
> constructed.
> -
>
> Key: KYLIN-4162
> URL: https://issues.apache.org/jira/browse/KYLIN-4162
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
>Affects Versions: v3.0.0-alpha2
>Reporter: Yaqian Zhang
>Priority: Major
> Fix For: Future
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (KYLIN-4162) After drop the build task on the monitor page, subsequent segments cannot be constructed.
[
https://issues.apache.org/jira/browse/KYLIN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925772#comment-16925772
]
Xiaoxiang Yu edited comment on KYLIN-4162 at 9/9/19 3:32 PM:
-
Hi, [~zhangyaqian], following is my understanding, did I understand it in right
way? Happy to hear your reply.
-
-
If a discard job be found in Coordinator, it maybe dropped because of no quota.
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1186
{code:java}
else if (ExecutableState.DISCARDED.equals(jobState)) {
// if the job has been discard manually, just think that the segment is not
in building
logger.info("job:{} is discard, reset the job state in metaStore", jobId);
SegmentBuildState.BuildState state = new SegmentBuildState.BuildState();
state.setBuildStartTime(0);
state.setState(SegmentBuildState.BuildState.State.WAIT);
state.setJobId(cubingJob.getId());
streamMetadataStore.updateSegmentBuildState(cubeName,
segmentState.getSegmentName(), state);
segmentState.setState(state);
}
{code}
In above code, if coordinator find the state of job is *DISCARDED*, coordinator
will change the state of current segment in stream metadata from *building* to
*wait*, and try to add to *result* to mark it as it need to be resubmit a new
build job.
But, before add the segment the *result*, coordinator will first check it has
enough quota. If not, coordinator won't add it into *result*, thus that segment
will be lost.
When check in the next time, it will be ignored because the state is *wait*.
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1160
{code:java}
if (segmentState.isInBuilding()) {
{code}
was (Author: hit_lacus):
If a discard job found in Coordinator, it maybe be detected but dropped .
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1186
{code:java}
else if (ExecutableState.DISCARDED.equals(jobState)) {
// if the job has been discard manually, just think that the segment is not
in building
logger.info("job:{} is discard, reset the job state in metaStore", jobId);
SegmentBuildState.BuildState state = new SegmentBuildState.BuildState();
state.setBuildStartTime(0);
state.setState(SegmentBuildState.BuildState.State.WAIT);
state.setJobId(cubingJob.getId());
streamMetadataStore.updateSegmentBuildState(cubeName,
segmentState.getSegmentName(), state);
segmentState.setState(state);
}
{code}
In above code, if coordinator find the state of job is *DISCARDED*, coordinator
will change the state of current segment in stream metadata from *building* to
*wait*, and try to add to *result* to mark it need to be rebuilt.
But, before add the segment the *result*, coordinator will first check it has
enough quota. If not, coordinator won't add it into *result*, thus that segment
will be lost.
When check in next time, it will be ignore because the state is *wait*.
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1160
{code:java}
if (segmentState.isInBuilding()) {
{code}
Did I understand it in right way? Happy to hear your reply.
> After drop the build task on the monitor page, subsequent segments cannot be
> constructed.
> -
>
> Key: KYLIN-4162
> URL: https://issues.apache.org/jira/browse/KYLIN-4162
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
>Affects Versions: v3.0.0-alpha2
>Reporter: Yaqian Zhang
>Priority: Major
> Fix For: Future
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
