[
https://issues.apache.org/jira/browse/KYLIN-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015147#comment-16015147
]
kangkaisen commented on KYLIN-2547:
-----------------------------------
This is the commit on KYLIN-2547 branch:
https://github.com/apache/kylin/commit/d0f7f1882ef02e84c112ea358152e4976ee081e0.
This commit has passed the IT.
This commit has two points:
1. move {{keepCubeRetention}} and {{mergeCubeSegment}} to
{{UpdateCubeInfoStep}} in order to ensure only the jobServer building the
segment could check keepCubeRetention and auto merge.
2. Add a lock for {{UpdateCubeInfoStep}} to make the this step is linearizable
and avoid retry.
> Fix multi-process concurrence bug in UpdateCubeInfoXXStep
> ---------------------------------------------------------
>
> Key: KYLIN-2547
> URL: https://issues.apache.org/jira/browse/KYLIN-2547
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
> Affects Versions: v2.0.0
> Reporter: kangkaisen
> Assignee: kangkaisen
> Priority: Minor
>
> There is a minor bug in "Update Cube Info" step when build a cube in
> distributed env, which will make this job failed, but the next auto merge
> job won't be affected and will succeed. So the cube data still is consistent
> and correct.
> {code:java}
> Caused by: java.lang.IllegalStateException: Segments overlap:
> waimai_dolphin_topic_flow_activity_expose_food_d_cube[20170405000000_20170412000000]
> and
> waimai_dolphin_topic_flow_activity_expose_food_d_cube[20170405000000_20170412000000]
> at org.apache.kylin.cube.CubeValidator.validate(CubeValidator.java:85)
> at
> org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:359)
> at
> org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:386)
> at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:302)
> at
> org.apache.kylin.cube.CubeManager.mergeSegments(CubeManager.java:533)
> at
> org.apache.kylin.rest.service.CubeService.mergeCubeSegment(CubeService.java:635)
> at
> org.apache.kylin.rest.service.CubeService.updateOnNewSegmentReady(CubeService.java:587)
> at
> org.apache.kylin.rest.service.CubeServiceFastClassBySpringCGLIB17a07c0e.invoke()
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.CglibAopProxyDynamicAdvisedInterceptor.intercept(CglibAopProxy.java:629)
> at
> org.apache.kylin.rest.service.CubeServiceEnhancerBySpringCGLIB$$c6fabb3f.updateOnNewSegmentReady()
> at
> org.apache.kylin.rest.service.CacheService.rebuildCubeCache(CacheService.java:237)
> at
> org.apache.kylin.rest.service.CacheService.access$000(CacheService.java:62)
> at
> org.apache.kylin.rest.service.CacheService$1.afterCubeUpdate(CacheService.java:86)
> at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:305)
> at
> org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:735)
> at
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:62)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
