[
https://issues.apache.org/jira/browse/KYLIN-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032766#comment-16032766
]
Shaofeng SHI commented on KYLIN-2547:
-------------------------------------
kaisen, I checked the change. Here are some comments:
1) Adding lock in job steps is not a good implementation in my opinion; as by
defaut Kylin only use a single job engine and no zk lock is used. I understand
you have the need that building segments in parallel in multiple job engines,
while that is what the framework (distributed scheduler) should handle, instead
of making each step to sync by themselves.
2) Is the "getCubeFromStore()" necessary? Is there the chance that the cache
in CubeManager is out of date?
3) Can we keep "keepCubeRetention()" and "mergeCubeSegment()" in CubeService
(not move to UpdateCubeInfoStep)? so the logics can be reused from other
places.
Thanks!
> Fix multi-process concurrence bug in UpdateCubeInfoXXStep
> ---------------------------------------------------------
>
> Key: KYLIN-2547
> URL: https://issues.apache.org/jira/browse/KYLIN-2547
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
> Affects Versions: v2.0.0
> Reporter: kangkaisen
> Assignee: kangkaisen
> Priority: Minor
>
> There is a minor bug in "Update Cube Info" step when build a cube in
> distributed env, which will make this job failed, but the next auto merge
> job won't be affected and will succeed. So the cube data still is consistent
> and correct.
> {code:java}
> Caused by: java.lang.IllegalStateException: Segments overlap:
> waimai_dolphin_topic_flow_activity_expose_food_d_cube[20170405000000_20170412000000]
> and
> waimai_dolphin_topic_flow_activity_expose_food_d_cube[20170405000000_20170412000000]
> at org.apache.kylin.cube.CubeValidator.validate(CubeValidator.java:85)
> at
> org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:359)
> at
> org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:386)
> at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:302)
> at
> org.apache.kylin.cube.CubeManager.mergeSegments(CubeManager.java:533)
> at
> org.apache.kylin.rest.service.CubeService.mergeCubeSegment(CubeService.java:635)
> at
> org.apache.kylin.rest.service.CubeService.updateOnNewSegmentReady(CubeService.java:587)
> at
> org.apache.kylin.rest.service.CubeServiceFastClassBySpringCGLIB17a07c0e.invoke()
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.CglibAopProxyDynamicAdvisedInterceptor.intercept(CglibAopProxy.java:629)
> at
> org.apache.kylin.rest.service.CubeServiceEnhancerBySpringCGLIB$$c6fabb3f.updateOnNewSegmentReady()
> at
> org.apache.kylin.rest.service.CacheService.rebuildCubeCache(CacheService.java:237)
> at
> org.apache.kylin.rest.service.CacheService.access$000(CacheService.java:62)
> at
> org.apache.kylin.rest.service.CacheService$1.afterCubeUpdate(CacheService.java:86)
> at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:305)
> at
> org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:735)
> at
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:62)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
