[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[hejian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822832#comment-16822832
 ] 

hejian commented on KYLIN-2849:
---

[~Shaofengshi] Shaofeng, as desc in KYLIN-3964

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: scott.zhai
>Assignee: Dong Li
>Priority: Major
>  Labels: scope
> Fix For: v2.3.0
>
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[hejian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822829#comment-16822829
 ] 

hejian commented on KYLIN-2849:
---

ok thanks Shaofeng

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: scott.zhai
>Assignee: Dong Li
>Priority: Major
>  Labels: scope
> Fix For: v2.3.0
>
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[Shaofeng SHI (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822816#comment-16822816
 ] 

Shaofeng SHI commented on KYLIN-2849:
-

[~hejian999] hejian, please open a new Jira with the necessary information, 
like your REST request, the error trace in kylin.log, etc. Thanks!

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: scott.zhai
>Assignee: Dong Li
>Priority: Major
>  Labels: scope
> Fix For: v2.3.0
>
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[hejian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821854#comment-16821854
 ] 

hejian commented on KYLIN-2849:
---

i use the kylin-2.4.0, and this issue is still there

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: scott.zhai
>Assignee: Dong Li
>Priority: Major
>  Labels: scope
> Fix For: v2.3.0
>
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[Billy Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16337327#comment-16337327
 ] 

Billy Liu commented on KYLIN-2849:
--

Hi [~lidong_sjtu] Do you think KYLIN-2849 has been fixed in KYLIN-3004. 

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: 翟玉勇
>Assignee: Dong Li
>Priority: Major
>  Labels: scope
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[Vladimir Prus (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16335548#comment-16335548
 ] 

Vladimir Prus commented on KYLIN-2849:
--

I have exactly the same problem. Segment merge job has failed, creating a new 
segment in the 'new' state, and now I cannot do anything, getting 'segments 
overlap' error, and I cannot delete the segment either. Would it be possible to 
add 'force' flag and also support specifying segment by uuid? Otherwise, the 
only solution is to stop everything, and either poke at metadata directly, or 
do save/reload.

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: 翟玉勇
>Assignee: Dong Li
>Priority: Major
>  Labels: scope
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[Pan, Julian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331876#comment-16331876
 ] 

Pan, Julian commented on KYLIN-2849:


I'm not sure whether it is duplicate with 
https://issues.apache.org/jira/browse/KYLIN-3004, which is resolved in version 
2.3.0

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: 翟玉勇
>Assignee: Dong Li
>Priority: Major
>  Labels: scope
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16265957#comment-16265957
 ] 

liyang commented on KYLIN-2849:
---

Looked code, the normal path is segment is deleted when an error job is 
discarded. So here we are talking about error scenario, that something went 
wrong and the orphan segment is left behind after job is discarded.

How about adding a 'force' parameter on the delete segment API to bypass the 
check? Also the API could accept UUID in addition to segment name to identify 
an error segment precisely.

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: 翟玉勇
>Assignee: Dong Li
>  Labels: scope
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[explorer (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258383#comment-16258383
 ] 

explorer commented on KYLIN-2849:
-

I also encountered this problem, waiting for solution.

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: 翟玉勇
>Assignee: Dong Li
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[Dmitry (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16253450#comment-16253450
 ] 

Dmitry commented on KYLIN-2849:
---

Duplicate segments can appear when something wrong happens with job. In our 
case there were a job error. And after its discarding we got two segments with 
the same names. The second segment has status = NEW:
{
"uuid": "4a6f9098-d95c-4cca-8865-6efe2fc1e2f8",
"name": "2017103000_2017121100",
"storage_location_identifier": "KYLIN_VTYLURM5N4",
"date_range_start": 150932160,
"date_range_end": 151295040,
"source_offset_start": 0,
"source_offset_end": 0,
"status": "NEW",
"size_kb": 0,
"input_records": 0,
"input_records_size": 0,
"last_build_time": 0,
"last_build_job_id": null,
"create_time_utc": 1510412403197,
"cuboid_shard_nums": {},
"total_shards": 0,
"blackout_cuboids": [],
"binary_signature": null,
"dictionaries": null,
"snapshots": null,
"rowkey_stats": []
}

And when we try to delete it, we get an error:
{{"msg": "Cannot delete segment '2017103000_2017121100' as its status 
is not READY. Discard the on-going job for it.".}} 
But there are no any jobs running. And there are no HBase table called 
KYLIN_VTYLURM5N4. Also it's not possible to REFRESH this segment. It says about 
Segments overlapping.
I guess it's related to KYLIN-2962 issue.
It would be great to be able to delete such segments from GUI or via RestfulAPI.

Billy Liu, what data should be removed from metadata to fix this issue? 

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: 翟玉勇
>Assignee: Dong Li
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[Billy Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156508#comment-16156508
 ] 

Billy Liu commented on KYLIN-2849:
--

Could you describe how the duplicate segments are generated? If there is a bug, 
we should fix it first. 
For the existing duplicated segments, you could export the metadata and remove 
the wrong segments manually, and import the metadata back. 

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: 翟玉勇
>Assignee: Dong Li
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[JIRA]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156422#comment-16156422
 ] 

翟玉勇 commented on KYLIN-2849:


how to solution duplicate segment？

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: 翟玉勇
>Assignee: Dong Li
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged

[Billy Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156298#comment-16156298
 ] 

Billy Liu commented on KYLIN-2849:
--

The Segment delete is only for the start and end segment, it's by design. For 
more info, check https://issues.apache.org/jira/browse/KYLIN-1540

> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: 翟玉勇
>Assignee: Dong Li
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE 
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300";
>   -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type: 
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last 
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
>  Cannot delete segment '2017082200_2017082300' as it is neither the 
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName) 
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) && 
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() - 
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" + 
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" + 
> segmentName + "' as its status is not READY. Discard the on-going job for 
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return CubeManager.getInstance(getConfig()).updateCube(update);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)