[jira] [Commented] (KYLIN-3994) StorageCleanupJob may delete data of newly built segment because of cube cache in CubeManager
[ https://issues.apache.org/jira/browse/KYLIN-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872019#comment-16872019 ] ASF subversion and git services commented on KYLIN-3994: Commit 0c1dd10fd98062df65b7b55ba46cba1eda0ff1ff in kylin's branch refs/heads/2.6.x from Liu Shaohui [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=0c1dd10 ] KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager (#633) * KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager > StorageCleanupJob may delete data of newly built segment because of cube > cache in CubeManager > - > > Key: KYLIN-3994 > URL: https://issues.apache.org/jira/browse/KYLIN-3994 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: v2.6.3 > > > In our production cluster, we found that the cube id data of a new-built > segment is deleted by the StorageCleanupJob. > After checking the code of cleanUnusedHdfsFiles in StorageCleanupJob, we > found that there is a bug here: CubeManager read all cube meta in > initiation and cache it for later > listAllCubes operations, the metadata will be out of data after listing the > hdfs working dir. > So the working directory of a finished job may be deleted unexpectedly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3994) StorageCleanupJob may delete data of newly built segment because of cube cache in CubeManager
[ https://issues.apache.org/jira/browse/KYLIN-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872018#comment-16872018 ] ASF subversion and git services commented on KYLIN-3994: Commit 0c1dd10fd98062df65b7b55ba46cba1eda0ff1ff in kylin's branch refs/heads/2.6.x from Liu Shaohui [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=0c1dd10 ] KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager (#633) * KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager > StorageCleanupJob may delete data of newly built segment because of cube > cache in CubeManager > - > > Key: KYLIN-3994 > URL: https://issues.apache.org/jira/browse/KYLIN-3994 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: v2.6.3 > > > In our production cluster, we found that the cube id data of a new-built > segment is deleted by the StorageCleanupJob. > After checking the code of cleanUnusedHdfsFiles in StorageCleanupJob, we > found that there is a bug here: CubeManager read all cube meta in > initiation and cache it for later > listAllCubes operations, the metadata will be out of data after listing the > hdfs working dir. > So the working directory of a finished job may be deleted unexpectedly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3994) StorageCleanupJob may delete data of newly built segment because of cube cache in CubeManager
[ https://issues.apache.org/jira/browse/KYLIN-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852889#comment-16852889 ] ASF GitHub Bot commented on KYLIN-3994: --- nichunen commented on pull request #633: KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager URL: https://github.com/apache/kylin/pull/633 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > StorageCleanupJob may delete data of newly built segment because of cube > cache in CubeManager > - > > Key: KYLIN-3994 > URL: https://issues.apache.org/jira/browse/KYLIN-3994 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: v2.6.3 > > > In our production cluster, we found that the cube id data of a new-built > segment is deleted by the StorageCleanupJob. > After checking the code of cleanUnusedHdfsFiles in StorageCleanupJob, we > found that there is a bug here: CubeManager read all cube meta in > initiation and cache it for later > listAllCubes operations, the metadata will be out of data after listing the > hdfs working dir. > So the working directory of a finished job may be deleted unexpectedly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3994) StorageCleanupJob may delete data of newly built segment because of cube cache in CubeManager
[ https://issues.apache.org/jira/browse/KYLIN-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852890#comment-16852890 ] ASF subversion and git services commented on KYLIN-3994: Commit cfce8bb628e0e4ade4e03ba9d544096859760907 in kylin's branch refs/heads/master from Liu Shaohui [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=cfce8bb ] KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager (#633) * KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager > StorageCleanupJob may delete data of newly built segment because of cube > cache in CubeManager > - > > Key: KYLIN-3994 > URL: https://issues.apache.org/jira/browse/KYLIN-3994 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: v2.6.3 > > > In our production cluster, we found that the cube id data of a new-built > segment is deleted by the StorageCleanupJob. > After checking the code of cleanUnusedHdfsFiles in StorageCleanupJob, we > found that there is a bug here: CubeManager read all cube meta in > initiation and cache it for later > listAllCubes operations, the metadata will be out of data after listing the > hdfs working dir. > So the working directory of a finished job may be deleted unexpectedly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3994) StorageCleanupJob may delete data of newly built segment because of cube cache in CubeManager
[ https://issues.apache.org/jira/browse/KYLIN-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852888#comment-16852888 ] ASF subversion and git services commented on KYLIN-3994: Commit cfce8bb628e0e4ade4e03ba9d544096859760907 in kylin's branch refs/heads/master from Liu Shaohui [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=cfce8bb ] KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager (#633) * KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager > StorageCleanupJob may delete data of newly built segment because of cube > cache in CubeManager > - > > Key: KYLIN-3994 > URL: https://issues.apache.org/jira/browse/KYLIN-3994 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: v2.6.3 > > > In our production cluster, we found that the cube id data of a new-built > segment is deleted by the StorageCleanupJob. > After checking the code of cleanUnusedHdfsFiles in StorageCleanupJob, we > found that there is a bug here: CubeManager read all cube meta in > initiation and cache it for later > listAllCubes operations, the metadata will be out of data after listing the > hdfs working dir. > So the working directory of a finished job may be deleted unexpectedly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3994) StorageCleanupJob may delete data of newly built segment because of cube cache in CubeManager
[ https://issues.apache.org/jira/browse/KYLIN-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851656#comment-16851656 ] Shaofeng SHI commented on KYLIN-3994: - I agree with Zhengshuai; The current PR only narrow down the possibility of reading a dirty data, not solve the problem totally. > StorageCleanupJob may delete data of newly built segment because of cube > cache in CubeManager > - > > Key: KYLIN-3994 > URL: https://issues.apache.org/jira/browse/KYLIN-3994 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: v2.6.3 > > > In our production cluster, we found that the cube id data of a new-built > segment is deleted by the StorageCleanupJob. > After checking the code of cleanUnusedHdfsFiles in StorageCleanupJob, we > found that there is a bug here: CubeManager read all cube meta in > initiation and cache it for later > listAllCubes operations, the metadata will be out of data after listing the > hdfs working dir. > So the working directory of a finished job may be deleted unexpectedly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
