[jira] [Commented] (KYLIN-2373) kyin1.5.3, the exposed tables often disappear from the kylin Insight page.
[ https://issues.apache.org/jira/browse/KYLIN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823335#comment-15823335 ] Billy Liu commented on KYLIN-2373: -- Thanks for the update. If the master is OK, that normally means the issue has been fixed. And no more action needed. > kyin1.5.3, the exposed tables often disappear from the kylin Insight page. > -- > > Key: KYLIN-2373 > URL: https://issues.apache.org/jira/browse/KYLIN-2373 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: v1.5.3 >Reporter: chelubaiq >Assignee: Zhong,Jason > Attachments: KYLIN-2373_v1.5.4.1-beeline.patch > > > 1 environment: > kylin1.5.3 > tow nodes: "query" server a and "all" server b, > with config: kylin.rest.servers=a_ip,b_ip > 2 problem: > the exposed tables often disappear from the kylin Insight page. > the log says: > ERROR [http-bio-7070-exec-9] project.ProjectL2Cache:240 : Realization > 'CUBE[name=custom_out_sales4]' reports column 'PROJECT1.TABLE1.COLUMN1', but > it is not equal to 'ColumnDesc [name=COLUMN1,table=PROJECT1.TABLE1]' > according to MetadataManager > 3 one way to reproduce: > in project1, cube1 is ready; > in server a, choose project1, and reload one table from the DataSource of > Model page, succeed; the table exist in Insight page. > in server b, choose project1, refresh Insight page, no tables found: "No > Result." > if click "reload metadata" in the system page, tables will show up. > 4 > reason of table not found in server b: > in ProjectL2Cache, ColumnDesc from project realization is not equal to > ColumnDesc from MetadataManager, > because table.equals(other.table) is false in the ColumnDesc.equals() > method, > because table's lastModified not equal. > and table's lastModified from MetadataManager > from project realization > the bug may exist in CacheService.rebuildCache(): > case TABLE: > getMetadataManager().reloadTableCache(cacheKey); > CubeDescManager.clearCache(); > break; > MetadataManager.reloadAllDataModel() may be also needed here. > or MetadataManager.reloadDataModelDesc(forEachModelNameOfTheProject) > or MetadataManager.clearCache() > 5 > why table exist in server a's Insight page? > according to 4, server a should also have same equality problem and should > also disappear? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2373) kyin1.5.3, the exposed tables often disappear from the kylin Insight page.
[ https://issues.apache.org/jira/browse/KYLIN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823122#comment-15823122 ] chelubaiq commented on KYLIN-2373: -- the master code should have no such problems: after loading hive table, SrcTableSyncListener#onEntityChange will notifyProjectSchemaUpdate for projects using this table, and DataModelSyncListener#onProjectSchemaChange will reloadDataModelDesc. > kyin1.5.3, the exposed tables often disappear from the kylin Insight page. > -- > > Key: KYLIN-2373 > URL: https://issues.apache.org/jira/browse/KYLIN-2373 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: v1.5.3 >Reporter: chelubaiq >Assignee: Zhong,Jason > Attachments: KYLIN-2373_v1.5.4.1-beeline.patch > > > 1 environment: > kylin1.5.3 > tow nodes: "query" server a and "all" server b, > with config: kylin.rest.servers=a_ip,b_ip > 2 problem: > the exposed tables often disappear from the kylin Insight page. > the log says: > ERROR [http-bio-7070-exec-9] project.ProjectL2Cache:240 : Realization > 'CUBE[name=custom_out_sales4]' reports column 'PROJECT1.TABLE1.COLUMN1', but > it is not equal to 'ColumnDesc [name=COLUMN1,table=PROJECT1.TABLE1]' > according to MetadataManager > 3 one way to reproduce: > in project1, cube1 is ready; > in server a, choose project1, and reload one table from the DataSource of > Model page, succeed; the table exist in Insight page. > in server b, choose project1, refresh Insight page, no tables found: "No > Result." > if click "reload metadata" in the system page, tables will show up. > 4 > reason of table not found in server b: > in ProjectL2Cache, ColumnDesc from project realization is not equal to > ColumnDesc from MetadataManager, > because table.equals(other.table) is false in the ColumnDesc.equals() > method, > because table's lastModified not equal. > and table's lastModified from MetadataManager > from project realization > the bug may exist in CacheService.rebuildCache(): > case TABLE: > getMetadataManager().reloadTableCache(cacheKey); > CubeDescManager.clearCache(); > break; > MetadataManager.reloadAllDataModel() may be also needed here. > or MetadataManager.reloadDataModelDesc(forEachModelNameOfTheProject) > or MetadataManager.clearCache() > 5 > why table exist in server a's Insight page? > according to 4, server a should also have same equality problem and should > also disappear? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2373) kyin1.5.3, the exposed tables often disappear from the kylin Insight page.
[ https://issues.apache.org/jira/browse/KYLIN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823047#comment-15823047 ] Billy Liu commented on KYLIN-2373: -- Thanks [~chelubaiq] for the patch. Could you submit the patch against the master, not 1.5.3? 1.5.3 is an old release, all patches will only be applied to the latest master branch. > kyin1.5.3, the exposed tables often disappear from the kylin Insight page. > -- > > Key: KYLIN-2373 > URL: https://issues.apache.org/jira/browse/KYLIN-2373 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: v1.5.3 >Reporter: chelubaiq >Assignee: Zhong,Jason > Attachments: KYLIN-2373_v1.5.4.1-beeline.patch > > > 1 environment: > kylin1.5.3 > tow nodes: "query" server a and "all" server b, > with config: kylin.rest.servers=a_ip,b_ip > 2 problem: > the exposed tables often disappear from the kylin Insight page. > the log says: > ERROR [http-bio-7070-exec-9] project.ProjectL2Cache:240 : Realization > 'CUBE[name=custom_out_sales4]' reports column 'PROJECT1.TABLE1.COLUMN1', but > it is not equal to 'ColumnDesc [name=COLUMN1,table=PROJECT1.TABLE1]' > according to MetadataManager > 3 one way to reproduce: > in project1, cube1 is ready; > in server a, choose project1, and reload one table from the DataSource of > Model page, succeed; the table exist in Insight page. > in server b, choose project1, refresh Insight page, no tables found: "No > Result." > if click "reload metadata" in the system page, tables will show up. > 4 > reason of table not found in server b: > in ProjectL2Cache, ColumnDesc from project realization is not equal to > ColumnDesc from MetadataManager, > because table.equals(other.table) is false in the ColumnDesc.equals() > method, > because table's lastModified not equal. > and table's lastModified from MetadataManager > from project realization > the bug may exist in CacheService.rebuildCache(): > case TABLE: > getMetadataManager().reloadTableCache(cacheKey); > CubeDescManager.clearCache(); > break; > MetadataManager.reloadAllDataModel() may be also needed here. > or MetadataManager.reloadDataModelDesc(forEachModelNameOfTheProject) > or MetadataManager.clearCache() > 5 > why table exist in server a's Insight page? > according to 4, server a should also have same equality problem and should > also disappear? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2373) kyin1.5.3, the exposed tables often disappear from the kylin Insight page.
[ https://issues.apache.org/jira/browse/KYLIN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822828#comment-15822828 ] chelubaiq commented on KYLIN-2373: -- The patch is attached. In the case of TABLE(for example, reload datasource table): DataModel in MetadataManager should be reloaded, because it caches instance of TableDesc which is reloaded; In the case of DATA_MODEL: 1 CubeInstance should be reloaded in CubeManager, because the cube may become invalid after editing model, which leads to kylin page not available to use(NPE for some api) until reload metadata; 2 then RealizationRegistry.clearCache() should be called, because RealizationRegistry caches the old instance of CubeManager, and the old will never be updated when creating new cube, which leads to tables not exposed for the Insight page after building cube; > kyin1.5.3, the exposed tables often disappear from the kylin Insight page. > -- > > Key: KYLIN-2373 > URL: https://issues.apache.org/jira/browse/KYLIN-2373 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: v1.5.3 >Reporter: chelubaiq >Assignee: Zhong,Jason > Attachments: KYLIN-2373_v1.5.4.1-beeline.patch > > > 1 environment: > kylin1.5.3 > tow nodes: "query" server a and "all" server b, > with config: kylin.rest.servers=a_ip,b_ip > 2 problem: > the exposed tables often disappear from the kylin Insight page. > the log says: > ERROR [http-bio-7070-exec-9] project.ProjectL2Cache:240 : Realization > 'CUBE[name=custom_out_sales4]' reports column 'PROJECT1.TABLE1.COLUMN1', but > it is not equal to 'ColumnDesc [name=COLUMN1,table=PROJECT1.TABLE1]' > according to MetadataManager > 3 one way to reproduce: > in project1, cube1 is ready; > in server a, choose project1, and reload one table from the DataSource of > Model page, succeed; the table exist in Insight page. > in server b, choose project1, refresh Insight page, no tables found: "No > Result." > if click "reload metadata" in the system page, tables will show up. > 4 > reason of table not found in server b: > in ProjectL2Cache, ColumnDesc from project realization is not equal to > ColumnDesc from MetadataManager, > because table.equals(other.table) is false in the ColumnDesc.equals() > method, > because table's lastModified not equal. > and table's lastModified from MetadataManager > from project realization > the bug may exist in CacheService.rebuildCache(): > case TABLE: > getMetadataManager().reloadTableCache(cacheKey); > CubeDescManager.clearCache(); > break; > MetadataManager.reloadAllDataModel() may be also needed here. > or MetadataManager.reloadDataModelDesc(forEachModelNameOfTheProject) > or MetadataManager.clearCache() > 5 > why table exist in server a's Insight page? > according to 4, server a should also have same equality problem and should > also disappear? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2373) kyin1.5.3, the exposed tables often disappear from the kylin Insight page.
[ https://issues.apache.org/jira/browse/KYLIN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815193#comment-15815193 ] Billy Liu commented on KYLIN-2373: -- Interesting found. That should be a bug. The patch is welcomed. > kyin1.5.3, the exposed tables often disappear from the kylin Insight page. > -- > > Key: KYLIN-2373 > URL: https://issues.apache.org/jira/browse/KYLIN-2373 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: v1.5.3 >Reporter: chelubaiq >Assignee: Zhong,Jason > > 1 environment: > kylin1.5.3 > tow nodes: "query" server a and "all" server b, > with config: kylin.rest.servers=a_ip,b_ip > 2 problem: > the exposed tables often disappear from the kylin Insight page. > the log says: > ERROR [http-bio-7070-exec-9] project.ProjectL2Cache:240 : Realization > 'CUBE[name=custom_out_sales4]' reports column 'PROJECT1.TABLE1.COLUMN1', but > it is not equal to 'ColumnDesc [name=COLUMN1,table=PROJECT1.TABLE1]' > according to MetadataManager > 3 one way to reproduce: > in project1, cube1 is ready; > in server a, choose project1, and reload one table from the DataSource of > Model page, succeed; the table exist in Insight page. > in server b, choose project1, refresh Insight page, no tables found: "No > Result." > if click "reload metadata" in the system page, tables will show up. > 4 > reason of table not found in server b: > in ProjectL2Cache, ColumnDesc from project realization is not equal to > ColumnDesc from MetadataManager, > because table.equals(other.table) is false in the ColumnDesc.equals() > method, > because table's lastModified not equal. > and table's lastModified from MetadataManager > from project realization > the bug may exist in CacheService.rebuildCache(): > case TABLE: > getMetadataManager().reloadTableCache(cacheKey); > CubeDescManager.clearCache(); > break; > MetadataManager.reloadAllDataModel() may be also needed here. > or MetadataManager.reloadDataModelDesc(forEachModelNameOfTheProject) > or MetadataManager.clearCache() > 5 > why table exist in server a's Insight page? > according to 4, server a should also have same equality problem and should > also disappear? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2373) kyin1.5.3, the exposed tables often disappear from the kylin Insight page.
[
https://issues.apache.org/jira/browse/KYLIN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815093#comment-15815093
]
chelubaiq commented on KYLIN-2373:
--
another probem: kylin is down after editing model。
1
create modle m1, with fact table and lookup table;
create cueb c1 using model m1;
edit modle m1, delete the lookup table and save;
refresh the "Model" page, the popup "show No cube detail info loaded." and
kylin page is not available for use until reload metadata.
after reload metadata, cubes can be loaded and cube c1 disappear.
the log says:
ERROR [http-bio-7070-exec-1912] controller.BasicController:44 :
java.lang.NullPointerException
at
org.apache.kylin.rest.service.CubeService.listAllCubes(CubeService.java:112)
at
org.apache.kylin.rest.service.CubeService.getCubes(CubeService.java:137)
ERROR [http-bio-7070-exec-8] controller.TableController:99 : Failed to deal
with the request.
java.lang.NullPointerException
at
org.apache.kylin.cube.CubeInstance.getAllColumns(CubeInstance.java:378)
at
org.apache.kylin.metadata.project.ProjectL2Cache.sanityCheck(ProjectL2Cache.java:228)
at
org.apache.kylin.metadata.project.ProjectL2Cache.loadCache(ProjectL2Cache.java:212)
at
org.apache.kylin.metadata.project.ProjectL2Cache.getCache(ProjectL2Cache.java:166)
at
org.apache.kylin.metadata.project.ProjectL2Cache.listDefinedTables(ProjectL2Cache.java:74)
at
org.apache.kylin.metadata.project.ProjectManager.listDefinedTables(ProjectManager.java:365)
at
org.apache.kylin.rest.controller.TableController.getHiveTables(TableController.java:97)
ERROR [http-bio-7070-exec-8] controller.BasicController:44 :
org.apache.kylin.rest.exception.InternalErrorException
at
org.apache.kylin.rest.controller.TableController.getHiveTables(TableController.java:100)
ERROR [http-bio-7070-exec-8] controller.BasicController:44 :
org.apache.kylin.rest.exception.InternalErrorException
at
org.apache.kylin.rest.controller.TableController.getHiveTables(TableController.java:100)
2
the reason is :
after editing model m1, kylin rebuild cache type DATA_MODEL:
case DATA_MODEL:
getMetadataManager().reloadDataModelDesc(cacheKey);
CubeDescManager.clearCache();
break;
during CubeDescManager.clearCache(), the CubeDesc of cube c1 fail to load:
ERROR [http-bio-7070-exec-1922] cube.CubeDescManager:207 : Error loading cube
desc /cube_desc/c1.json
java.lang.IllegalStateException: Derived can only be defined on lookup table,
cube CubeDesc [name=c1], DimensionDesc{name=..., table=db.tb, column=null,
derived=[col1, col2], join=null}
at
org.apache.kylin.cube.model.DimensionDesc.init(DimensionDesc.java:103)
at org.apache.kylin.cube.model.CubeDesc.init(CubeDesc.java:546)
at
org.apache.kylin.cube.CubeDescManager.loadCubeDesc(CubeDescManager.java:134)
at
org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:205)
at org.apache.kylin.cube.CubeDescManager.(CubeDescManager.java:94)
at
org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:68)
at
org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:135)
at
org.apache.kylin.rest.controller.CubeDescController.getCube(CubeDescController.java:59)
3
so, in the case of DATA_MODEL, we should add CubeManager.clearCache()？
4 whether we should allow updating the model when there is cubes refering this
model?
> kyin1.5.3, the exposed tables often disappear from the kylin Insight page.
> --
>
> Key: KYLIN-2373
> URL: https://issues.apache.org/jira/browse/KYLIN-2373
> Project: Kylin
> Issue Type: Bug
> Components: REST Service
>Affects Versions: v1.5.3
>Reporter: chelubaiq
>Assignee: Zhong,Jason
>
> 1 environment:
> kylin1.5.3
> tow nodes: "query" server a and "all" server b,
> with config: kylin.rest.servers=a_ip,b_ip
> 2 problem:
> the exposed tables often disappear from the kylin Insight page.
> the log says:
> ERROR [http-bio-7070-exec-9] project.ProjectL2Cache:240 : Realization
> 'CUBE[name=custom_out_sales4]' reports column 'PROJECT1.TABLE1.COLUMN1', but
> it is not equal to 'ColumnDesc [name=COLUMN1,table=PROJECT1.TABLE1]'
> according to MetadataManager
> 3 one way to reproduce:
> in project1, cube1 is ready;
> in server a, choose project1, and reload one table from the DataSource of
> Model page, succeed; the table exist in Insight page.
> in server b, choose project1, refresh Insight page, no tables found: "No
> Result."
> if click "reload metadata" in the system page, tables will show up.
> 4
> reason of table not found in server b:
> in ProjectL2Cache, ColumnDesc from project realization is not equal to
[jira] [Commented] (KYLIN-2373) kyin1.5.3, the exposed tables often disappear from the kylin Insight page.
[ https://issues.apache.org/jira/browse/KYLIN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813729#comment-15813729 ] Shaofeng SHI commented on KYLIN-2373: - [~chelubaiq] good catch! would like to contribute a patch for Kylin? Thanks! For your question "why table exist in server a's Insight page?", the tables/columns in "Insight" page are those tables/columns exposed by Kylin cube, which can be queried with SQL here; this is different with the tables/columns in the "model/data sources" page. Hope this answers your question. > kyin1.5.3, the exposed tables often disappear from the kylin Insight page. > -- > > Key: KYLIN-2373 > URL: https://issues.apache.org/jira/browse/KYLIN-2373 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: v1.5.3 >Reporter: chelubaiq >Assignee: Zhong,Jason > > 1 environment: > kylin1.5.3 > tow nodes: "query" server a and "all" server b, > with config: kylin.rest.servers=a_ip,b_ip > 2 problem: > the exposed tables often disappear from the kylin Insight page. > the log says: > ERROR [http-bio-7070-exec-9] project.ProjectL2Cache:240 : Realization > 'CUBE[name=custom_out_sales4]' reports column 'PROJECT1.TABLE1.COLUMN1', but > it is not equal to 'ColumnDesc [name=COLUMN1,table=PROJECT1.TABLE1]' > according to MetadataManager > 3 one way to reproduce: > in project1, cube1 is ready; > in server a, choose project1, and reload one table from the DataSource of > Model page, succeed; the table exist in Insight page. > in server b, choose project1, refresh Insight page, no tables found: "No > Result." > if click "reload metadata" in the system page, tables will show up. > 4 > reason of table not found in server b: > in ProjectL2Cache, ColumnDesc from project realization is not equal to > ColumnDesc from MetadataManager, > because table.equals(other.table) is false in the ColumnDesc.equals() > method, > because table's lastModified not equal. > and table's lastModified from MetadataManager > from project realization > the bug may exist in CacheService.rebuildCache(): > case TABLE: > getMetadataManager().reloadTableCache(cacheKey); > CubeDescManager.clearCache(); > break; > MetadataManager.reloadAllDataModel() may be also needed here. > or MetadataManager.reloadDataModelDesc(forEachModelNameOfTheProject) > or MetadataManager.clearCache() > 5 > why table exist in server a's Insight page? > according to 4, server a should also have same equality problem and should > also disappear? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
