[jira] [Commented] (KYLIN-3841) Build Global Dict by MR/Hive
[ https://issues.apache.org/jira/browse/KYLIN-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17141007#comment-17141007 ] Yuzhang QIU commented on KYLIN-3841: One more question, how to shift the older global dict to the Mr/Hive global dict? > Build Global Dict by MR/Hive > > > Key: KYLIN-3841 > URL: https://issues.apache.org/jira/browse/KYLIN-3841 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: jinguowei >Assignee: jinguowei >Priority: Major > Fix For: v3.0.0-alpha2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4322) Cost–benefit of compression HBase result
[ https://issues.apache.org/jira/browse/KYLIN-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007182#comment-17007182 ] Yuzhang QIU commented on KYLIN-4322: I think it's same with KYLIN-3512 > Cost–benefit of compression HBase result > > > Key: KYLIN-4322 > URL: https://issues.apache.org/jira/browse/KYLIN-4322 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Priority: Major > > kylin.storage.hbase.endpoint-compress-result is TRUE as default. > In our production environment, when the hbase scan result is larger than > 200M, it will take more than 10s to compress data. > We can find this by hbase's log: > ||Size||avg rate||min rate||avg time||max time|| > |<1M|0.12|0.25|0.18ms|0.7s| > |1M ~ 10M|0.39|0.97|0.2s|0.6s| > |10M ~ 100M|0.47|0.81|2s|6.3s| > |>100M|0.95|0.96|15.7s|24.8s| > Notice： > # rate: compressed data size / origin data size > # when the source data size is < 1M, compressed data may larger than the > source data. So the table(Row 1) only calculate then compressed data less > than the source data > # In our environment, 65% compression data (<1M) is larger than source data > When source data is less then 10M, the latency of data transmission is > acceptability. When data is larger then 100M, it will take a long time to > compress data. > > So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by > default; > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4188) Parquet as Cube storage V2
[ https://issues.apache.org/jira/browse/KYLIN-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955112#comment-16955112 ] Yuzhang QIU commented on KYLIN-4188: +1 > Parquet as Cube storage V2 > -- > > Key: KYLIN-4188 > URL: https://issues.apache.org/jira/browse/KYLIN-4188 > Project: Kylin > Issue Type: New Feature > Components: Storage - Parquet >Reporter: nichunen >Assignee: yiming.xu >Priority: Major > Fix For: Future > > > Branch "kylin-on-parquet" has implemented and it's ready for public review > and evaluation, the issue is https://issues.apache.org/jira/browse/KYLIN-3621. > However, as the discussion in > https://sematext.com/opensee/m/Kylin/8WImho86LC1RxadI?subj=Evaluate+Kylin+on+Parquet, > to make it more stable and with higher performance, some tasks of > improvement should be done. > The key tasks may be: > 1. Calculation of queries on Kylin should be all distributed > 2. Remove dictionaries for dimensions > 3. Store value with its data type(not binary) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4080) Project schema update event casues error reload NEW DataModelDesc
[ https://issues.apache.org/jira/browse/KYLIN-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuzhang QIU reassigned KYLIN-4080: -- Assignee: Yuzhang QIU > Project schema update event casues error reload NEW DataModelDesc > - > > Key: KYLIN-4080 > URL: https://issues.apache.org/jira/browse/KYLIN-4080 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Blocker > > Hi, dear Kylin dev team: >When create new DataModelDesc, DataModelManager.createDataModelDese:246 > will temporarily add the new model name into selected project(project1) > cache, but won't persist it. The TEMPORARY ADD operation will make the model > reloading successful, rather than throw "No project found for model ..." > exception(at ProjectManager:391). >However, If there have another threads are processing "Broadcasting > update project_schema, project1", it will clean up cache of project1 and > reload it, which will reset the "TEMPORARY ADD" operation. Meanwhile, the > model saving thread has persisted the DataModelDesc and start to reload it, > but will find there have "No project for this model". > The new model can't be created again because the conflict timestamp and > can't be reloaded into cache because the abrove problem. >How do you think about this?? > >Best regards > >yuzhang -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KYLIN-4080) Project schema update event casues error reload NEW DataModelDesc
[ https://issues.apache.org/jira/browse/KYLIN-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925014#comment-16925014 ] Yuzhang QIU commented on KYLIN-4080: Sure : ) > Project schema update event casues error reload NEW DataModelDesc > - > > Key: KYLIN-4080 > URL: https://issues.apache.org/jira/browse/KYLIN-4080 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Blocker > > Hi, dear Kylin dev team: >When create new DataModelDesc, DataModelManager.createDataModelDese:246 > will temporarily add the new model name into selected project(project1) > cache, but won't persist it. The TEMPORARY ADD operation will make the model > reloading successful, rather than throw "No project found for model ..." > exception(at ProjectManager:391). >However, If there have another threads are processing "Broadcasting > update project_schema, project1", it will clean up cache of project1 and > reload it, which will reset the "TEMPORARY ADD" operation. Meanwhile, the > model saving thread has persisted the DataModelDesc and start to reload it, > but will find there have "No project for this model". > The new model can't be created again because the conflict timestamp and > can't be reloaded into cache because the abrove problem. >How do you think about this?? > >Best regards > >yuzhang -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KYLIN-4142) Upgrade ehcache version from 2 to 3
Yuzhang QIU created KYLIN-4142: -- Summary: Upgrade ehcache version from 2 to 3 Key: KYLIN-4142 URL: https://issues.apache.org/jira/browse/KYLIN-4142 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: v2.5.2 Reporter: Yuzhang QIU Assignee: Yuzhang QIU -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KYLIN-4080) Project schema update event casues error reload NEW DataModelDesc
Yuzhang QIU created KYLIN-4080: -- Summary: Project schema update event casues error reload NEW DataModelDesc Key: KYLIN-4080 URL: https://issues.apache.org/jira/browse/KYLIN-4080 Project: Kylin Issue Type: Bug Components: Metadata Affects Versions: v2.5.2 Reporter: Yuzhang QIU Hi, dear Kylin dev team: When create new DataModelDesc, DataModelManager.createDataModelDese:246 will temporarily add the new model name into selected project(project1) cache, but won't persist it. The TEMPORARY ADD operation will make the model reloading successful, rather than throw "No project found for model ..." exception(at ProjectManager:391). However, If there have another threads are processing "Broadcasting update project_schema, project1", it will clean up cache of project1 and reload it, which will reset the "TEMPORARY ADD" operation. Meanwhile, the model saving thread has persisted the DataModelDesc and start to reload it, but will find there have "No project for this model". The new model can't be created again because the conflict timestamp and can't be reloaded into cache because the abrove problem. How do you think about this?? Best regards yuzhang -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (KYLIN-3982) Add measures without purging segments
[ https://issues.apache.org/jira/browse/KYLIN-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16870456#comment-16870456 ] Yuzhang QIU commented on KYLIN-3982: Attached pictures are web ui > Add measures without purging segments > - > > Key: KYLIN-3982 > URL: https://issues.apache.org/jira/browse/KYLIN-3982 > Project: Kylin > Issue Type: New Feature > Components: Metadata, Query Engine, Tools, Build and Test >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Minor > Attachments: miss_info.png, show_measures_on_this_segment.png, > show_time_range_contain_this_measure.png > > > Here is the discussion > https://lists.apache.org/thread.html/44bf088f278d0ca3087bb8bdffda158534994d4c41be5405eb4699d8@%3Cdev.kylin.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3982) Add measures without purging segments
[ https://issues.apache.org/jira/browse/KYLIN-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuzhang QIU updated KYLIN-3982: --- Attachment: show_measures_on_this_segment.png miss_info.png > Add measures without purging segments > - > > Key: KYLIN-3982 > URL: https://issues.apache.org/jira/browse/KYLIN-3982 > Project: Kylin > Issue Type: New Feature > Components: Metadata, Query Engine, Tools, Build and Test >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Minor > Attachments: miss_info.png, show_measures_on_this_segment.png, > show_time_range_contain_this_measure.png > > > Here is the discussion > https://lists.apache.org/thread.html/44bf088f278d0ca3087bb8bdffda158534994d4c41be5405eb4699d8@%3Cdev.kylin.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3982) Add measures without purging segments
[ https://issues.apache.org/jira/browse/KYLIN-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuzhang QIU updated KYLIN-3982: --- Attachment: show_time_range_contain_this_measure.png > Add measures without purging segments > - > > Key: KYLIN-3982 > URL: https://issues.apache.org/jira/browse/KYLIN-3982 > Project: Kylin > Issue Type: New Feature > Components: Metadata, Query Engine, Tools, Build and Test >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Minor > Attachments: show_time_range_contain_this_measure.png > > > Here is the discussion > https://lists.apache.org/thread.html/44bf088f278d0ca3087bb8bdffda158534994d4c41be5405eb4699d8@%3Cdev.kylin.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4032) Add tools to show kylin instance which schedule the running job
[ https://issues.apache.org/jira/browse/KYLIN-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16870421#comment-16870421 ] Yuzhang QIU commented on KYLIN-4032: I think so, same as KYLIN-3780 . > Add tools to show kylin instance which schedule the running job > --- > > Key: KYLIN-4032 > URL: https://issues.apache.org/jira/browse/KYLIN-4032 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Priority: Minor > Attachments: image-2019-06-14-14-09-21-648.png > > > Hi team: > Sometime, the operator need to know the running/error job owner to trace > the log file among the kylin cluster. A simple tool to show this may be > helpful. > > > Best regards > > yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[
https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16860484#comment-16860484
]
Yuzhang QIU commented on KYLIN-2363:
Hi dear all:
I wonder how to config the "parentForward" in CubeDesc to limit the number of
interval cuboid mentioned in the . I can't find it in web
ui.
Hope someone kind help.
Best
regards
yuzhang
> Prune cuboids by capping number of dimensions
> -
>
> Key: KYLIN-2363
> URL: https://issues.apache.org/jira/browse/KYLIN-2363
> Project: Kylin
> Issue Type: Improvement
>Reporter: fengYu
>Assignee: Roger Shi
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: Dimension Capping.md
>
>
> the scene like this:
> I have 20+ dimensions, However the query will only use at most 5 dimensions
> in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid)
> is useless.
> I think we can add a configuration in cube, which limit the max dimensions
> that cuboid includes.
> What's more, we can config which level(number of dimension) need to
> calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+
> =
> The dimension capping is turned on by adding dim_cap property in
> aggregation_groups definition.
> For example, the following aggregation group sets the dimension cap to 3. All
> cuboids containing more than 3 dimensions are skipped in this aggregation
> group.
> {code:none}
> "aggregation_groups" : [ {
> "includes" : [ "PART_DT", "META_CATEG_NAME", "CATEG_LVL2_NAME",
> "CATEG_LVL3_NAME", "LEAF_CATEG_ID", "LSTG_FORMAT_NAME", "LSTG_SITE_ID",
> "OPS_USER_ID", "OPS_REGION",
>"BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
> "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL", "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
> "SELLER_ACCOUNT.ACCOUNT_COUNTRY", "BUYER_COUNTRY.NAME", "SELLER_COUNTRY.NAME"
> ],
> "select_rule" : {
> "hierarchy_dims" : [ [ "META_CATEG_NAME", "CATEG_LVL2_NAME",
> "CATEG_LVL3_NAME", "LEAF_CATEG_ID" ] ],
> "mandatory_dims" : [ "PART_DT" ],
> "joint_dims" : [ [ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
> "BUYER_COUNTRY.NAME" ], [ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
> "SELLER_COUNTRY.NAME" ],
>[ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
> "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL" ], [ "LSTG_FORMAT_NAME", "LSTG_SITE_ID"
> ], [ "OPS_USER_ID", "OPS_REGION" ] ],
> "dim_cap" : 3
> }
> } ]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-4032) Add tools to show kylin instance which schedule the running job
Yuzhang QIU created KYLIN-4032: -- Summary: Add tools to show kylin instance which schedule the running job Key: KYLIN-4032 URL: https://issues.apache.org/jira/browse/KYLIN-4032 Project: Kylin Issue Type: Improvement Components: Job Engine Affects Versions: v2.5.2 Reporter: Yuzhang QIU Hi team: Sometime, the operator need to know the running/error job owner to trace the log file among the kylin cluster. A simple tool to show this may be helpful. Best regards yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4031) RestClient will throw exception with message contains clear-text password
[ https://issues.apache.org/jira/browse/KYLIN-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855622#comment-16855622 ] Yuzhang QIU commented on KYLIN-4031: Sure > RestClient will throw exception with message contains clear-text password > - > > Key: KYLIN-4031 > URL: https://issues.apache.org/jira/browse/KYLIN-4031 > Project: Kylin > Issue Type: Improvement > Components: REST Service >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Priority: Major > > Hi dear kylin team: > I found that RestClient:97 will throw IllegalArgumentException with > message contains clear-text password when set error uri with user:pwd. This > may casue some security problem, I think. > How do you think about this? > > Best Regards > >yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4031) RestClient will throw exception with message contains clear-text password
Yuzhang QIU created KYLIN-4031: -- Summary: RestClient will throw exception with message contains clear-text password Key: KYLIN-4031 URL: https://issues.apache.org/jira/browse/KYLIN-4031 Project: Kylin Issue Type: Improvement Components: REST Service Affects Versions: v2.5.2 Reporter: Yuzhang QIU Hi dear kylin team: I found that RestClient:97 will throw IllegalArgumentException with message contains clear-text password when set error uri with user:pwd. This may casue some security problem, I think. How do you think about this? Best Regards yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4020) fix_length rowkey encode without sepecified length can be saved but cause CreateHTable step failed
[ https://issues.apache.org/jira/browse/KYLIN-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852644#comment-16852644 ] Yuzhang QIU commented on KYLIN-4020: Same as https://issues.apache.org/jira/browse/KYLIN-3893. But I just fix this bug from frontend > fix_length rowkey encode without sepecified length can be saved but cause > CreateHTable step failed > -- > > Key: KYLIN-4020 > URL: https://issues.apache.org/jira/browse/KYLIN-4020 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Priority: Major > > Hi dear team: > Just as title said. > Maybe there should have more strict check for advanced settings, I think. > How do you think about this? > If there already have same JIRA，please inform me and close this one. > > Best regards > > yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4020) fix_length rowkey encode without sepecified length can be saved but cause CreateHTable step failed
Yuzhang QIU created KYLIN-4020: -- Summary: fix_length rowkey encode without sepecified length can be saved but cause CreateHTable step failed Key: KYLIN-4020 URL: https://issues.apache.org/jira/browse/KYLIN-4020 Project: Kylin Issue Type: Improvement Components: Metadata Affects Versions: v2.5.2 Reporter: Yuzhang QIU Hi dear team: Just as title said. Maybe there should have more strict check for advanced settings, I think. How do you think about this? If there already have same JIRA，please inform me and close this one. Best regards yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4013) Only show the cubes under one model
Yuzhang QIU created KYLIN-4013: -- Summary: Only show the cubes under one model Key: KYLIN-4013 URL: https://issues.apache.org/jira/browse/KYLIN-4013 Project: Kylin Issue Type: Improvement Components: Web Affects Versions: v2.5.2 Reporter: Yuzhang QIU Assignee: Yuzhang QIU Some improvement for UI。 User may want to see the cubes under specified model. Add an extra action 'Cubes' in the drop-down list of model to filter cube list -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3986) Add hint about the absent measures after a successful query
[ https://issues.apache.org/jira/browse/KYLIN-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuzhang QIU updated KYLIN-3986: --- Summary: Add hint about the absent measures after a successful query (was: Add hint about the absent measures after a success query) > Add hint about the absent measures after a successful query > --- > > Key: KYLIN-3986 > URL: https://issues.apache.org/jira/browse/KYLIN-3986 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3986) Add hint about the absent measures after a success query
Yuzhang QIU created KYLIN-3986: -- Summary: Add hint about the absent measures after a success query Key: KYLIN-3986 URL: https://issues.apache.org/jira/browse/KYLIN-3986 Project: Kylin Issue Type: Sub-task Components: Query Engine Affects Versions: v2.5.2 Reporter: Yuzhang QIU Assignee: Yuzhang QIU -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3985) [Web UI] Support map measures to muti-qualifier in column family
Yuzhang QIU created KYLIN-3985: -- Summary: [Web UI] Support map measures to muti-qualifier in column family Key: KYLIN-3985 URL: https://issues.apache.org/jira/browse/KYLIN-3985 Project: Kylin Issue Type: Sub-task Components: Web Affects Versions: v2.5.2 Reporter: Yuzhang QIU Assignee: Yuzhang QIU -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3984) Update measure metadata after job finished
[ https://issues.apache.org/jira/browse/KYLIN-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuzhang QIU updated KYLIN-3984: --- Component/s: Job Engine > Update measure metadata after job finished > -- > > Key: KYLIN-3984 > URL: https://issues.apache.org/jira/browse/KYLIN-3984 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Minor > > Merge, build and refresh cube will update measure metadata -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3984) Update measure metadata after job finished
Yuzhang QIU created KYLIN-3984: -- Summary: Update measure metadata after job finished Key: KYLIN-3984 URL: https://issues.apache.org/jira/browse/KYLIN-3984 Project: Kylin Issue Type: Sub-task Affects Versions: v2.5.2 Reporter: Yuzhang QIU Assignee: Yuzhang QIU Merge, build and refresh cube will update measure metadata -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3983) Add extra metadata for measure
Yuzhang QIU created KYLIN-3983: -- Summary: Add extra metadata for measure Key: KYLIN-3983 URL: https://issues.apache.org/jira/browse/KYLIN-3983 Project: Kylin Issue Type: Sub-task Components: Metadata Affects Versions: v2.5.2 Reporter: Yuzhang QIU Assignee: Yuzhang QIU Just like CubeDesc and CubeInstance, we need to add extra metadata for measure to persist some runtime data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3982) Add measures without purging segments
Yuzhang QIU created KYLIN-3982: -- Summary: Add measures without purging segments Key: KYLIN-3982 URL: https://issues.apache.org/jira/browse/KYLIN-3982 Project: Kylin Issue Type: New Feature Components: Metadata, Query Engine, Tools, Build and Test Affects Versions: v2.5.2 Reporter: Yuzhang QIU Assignee: Yuzhang QIU Here is the discussion https://lists.apache.org/thread.html/44bf088f278d0ca3087bb8bdffda158534994d4c41be5405eb4699d8@%3Cdev.kylin.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819756#comment-16819756
]
Yuzhang QIU commented on KYLIN-3562:
Hi jiatao Tao:
I meet this exception too, so what operation will make
`!userService.loadUserByUsername(username).equals(user)` then update user
during query through JDBC?
And ManagerUser compare every field in it to judge equality, it's necessary?
If you find any mistake or misunderstanding, please feel free to correct me.
Best regards
yuzhang
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Assignee: Jiatao Tao
>Priority: Major
> Fix For: v2.4.2, v2.5.1
>
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3956) Segments of not only streaming cube but also batch cube need to show their status
Yuzhang QIU created KYLIN-3956: -- Summary: Segments of not only streaming cube but also batch cube need to show their status Key: KYLIN-3956 URL: https://issues.apache.org/jira/browse/KYLIN-3956 Project: Kylin Issue Type: Improvement Components: Web Affects Versions: v2.6.1 Reporter: Yuzhang QIU Hi team: In file 'cube_detail.html'(arround 112 line), only segments of streaming cube will show their segment status. When refresh an old segment of batch cube, there have two same time range segment, which may make confuse for user. So show their status may be neccessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3842) kylinProperties.js Unable to get the public configuration of the first line in the front end
[
https://issues.apache.org/jira/browse/KYLIN-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16813577#comment-16813577
]
Yuzhang QIU commented on KYLIN-3842:
yes you are right , check whether the charAt(index-1) is '\n' will be more
complete
> kylinProperties.js Unable to get the public configuration of the first line
> in the front end
>
>
> Key: KYLIN-3842
> URL: https://issues.apache.org/jira/browse/KYLIN-3842
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Assignee: Yuzhang QIU
>Priority: Minor
> Fix For: v2.6.2
>
>
> Hi dear team:
> I'm developing OLAP Platform based on Kylin2.5.2. During my work, I found
> that kylinProperties.js:37(getProperty(name)) can't get the property of the
> first line in the '_config' which initialized through /admin/public_config.
> For example, the public config is
> 'kylin.restclient.connection.default-max-per-route=20\nkylin.restclient.connection.max-total=200\nkylin.engine.default=2\nkylin.storage.default=2\n
> kylin.web.hive-limit=20\nkylin.web.help.length=4\n'. I expected to get 20
> but got '' when I want to get config by key
> 'kylin.restclient.connection.default-max-per-route'. This problem caused by
> 'var keyIndex = _config.indexOf('\n' + name + '=');'(at
> kylinProperties.js:37) return -1 for those names before which don't have an
> \n(at the first line).
> Then, I debug the AdminService.java, KylinConfig.java and found that the
> KylinConfig.java:517(around this line, in method
> exportToString(Collection propertyKeys)) build the public config
> string with a char '\n' after each property, which cause the first property
> don't has '\n' before it.
> Those are what I found, which will cause problem for developers.
> How do you think?
> Best regard
> yuzhang
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3920) Don't merge same dictionaries when merge dictionary
Yuzhang QIU created KYLIN-3920:
--
Summary: Don't merge same dictionaries when merge dictionary
Key: KYLIN-3920
URL: https://issues.apache.org/jira/browse/KYLIN-3920
Project: Kylin
Issue Type: Improvement
Components: Others
Affects Versions: v2.5.2
Reporter: Yuzhang QIU
Hi team:
I found DictionaryManager will pass some dictionaries to DictionaryGenerator
to merge them when there is different one among them. But If there are 3
dictionaries {Dic1, Dic1, Dic2} in 3 segments, kylin may don't need to merge
Dic1 and Dic1, which won't add same value into new dictionary twice.
If I misunderstand the merge job logic, please feel free to correct me!
Here is the code snapshot at DictionaryManager.java:251
```
boolean identicalSourceDicts = true;
for (int i = 1; i < dicts.size(); ++i) {
if
(!dicts.get(0).getDictionaryObject().equals(dicts.get(i).getDictionaryObject()))
{
identicalSourceDicts = false;
break;
}
}
if (identicalSourceDicts) {
logger.info("Use one of the merging dictionaries directly");
return dicts.get(0);
} else {
Dictionary newDict =
DictionaryGenerator.mergeDictionaries(DataType.getType(newDictInfo.getDataType()),
dicts);
return trySaveNewDict(newDict, newDictInfo);
}
```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3907) Sort the cube list by create time in descending order.
[ https://issues.apache.org/jira/browse/KYLIN-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801853#comment-16801853 ] Yuzhang QIU commented on KYLIN-3907: So, config some properties to control this order will be better? > Sort the cube list by create time in descending order. > -- > > Key: KYLIN-3907 > URL: https://issues.apache.org/jira/browse/KYLIN-3907 > Project: Kylin > Issue Type: Improvement > Components: REST Service >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Priority: Minor > > Hi team: > Maybe there have a use experience problem in the Web UI of cube list. We > will create many cubes over time and need click "MORE" to show the lastest > cube when the number cubes increate to over 15. >In most cases, I think, the older cube should be steady and the new cube > may need to be debuged. So, sort the cube list by create time in descending > order may be better. > How do you think about this? > Best regards > yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3907) Sort the cube list by create time in descending order.
Yuzhang QIU created KYLIN-3907: -- Summary: Sort the cube list by create time in descending order. Key: KYLIN-3907 URL: https://issues.apache.org/jira/browse/KYLIN-3907 Project: Kylin Issue Type: Improvement Components: REST Service Affects Versions: v2.5.2 Reporter: Yuzhang QIU Hi team: Maybe there have a use experience problem in the Web UI of cube list. We will create many cubes over time and need click "MORE" to show the lastest cube when the number cubes increate to over 15. In most cases, I think, the older cube should be steady and the new cube may need to be debuged. So, sort the cube list by create time in descending order may be better. How do you think about this? Best regards yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3900) Discard all expired ERROR or STOPPED jobs to cleanup kylin metadata
[ https://issues.apache.org/jira/browse/KYLIN-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800843#comment-16800843 ] Yuzhang QIU commented on KYLIN-3900: Hi, Shaofeng and Shaohui. I think this feature will help kylin operator to reduce their work. However, this function should has a switch and some configuration in properties. > Discard all expired ERROR or STOPPED jobs to cleanup kylin metadata > --- > > Key: KYLIN-3900 > URL: https://issues.apache.org/jira/browse/KYLIN-3900 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > Currently metadata cleanup job only delete expired discarded and succeed > jobs, ERROR or STOPPED jobs are left which may cause too many meta in hbase > in a long term. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3892) Set cubing job priority
[ https://issues.apache.org/jira/browse/KYLIN-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16796649#comment-16796649 ] Yuzhang QIU commented on KYLIN-3892: +1 nice feature! > Set cubing job priority > --- > > Key: KYLIN-3892 > URL: https://issues.apache.org/jira/browse/KYLIN-3892 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Affects Versions: v2.4.0, v2.5.0, v2.6.0 >Reporter: Temple Zhou >Assignee: Temple Zhou >Priority: Minor > > The cubing job with high priority will be delayed when there are too many > tasks running. > So I want to set the job priority for the important cubing jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3890) Add doc about usage of ./bin/metadata.sh
Yuzhang QIU created KYLIN-3890: -- Summary: Add doc about usage of ./bin/metadata.sh Key: KYLIN-3890 URL: https://issues.apache.org/jira/browse/KYLIN-3890 Project: Kylin Issue Type: Improvement Components: Documentation Affects Versions: v2.5.2 Reporter: Yuzhang QIU JIRA title descript the JIRA -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3875) During cube or model design, from 1th-step jump to 4th-step doesn't check validity of step between 1th-step and 4th-step, which click next button does.
Yuzhang QIU created KYLIN-3875: -- Summary: During cube or model design, from 1th-step jump to 4th-step doesn't check validity of step between 1th-step and 4th-step, which click next button does. Key: KYLIN-3875 URL: https://issues.apache.org/jira/browse/KYLIN-3875 Project: Kylin Issue Type: Bug Components: Web Affects Versions: v2.5.2 Reporter: Yuzhang QIU Hi dear team: I found a minor problem in webapp. When I designing a model, I clear all dimension and click next. Then an alert window show the warnning about null dimension. But when I return to pre step and click "Measure" step, it pass and can be saved successfully. May be same problems will happen when design an cube. How do you think about this? Best regards yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3830) return wrong result when 'SELECT SUM(dim1)' without set a relative metric of dim1.
[ https://issues.apache.org/jira/browse/KYLIN-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16790283#comment-16790283 ] Yuzhang QIU commented on KYLIN-3830: Hi Shaofeng, What's the "on-the-fly" means? I found same comment on the code CubeCapabilityChecker:228. > return wrong result when 'SELECT SUM(dim1)' without set a relative metric of > dim1. > -- > > Key: KYLIN-3830 > URL: https://issues.apache.org/jira/browse/KYLIN-3830 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Priority: Major > > Hi, dear team: > I design an cube1 based on table table1 with dim1, dim2, dim3 and only > one metric count(1), and 'SELECT SUM(dim1) FROM table1 group by dim2', Kylin > process this SQL and return some result1. It seems ok. But as we know, Kylin > don't store the detail data, the dimensions' members have been encoded and > stored in Hbase as rowkey(cause I don't set any metric with an column). So, > is the result1 right? > Then, I clone cube1 to cube2, and set a metric SUM(dim1). the same SQL has > been passed to kylin and got result2. It's different from result1 at the > aggregation field. I also pass same SQL to hive and got result3, it's same > with result2. > Yes, I turn off the pushdown. > I think there are some problems. > I can't upload some picture of results for secret policy, sorry for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3830) return wrong result when 'SELECT SUM(dim1)' without set a relative metric of dim1.
[ https://issues.apache.org/jira/browse/KYLIN-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16790145#comment-16790145 ] Yuzhang QIU commented on KYLIN-3830: Query kylin_sales by this sql in kylin ``` SELECT SUM(SELLER_ID) FROM KYLIN_SALES ; ``` return 1499500. But it's return 14990384 in hive. > return wrong result when 'SELECT SUM(dim1)' without set a relative metric of > dim1. > -- > > Key: KYLIN-3830 > URL: https://issues.apache.org/jira/browse/KYLIN-3830 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Priority: Major > > Hi, dear team: > I design an cube1 based on table table1 with dim1, dim2, dim3 and only > one metric count(1), and 'SELECT SUM(dim1) FROM table1 group by dim2', Kylin > process this SQL and return some result1. It seems ok. But as we know, Kylin > don't store the detail data, the dimensions' members have been encoded and > stored in Hbase as rowkey(cause I don't set any metric with an column). So, > is the result1 right? > Then, I clone cube1 to cube2, and set a metric SUM(dim1). the same SQL has > been passed to kylin and got result2. It's different from result1 at the > aggregation field. I also pass same SQL to hive and got result3, it's same > with result2. > Yes, I turn off the pushdown. > I think there are some problems. > I can't upload some picture of results for secret policy, sorry for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3859) Unknown column 'A0.CATALOG_NAME' in 'where clause'
[ https://issues.apache.org/jira/browse/KYLIN-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787465#comment-16787465 ] Yuzhang QIU commented on KYLIN-3859: Hi: Here are some similar problems and answers may you can refer to. > Unknown column 'A0.CATALOG_NAME' in 'where clause' > --- > > Key: KYLIN-3859 > URL: https://issues.apache.org/jira/browse/KYLIN-3859 > Project: Kylin > Issue Type: Test > Components: Storage - HBase >Reporter: praveenece >Priority: Blocker > > Find below software and its version installed. > 1.apache-kylin-2.6.0-bin-hadoop3 > 2.hadoop-3.1.1 > 3.hbase-2.0.0 > 4.apache-hive-3.1.1-bin > the below exception is being thrown while trying to load data source from > Kylin UI. > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column > 'A0.CATALOG_NAME' in 'where clause'. > after updating columns in DBS and TBLS in Hive metastore i am able to view db > and table names but we tried to do sync i am getting exceptions like : > org.apache.kylin.engine.mr.exception.HadoopShellException: > java.lang.RuntimeException: cannot get HiveTableMeta > at org.apache.kylin.source.hive.HiveTable.(HiveTable.java:50) > at > org.apache.kylin.source.hive.HiveSource.createReadableTable(HiveSource.java:68) > at > org.apache.kylin.source.SourceManager.createReadableTable(SourceManager.java:145) > at > org.apache.kylin.cube.CubeManager$DictionaryAssist.buildSnapshotTable(CubeManager.java:1173) > at > org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:1090) > at > org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:97) > at > org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55) > at > org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73) > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93) > at > org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: MetaException(message:java.lang.UnsupportedOperationException: > Storage schema reading not supported) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6935) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_fields_with_environment_context(HiveMetaStore.java:5182) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_fields(HiveMetaStore.java:5144) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) > at com.sun.proxy.$Proxy66.get_fields(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1845) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1839) > at > org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:78) > at org.apache.kylin.source.hive.HiveTable.(HiveTable.java:48) > ... 16 more > Caused by: java.lang.UnsupportedOperationException: Storage schema reading > not supported > at > org.apache.hadoop.hive.metastore.DefaultStorageSchemaReader.readSchema(DefaultStorageSchemaReader.java:36) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_fields_with_environment_context(HiveMetaStore.java:5173) > ... 28 more > result code:2 > at > org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.j
[jira] [Created] (KYLIN-3860) Add doc about configuration of kylin.web.hide-measures
Yuzhang QIU created KYLIN-3860: -- Summary: Add doc about configuration of kylin.web.hide-measures Key: KYLIN-3860 URL: https://issues.apache.org/jira/browse/KYLIN-3860 Project: Kylin Issue Type: Improvement Components: Documentation Affects Versions: v2.5.2 Reporter: Yuzhang QIU Wellanother jira about document. *kylin.web.hide-measures* can be used to hide some measures (such as TOP_N, Percentile) in some bussiness. Can the configuration document add some instruction about this config even though it's easy to understand and use ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3859) Unknown column 'A0.CATALOG_NAME' in 'where clause'
[ https://issues.apache.org/jira/browse/KYLIN-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787465#comment-16787465 ] Yuzhang QIU edited comment on KYLIN-3859 at 3/8/19 2:35 AM: Hi: Here are some similar problems and answers may you can refer to. [link title|http://apache-kylin.74782.x6.nabble.com/hive-Exception-Storage-schema-reading-not-supported-td12520.html] was (Author: simpleyuzhang): Hi: Here are some similar problems and answers may you can refer to. > Unknown column 'A0.CATALOG_NAME' in 'where clause' > --- > > Key: KYLIN-3859 > URL: https://issues.apache.org/jira/browse/KYLIN-3859 > Project: Kylin > Issue Type: Test > Components: Storage - HBase >Reporter: praveenece >Priority: Blocker > > Find below software and its version installed. > 1.apache-kylin-2.6.0-bin-hadoop3 > 2.hadoop-3.1.1 > 3.hbase-2.0.0 > 4.apache-hive-3.1.1-bin > the below exception is being thrown while trying to load data source from > Kylin UI. > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column > 'A0.CATALOG_NAME' in 'where clause'. > after updating columns in DBS and TBLS in Hive metastore i am able to view db > and table names but we tried to do sync i am getting exceptions like : > org.apache.kylin.engine.mr.exception.HadoopShellException: > java.lang.RuntimeException: cannot get HiveTableMeta > at org.apache.kylin.source.hive.HiveTable.(HiveTable.java:50) > at > org.apache.kylin.source.hive.HiveSource.createReadableTable(HiveSource.java:68) > at > org.apache.kylin.source.SourceManager.createReadableTable(SourceManager.java:145) > at > org.apache.kylin.cube.CubeManager$DictionaryAssist.buildSnapshotTable(CubeManager.java:1173) > at > org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:1090) > at > org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:97) > at > org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55) > at > org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73) > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93) > at > org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: MetaException(message:java.lang.UnsupportedOperationException: > Storage schema reading not supported) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6935) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_fields_with_environment_context(HiveMetaStore.java:5182) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_fields(HiveMetaStore.java:5144) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) > at com.sun.proxy.$Proxy66.get_fields(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1845) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1839) > at > org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:78) > at org.apache.kylin.source.hive.HiveTable.(HiveTable.java:48) > ... 16 more > Caused by: java.lang.UnsupportedOperationException: Storage schema reading > not supported > at > org.apache.hadoop.hive.metastore.DefaultStorageSchemaReader.readSchema(DefaultStorageSchemaReader.java:36) >
[jira] [Commented] (KYLIN-3842) kylinProperties.js Unable to get the public configuration of the first line in the front end
[
https://issues.apache.org/jira/browse/KYLIN-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784244#comment-16784244
]
Yuzhang QIU commented on KYLIN-3842:
Yes, I'll commit bugfix through pull request shortly.
> kylinProperties.js Unable to get the public configuration of the first line
> in the front end
>
>
> Key: KYLIN-3842
> URL: https://issues.apache.org/jira/browse/KYLIN-3842
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Priority: Minor
>
> Hi dear team:
> I'm developing OLAP Platform based on Kylin2.5.2. During my work, I found
> that kylinProperties.js:37(getProperty(name)) can't get the property of the
> first line in the '_config' which initialized through /admin/public_config.
> For example, the public config is
> 'kylin.restclient.connection.default-max-per-route=20\nkylin.restclient.connection.max-total=200\nkylin.engine.default=2\nkylin.storage.default=2\n
> kylin.web.hive-limit=20\nkylin.web.help.length=4\n'. I expected to get 20
> but got '' when I want to get config by key
> 'kylin.restclient.connection.default-max-per-route'. This problem caused by
> 'var keyIndex = _config.indexOf('\n' + name + '=');'(at
> kylinProperties.js:37) return -1 for those names before which don't have an
> \n(at the first line).
> Then, I debug the AdminService.java, KylinConfig.java and found that the
> KylinConfig.java:517(around this line, in method
> exportToString(Collection propertyKeys)) build the public config
> string with a char '\n' after each property, which cause the first property
> don't has '\n' before it.
> Those are what I found, which will cause problem for developers.
> How do you think?
> Best regard
> yuzhang
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3844) some instruction about config 'kylin.metadata.hbasemapping-adapter'
[ https://issues.apache.org/jira/browse/KYLIN-3844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784207#comment-16784207 ] Yuzhang QIU commented on KYLIN-3844: And for extensibility and robustness, Implementing a interface of extending an abstract class for hbaseMappingAdapter may be more appropriate rather then any class which contain static method 'initHBaseMapping(CubeDesc)' and 'initMeasureReferenceToColumnFailyWithChecking(CubeDesc)'. > some instruction about config 'kylin.metadata.hbasemapping-adapter' > --- > > Key: KYLIN-3844 > URL: https://issues.apache.org/jira/browse/KYLIN-3844 > Project: Kylin > Issue Type: Improvement > Components: Documentation >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Priority: Minor > > Hi team: > When someone want to self-define hbase column family mapping, they may > need to know how to config 'kylin.metadata.hbasemapping-adapter'. > Although tracing the code at CubeDesc:around 678 will show the usage of > this configuration, some official instruction in document may be better. > Just a small suggestion. :) > >Best regards > > yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3844) some instruction about config 'kylin.metadata.hbasemapping-adapter'
Yuzhang QIU created KYLIN-3844: -- Summary: some instruction about config 'kylin.metadata.hbasemapping-adapter' Key: KYLIN-3844 URL: https://issues.apache.org/jira/browse/KYLIN-3844 Project: Kylin Issue Type: Improvement Components: Documentation Affects Versions: v2.5.2 Reporter: Yuzhang QIU Hi team: When someone want to self-define hbase column family mapping, they may need to know how to config 'kylin.metadata.hbasemapping-adapter'. Although tracing the code at CubeDesc:around 678 will show the usage of this configuration, some official instruction in document may be better. Just a small suggestion. :) Best regards yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3842) kylinProperties.js Unable to get the public configuration of the first line in the front end
[
https://issues.apache.org/jira/browse/KYLIN-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuzhang QIU updated KYLIN-3842:
---
Summary: kylinProperties.js Unable to get the public configuration of the
first line in the front end (was: [Defective kylinProperties.js]Unable to get
the public configuration of the first line in the front end)
> kylinProperties.js Unable to get the public configuration of the first line
> in the front end
>
>
> Key: KYLIN-3842
> URL: https://issues.apache.org/jira/browse/KYLIN-3842
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Priority: Minor
>
> Hi dear team:
> I'm developing OLAP Platform based on Kylin2.5.2. During my work, I found
> that kylinProperties.js:37(getProperty(name)) can't get the property of the
> first line in the '_config' which initialized through /admin/public_config.
> For example, the public config is
> 'kylin.restclient.connection.default-max-per-route=20\nkylin.restclient.connection.max-total=200\nkylin.engine.default=2\nkylin.storage.default=2\n
> kylin.web.hive-limit=20\nkylin.web.help.length=4\n'. I expected to get 20
> but got '' when I want to get config by key
> 'kylin.restclient.connection.default-max-per-route'. This problem caused by
> 'var keyIndex = _config.indexOf('\n' + name + '=');'(at
> kylinProperties.js:37) return -1 for those names before which don't have an
> \n(at the first line).
> Then, I debug the AdminService.java, KylinConfig.java and found that the
> KylinConfig.java:517(around this line, in method
> exportToString(Collection propertyKeys)) build the public config
> string with a char '\n' after each property, which cause the first property
> don't has '\n' before it.
> Those are what I found, which will cause problem for developers.
> How do you think?
> Best regard
> yuzhang
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3842) [Defective kylinProperties.js]Unable to get the public configuration of the first line in the front end
Yuzhang QIU created KYLIN-3842:
--
Summary: [Defective kylinProperties.js]Unable to get the public
configuration of the first line in the front end
Key: KYLIN-3842
URL: https://issues.apache.org/jira/browse/KYLIN-3842
Project: Kylin
Issue Type: Bug
Components: Web
Affects Versions: v2.5.2
Reporter: Yuzhang QIU
Hi dear team:
I'm developing OLAP Platform based on Kylin2.5.2. During my work, I found
that kylinProperties.js:37(getProperty(name)) can't get the property of the
first line in the '_config' which initialized through /admin/public_config.
For example, the public config is
'kylin.restclient.connection.default-max-per-route=20\nkylin.restclient.connection.max-total=200\nkylin.engine.default=2\nkylin.storage.default=2\n
kylin.web.hive-limit=20\nkylin.web.help.length=4\n'. I expected to get 20 but
got '' when I want to get config by key
'kylin.restclient.connection.default-max-per-route'. This problem caused by
'var keyIndex = _config.indexOf('\n' + name + '=');'(at kylinProperties.js:37)
return -1 for those names before which don't have an \n(at the first line).
Then, I debug the AdminService.java, KylinConfig.java and found that the
KylinConfig.java:517(around this line, in method
exportToString(Collection propertyKeys)) build the public config string
with a char '\n' after each property, which cause the first property don't has
'\n' before it.
Those are what I found, which will cause problem for developers.
How do you think?
Best regard
yuzhang
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3835) TableSchemaUpdateChecker didn't check models when reload table
[ https://issues.apache.org/jira/browse/KYLIN-3835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783324#comment-16783324 ] Yuzhang QIU commented on KYLIN-3835: My pleasure :D > TableSchemaUpdateChecker didn't check models when reload table > -- > > Key: KYLIN-3835 > URL: https://issues.apache.org/jira/browse/KYLIN-3835 > Project: Kylin > Issue Type: Bug > Components: Metadata, REST Service >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Major > Fix For: v2.6.2 > > > 1. load table1 from hive. > 2. create model1 based on table1 and use table1.column1 as dimension1 > 3. alter table1.column1 to table1.column11 in hive. > 4. reload table1 successfully. (it's bug) > 5. swicth to model, the model1 still exist. I can create cube1 based on > model1 and launch a build job, of course, the job turn out error after a > period of time. (can't find table1.column1, etc) > 6. reload metadata in system page, the model1 is disappeared from Web UI, and > cube1 change to DESCBROKEN, and can't be deleted due to "null" (trace the > log, I found it's caused by null DataModelDesc in CubeInstance). > 7. I want to recreated the model1, but Kylin tell me model1 already existed > in current project. yes, I use 'sh bin/metadata.sh backup', I found the > model1's metadata is still stored in Hbase. > 8. I hacked the code, the reload table validation is checked in > TableSchemaUpdateChecker.allowLoad(), but it just check the used cubes. If a > model using the changed table without any cube based on it, the table can be > reloaded successfully! > I think it shouldn't be like this. > Best regard -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3835) [Defective TableSchemaUpdateChecker] Don't check used models when reload table
Yuzhang QIU created KYLIN-3835: -- Summary: [Defective TableSchemaUpdateChecker] Don't check used models when reload table Key: KYLIN-3835 URL: https://issues.apache.org/jira/browse/KYLIN-3835 Project: Kylin Issue Type: Bug Components: REST Service Affects Versions: v2.5.2 Reporter: Yuzhang QIU 1. load table1 from hive. 2. create model1 based on table1 and use table1.column1 as dimension1 3. alter table1.column1 to table1.column11 in hive. 4. reload table1 successfully. (it's bug) 5. swicth to model, the model1 still exist. I can create cube1 based on model1 and launch a build job, of course, the job turn out error after a period of time. (can't find table1.column1, etc) 6. reload metadata in system page, the model1 is disappeared from Web UI, and cube1 change to DESCBROKEN, and can't be deleted due to "null" (trace the log, I found it's caused by null DataModelDesc in CubeInstance). 7. I want to recreated the model1, but Kylin tell me model1 already existed in current project. yes, I use 'sh bin/metadata.sh backup', I found the model1's metadata is still stored in Hbase. 8. I hacked the code, the reload table validation is checked in TableSchemaUpdateChecker.allowLoad(), but it just check the used cubes. If a model using the changed table without any cube based on it, the table can be reloaded successfully! I think it shouldn't be like this. Best regard -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3830) return wrong result when 'SELECT SUM(dim1)' without set a relative metric of dim1.
Yuzhang QIU created KYLIN-3830: -- Summary: return wrong result when 'SELECT SUM(dim1)' without set a relative metric of dim1. Key: KYLIN-3830 URL: https://issues.apache.org/jira/browse/KYLIN-3830 Project: Kylin Issue Type: Bug Affects Versions: v2.5.2 Reporter: Yuzhang QIU Hi, dear team: I design an cube1 based on table table1 with dim1, dim2, dim3 and only one metric count(1), and 'SELECT SUM(dim1) FROM table1 group by dim2', Kylin process this SQL and return some result1. It seems ok. But as we know, Kylin don't store the detail data, the dimensions' members have been encoded and stored in Hbase as rowkey(cause I don't set any metric with an column). So, is the result1 right? Then, I clone cube1 to cube2, and set a metric SUM(dim1). the same SQL has been passed to kylin and got result2. It's different from result1 at the aggregation field. I also pass same SQL to hive and got result3, it's same with result2. Yes, I turn off the pushdown. I think there are some problems. I can't upload some picture of results for secret policy, sorry for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3803) query result not correct when lookup table is joined twice
[
https://issues.apache.org/jira/browse/KYLIN-3803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772954#comment-16772954
]
Yuzhang QIU commented on KYLIN-3803:
Do you mean the aliases follow the columns in select?
> query result not correct when lookup table is joined twice
> --
>
> Key: KYLIN-3803
> URL: https://issues.apache.org/jira/browse/KYLIN-3803
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Zhong Yanghong
>Priority: Major
>
> {code}
> select SELLER_COUNTRY.NAME, BUYER_COUNTRY.NAME, sum(PRICE)
> FROM TEST_KYLIN_FACT
> LEFT JOIN TEST_ORDER as TEST_ORDER
> ON TEST_KYLIN_FACT.ORDER_ID = TEST_ORDER.ORDER_ID
> LEFT JOIN EDW.TEST_CAL_DT as TEST_CAL_DT
> ON TEST_KYLIN_FACT.CAL_DT = TEST_CAL_DT.CAL_DT
> LEFT JOIN TEST_CATEGORY_GROUPINGS
> ON TEST_KYLIN_FACT.LEAF_CATEG_ID = TEST_CATEGORY_GROUPINGS.LEAF_CATEG_ID AND
> TEST_KYLIN_FACT.LSTG_SITE_ID = TEST_CATEGORY_GROUPINGS.SITE_ID
> LEFT JOIN EDW.TEST_SITES as TEST_SITES
> ON TEST_KYLIN_FACT.LSTG_SITE_ID = TEST_SITES.SITE_ID
> LEFT JOIN EDW.TEST_SELLER_TYPE_DIM as TEST_SELLER_TYPE_DIM
> ON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD
> LEFT JOIN TEST_ACCOUNT as SELLER_ACCOUNT
> ON TEST_KYLIN_FACT.SELLER_ID = SELLER_ACCOUNT.ACCOUNT_ID
> LEFT JOIN TEST_ACCOUNT as BUYER_ACCOUNT
> ON TEST_ORDER.BUYER_ID = BUYER_ACCOUNT.ACCOUNT_ID
> LEFT JOIN TEST_COUNTRY as SELLER_COUNTRY
> ON SELLER_ACCOUNT.ACCOUNT_COUNTRY = SELLER_COUNTRY.COUNTRY
> LEFT JOIN TEST_COUNTRY as BUYER_COUNTRY
> ON BUYER_ACCOUNT.ACCOUNT_COUNTRY = BUYER_COUNTRY.COUNTRY
> group by SELLER_COUNTRY.NAME, BUYER_COUNTRY.NAME
> order by SELLER_COUNTRY.NAME, BUYER_COUNTRY.NAME
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-2575) Experimental feature: Computed Column
[ https://issues.apache.org/jira/browse/KYLIN-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16753741#comment-16753741 ] Yuzhang QIU commented on KYLIN-2575: Hi Shaofeng, Kylin Enterprise Edition has this feature, and it makes the dataware house transparent to analysts. We don't need extra maintanence on view when change the computed column expression. > Experimental feature: Computed Column > - > > Key: KYLIN-2575 > URL: https://issues.apache.org/jira/browse/KYLIN-2575 > Project: Kylin > Issue Type: New Feature >Reporter: hongbin ma >Assignee: hongbin ma >Priority: Major > > Computed column is a virtual column that is calculated from an expression of > existing columns. For example, TAX is computed from PRICE * TAX_RATE; TX_YEAR > is from EXTRACT(year from TX_DATE). > Currently user have to create a view to enrich these computed columns, then > feed the view to cube. This has two inconvenience: > Create a view is not easy. > The query has to be rewritten to use view instead of the original table. > Let Kylin directly support computed column will be a big step forward. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-2575) Experimental feature: Computed Column
[ https://issues.apache.org/jira/browse/KYLIN-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751067#comment-16751067 ] yuzhang qiu edited comment on KYLIN-2575 at 1/24/19 12:52 PM: -- Our group add the computed column feature based on Kylin2.5.2. We define the computed column on DataSource page and add the computed column data into table detail metadata. We found that there was some commit for this feature in history, but now it's reverted. So we have some questions: 1. what's the plan for develop the computed column feature. 2. Why the commits for computed column feature have been reverted? 3. Could we contribute the computed column feature to Apache Kylin? was (Author: simpleyuzhang): Our group add the computed column feature based on Kylin2.5.2. We define the computed column on DataSource page and add the computed column data into table detail metadata. > Experimental feature: Computed Column > - > > Key: KYLIN-2575 > URL: https://issues.apache.org/jira/browse/KYLIN-2575 > Project: Kylin > Issue Type: New Feature >Reporter: hongbin ma >Assignee: hongbin ma >Priority: Major > > Computed column is a virtual column that is calculated from an expression of > existing columns. For example, TAX is computed from PRICE * TAX_RATE; TX_YEAR > is from EXTRACT(year from TX_DATE). > Currently user have to create a view to enrich these computed columns, then > feed the view to cube. This has two inconvenience: > Create a view is not easy. > The query has to be rewritten to use view instead of the original table. > Let Kylin directly support computed column will be a big step forward. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2681) Convert input sql's expression to computed column if computed colum defined
[ https://issues.apache.org/jira/browse/KYLIN-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751077#comment-16751077 ] yuzhang qiu commented on KYLIN-2681: It's resovled? Could you show me the commit? > Convert input sql's expression to computed column if computed colum defined > --- > > Key: KYLIN-2681 > URL: https://issues.apache.org/jira/browse/KYLIN-2681 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v2.0.0 >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Minor > Fix For: Future > > > convert user's input sql's expression to computed column if computed column > already defined.For example : > *define computed column:* > cc1:a*b+c > *user's input sql* > select t1.a*t1.b+t1.c from table1 t1 > and then will be tranformed to: > select t1.cc1 from table1 t1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2575) Experimental feature: Computed Column
[ https://issues.apache.org/jira/browse/KYLIN-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751067#comment-16751067 ] yuzhang qiu commented on KYLIN-2575: Our group add the computed column feature based on Kylin2.5.2. We define the computed column on DataSource page and add the computed column data into table detail metadata. > Experimental feature: Computed Column > - > > Key: KYLIN-2575 > URL: https://issues.apache.org/jira/browse/KYLIN-2575 > Project: Kylin > Issue Type: New Feature >Reporter: hongbin ma >Assignee: hongbin ma >Priority: Major > > Computed column is a virtual column that is calculated from an expression of > existing columns. For example, TAX is computed from PRICE * TAX_RATE; TX_YEAR > is from EXTRACT(year from TX_DATE). > Currently user have to create a view to enrich these computed columns, then > feed the view to cube. This has two inconvenience: > Create a view is not easy. > The query has to be rewritten to use view instead of the original table. > Let Kylin directly support computed column will be a big step forward. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3783) The mapreduce.map.java.opts config in kylin_job_conf_inmem.xml overrides the krb5.conf config in Cluster
[ https://issues.apache.org/jira/browse/KYLIN-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuzhang qiu closed KYLIN-3783. -- Resolution: Fixed Fix Version/s: v2.6.0 > The mapreduce.map.java.opts config in kylin_job_conf_inmem.xml overrides the > krb5.conf config in Cluster > > > Key: KYLIN-3783 > URL: https://issues.apache.org/jira/browse/KYLIN-3783 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 > Environment: hadoop 2.7 >Reporter: yuzhang qiu >Priority: Minor > Fix For: v2.6.0 > > > In our cluster, we use kerberos for authorization, and config > `-Djava.security.krb5.conf` in `mapreduce.map.java.opts`. But the default > configuration in kylin_job_conf_inmem.xml is `-Xmx2700m > -XX:OnOutOfMemoryError='kill -9 %p'`, which will overrides the krb5 config > when the cubing algorithm is **inmem**. Then, we got `Caused by: > KrbException: Cannot locate default realm > ` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3783) The mapreduce.map.java.opts config in kylin_job_conf_inmem.xml overrides the krb5.conf config in Cluster
[ https://issues.apache.org/jira/browse/KYLIN-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750719#comment-16750719 ] yuzhang qiu commented on KYLIN-3783: got it > The mapreduce.map.java.opts config in kylin_job_conf_inmem.xml overrides the > krb5.conf config in Cluster > > > Key: KYLIN-3783 > URL: https://issues.apache.org/jira/browse/KYLIN-3783 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 > Environment: hadoop 2.7 >Reporter: yuzhang qiu >Priority: Minor > > In our cluster, we use kerberos for authorization, and config > `-Djava.security.krb5.conf` in `mapreduce.map.java.opts`. But the default > configuration in kylin_job_conf_inmem.xml is `-Xmx2700m > -XX:OnOutOfMemoryError='kill -9 %p'`, which will overrides the krb5 config > when the cubing algorithm is **inmem**. Then, we got `Caused by: > KrbException: Cannot locate default realm > ` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3783) The mapreduce.map.java.opts config in kylin_job_conf_inmem.xml overrides the krb5.conf config in Cluster
yuzhang qiu created KYLIN-3783: -- Summary: The mapreduce.map.java.opts config in kylin_job_conf_inmem.xml overrides the krb5.conf config in Cluster Key: KYLIN-3783 URL: https://issues.apache.org/jira/browse/KYLIN-3783 Project: Kylin Issue Type: Bug Affects Versions: v2.5.2 Environment: hadoop 2.7 Reporter: yuzhang qiu In our cluster, we use kerberos for authorization, and config `-Djava.security.krb5.conf` in `mapreduce.map.java.opts`. But the default configuration in kylin_job_conf_inmem.xml is `-Xmx2700m -XX:OnOutOfMemoryError='kill -9 %p'`, which will overrides the krb5 config when the cubing algorithm is **inmem**. Then, we got `Caused by: KrbException: Cannot locate default realm ` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3694) Kylin On Druid Storage
[ https://issues.apache.org/jira/browse/KYLIN-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740096#comment-16740096 ] yuzhang qiu commented on KYLIN-3694: Thanks for your reply:D > Kylin On Druid Storage > -- > > Key: KYLIN-3694 > URL: https://issues.apache.org/jira/browse/KYLIN-3694 > Project: Kylin > Issue Type: New Feature > Components: Job Engine, Metadata, Query Engine >Affects Versions: v2.5.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Attachments: Kylin On Druid Storage.pdf > > > Meituan Kylin team has implemented a new storage engine for Kylin: Druid > Storage Engine. > The attach file is the Kylin On Druid Storage Engine architecture design > doc. > We would like to contribute the feature to community, please let us know if > you have any concern. [^Kylin On Druid Storage.pdf] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3694) Kylin On Druid Storage
[ https://issues.apache.org/jira/browse/KYLIN-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739921#comment-16739921 ] yuzhang qiu commented on KYLIN-3694: May I ask for the release plan which contain this feature "Druid Storage"? As I know, the relative commits have ever been merged into master branch, but now it's reverted. > Kylin On Druid Storage > -- > > Key: KYLIN-3694 > URL: https://issues.apache.org/jira/browse/KYLIN-3694 > Project: Kylin > Issue Type: New Feature > Components: Job Engine, Metadata, Query Engine >Affects Versions: v2.5.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Attachments: Kylin On Druid Storage.pdf > > > Meituan Kylin team has implemented a new storage engine for Kylin: Druid > Storage Engine. > The attach file is the Kylin On Druid Storage Engine architecture design > doc. > We would like to contribute the feature to community, please let us know if > you have any concern. [^Kylin On Druid Storage.pdf] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3453) Improve cube size estimation for TOPN, COUNT DISTINCT
[ https://issues.apache.org/jira/browse/KYLIN-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722873#comment-16722873 ] yuzhang qiu commented on KYLIN-3453: Yes, I got that. So why the "create hbase table" step need done so early before "convert cuboid data to HFile"? Build Cuboid will use some info from the "create table" step?? Use for RowkeyEncoder? > Improve cube size estimation for TOPN, COUNT DISTINCT > - > > Key: KYLIN-3453 > URL: https://issues.apache.org/jira/browse/KYLIN-3453 > Project: Kylin > Issue Type: Improvement >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.5.0 > > Attachments: image-2018-07-24-16-29-07-359.png, > image-2018-07-24-16-30-50-804.png, image-2018-07-24-16-33-43-231.png, > image-2018-07-24-16-37-09-199.png, image-2018-07-24-17-11-26-283.png, > image-2018-07-24-17-11-27-829.png, image-2018-07-24-17-12-25-880.png > > > Currently, Kylin has poor cube size estimation for TOPN, COUNT DISTINCT. We > should improve it, then we can get a reasonable split num when cube building. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3453) Improve cube size estimation for TOPN, COUNT DISTINCT
[ https://issues.apache.org/jira/browse/KYLIN-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722811#comment-16722811 ] yuzhang qiu commented on KYLIN-3453: But why does kylin use the estimated result to split region(cuboid shard), rather than use the real size after step "convert cuboid data to HFile"？ > Improve cube size estimation for TOPN, COUNT DISTINCT > - > > Key: KYLIN-3453 > URL: https://issues.apache.org/jira/browse/KYLIN-3453 > Project: Kylin > Issue Type: Improvement >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.5.0 > > Attachments: image-2018-07-24-16-29-07-359.png, > image-2018-07-24-16-30-50-804.png, image-2018-07-24-16-33-43-231.png, > image-2018-07-24-16-37-09-199.png, image-2018-07-24-17-11-26-283.png, > image-2018-07-24-17-11-27-829.png, image-2018-07-24-17-12-25-880.png > > > Currently, Kylin has poor cube size estimation for TOPN, COUNT DISTINCT. We > should improve it, then we can get a reasonable split num when cube building. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3723) Can't find bad query configuration in kylin config doc
yuzhang qiu created KYLIN-3723: -- Summary: Can't find bad query configuration in kylin config doc Key: KYLIN-3723 URL: https://issues.apache.org/jira/browse/KYLIN-3723 Project: Kylin Issue Type: Improvement Components: Documentation Affects Versions: v2.5.2 Reporter: yuzhang qiu Well，I want to self-define the threshold of bad(slow) query in kylin, but can't find the relative configuration in kylin config document. However, I find the config properties by trace the code(KylinConfigBase.java). So, I wonder why the document doesn't contain the bad query config? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
