[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[
https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16860574#comment-16860574
]
Shaofeng SHI commented on KYLIN-2363:
-
No GUI for it I think. This is an advanced feature only open to expert :)
> Prune cuboids by capping number of dimensions
> -
>
> Key: KYLIN-2363
> URL: https://issues.apache.org/jira/browse/KYLIN-2363
> Project: Kylin
> Issue Type: Improvement
>Reporter: fengYu
>Assignee: Roger Shi
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: Dimension Capping.md
>
>
> the scene like this:
> I have 20+ dimensions, However the query will only use at most 5 dimensions
> in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid)
> is useless.
> I think we can add a configuration in cube, which limit the max dimensions
> that cuboid includes.
> What's more, we can config which level(number of dimension) need to
> calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+
> =
> The dimension capping is turned on by adding dim_cap property in
> aggregation_groups definition.
> For example, the following aggregation group sets the dimension cap to 3. All
> cuboids containing more than 3 dimensions are skipped in this aggregation
> group.
> {code:none}
> "aggregation_groups" : [ {
> "includes" : [ "PART_DT", "META_CATEG_NAME", "CATEG_LVL2_NAME",
> "CATEG_LVL3_NAME", "LEAF_CATEG_ID", "LSTG_FORMAT_NAME", "LSTG_SITE_ID",
> "OPS_USER_ID", "OPS_REGION",
>"BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
> "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL", "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
> "SELLER_ACCOUNT.ACCOUNT_COUNTRY", "BUYER_COUNTRY.NAME", "SELLER_COUNTRY.NAME"
> ],
> "select_rule" : {
> "hierarchy_dims" : [ [ "META_CATEG_NAME", "CATEG_LVL2_NAME",
> "CATEG_LVL3_NAME", "LEAF_CATEG_ID" ] ],
> "mandatory_dims" : [ "PART_DT" ],
> "joint_dims" : [ [ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
> "BUYER_COUNTRY.NAME" ], [ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
> "SELLER_COUNTRY.NAME" ],
>[ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
> "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL" ], [ "LSTG_FORMAT_NAME", "LSTG_SITE_ID"
> ], [ "OPS_USER_ID", "OPS_REGION" ] ],
> "dim_cap" : 3
> }
> } ]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[
https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16860484#comment-16860484
]
Yuzhang QIU commented on KYLIN-2363:
Hi dear all:
I wonder how to config the "parentForward" in CubeDesc to limit the number of
interval cuboid mentioned in the . I can't find it in web
ui.
Hope someone kind help.
Best
regards
yuzhang
> Prune cuboids by capping number of dimensions
> -
>
> Key: KYLIN-2363
> URL: https://issues.apache.org/jira/browse/KYLIN-2363
> Project: Kylin
> Issue Type: Improvement
>Reporter: fengYu
>Assignee: Roger Shi
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: Dimension Capping.md
>
>
> the scene like this:
> I have 20+ dimensions, However the query will only use at most 5 dimensions
> in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid)
> is useless.
> I think we can add a configuration in cube, which limit the max dimensions
> that cuboid includes.
> What's more, we can config which level(number of dimension) need to
> calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+
> =
> The dimension capping is turned on by adding dim_cap property in
> aggregation_groups definition.
> For example, the following aggregation group sets the dimension cap to 3. All
> cuboids containing more than 3 dimensions are skipped in this aggregation
> group.
> {code:none}
> "aggregation_groups" : [ {
> "includes" : [ "PART_DT", "META_CATEG_NAME", "CATEG_LVL2_NAME",
> "CATEG_LVL3_NAME", "LEAF_CATEG_ID", "LSTG_FORMAT_NAME", "LSTG_SITE_ID",
> "OPS_USER_ID", "OPS_REGION",
>"BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
> "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL", "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
> "SELLER_ACCOUNT.ACCOUNT_COUNTRY", "BUYER_COUNTRY.NAME", "SELLER_COUNTRY.NAME"
> ],
> "select_rule" : {
> "hierarchy_dims" : [ [ "META_CATEG_NAME", "CATEG_LVL2_NAME",
> "CATEG_LVL3_NAME", "LEAF_CATEG_ID" ] ],
> "mandatory_dims" : [ "PART_DT" ],
> "joint_dims" : [ [ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
> "BUYER_COUNTRY.NAME" ], [ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
> "SELLER_COUNTRY.NAME" ],
>[ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
> "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL" ], [ "LSTG_FORMAT_NAME", "LSTG_SITE_ID"
> ], [ "OPS_USER_ID", "OPS_REGION" ] ],
> "dim_cap" : 3
> }
> } ]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270288#comment-16270288 ] Roger Shi commented on KYLIN-2363: -- [~Shaofengshi] Thanks for your kind remind. I'll update it in the description. > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu >Assignee: Roger Shi > Fix For: v2.3.0 > > Attachments: Dimension Capping.md > > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270249#comment-16270249 ] Shaofeng SHI commented on KYLIN-2363: - [~R0ger] Roger, could you please provide a document and connect with Guosheng to add the GUI? > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu >Assignee: Roger Shi > Attachments: Dimension Capping.md > > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169212#comment-16169212 ] liyang commented on KYLIN-2363: --- The work has been mostly done. There is a "dim_cap" field in aggregation group which does what this JIRA want. I'm not sure how it reflects on GUI however. commit e0f30d100b7afc73326538d4d8a57b973f57013b Author: lidongsjtu Date: Thu May 25 21:27:39 2017 +0800 KYLIN-2363 minor update for cuboid api commit a1ccf02e297c3b655b707880aa27c9049f4b1b8b Author: Roger Shi Date: Thu May 25 19:22:15 2017 +0800 KYLIN-2363 capping number of dimensions > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu >Assignee: Roger Shi > Attachments: Dimension Capping.md > > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165665#comment-16165665 ] Yang Hao commented on KYLIN-2363: - [~Shaofengshi] Yes, it's the simplest way, but it's time-consuming and space-consuming. Best way is to filter the data in the cuboid generated step. > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu >Assignee: Roger Shi > Attachments: Dimension Capping.md > > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165661#comment-16165661 ] Shaofeng SHI commented on KYLIN-2363: - Filtering at the "Convert Cuboid Data to HFile" step is too late, as those cuboids have already been calculated. > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu >Assignee: Roger Shi > Attachments: Dimension Capping.md > > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165651#comment-16165651 ] Yang Hao commented on KYLIN-2363: - If there is no one to solve the problem, I want to solve it by not saving data in step "Convert Cuboid Data to HFile". If the max dimension N has set, then a row key with more than N column will be filtered. How about this solution. [~R0ger] [~liyang.g...@gmail.com] [~feng_xiao_yu] > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu >Assignee: Roger Shi > Attachments: Dimension Capping.md > > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164566#comment-16164566 ] Yang Hao commented on KYLIN-2363: - [~R0ger] I have met the same problem. Some data are useless, we want another paramter to filter the data that have dimension with null. Can you consider it? > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu >Assignee: Roger Shi > Attachments: Dimension Capping.md > > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16134330#comment-16134330 ] liyang commented on KYLIN-2363: --- Think this is done. Right? [~rogershi] > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu >Assignee: Roger Shi > Attachments: Dimension Capping.md > > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982690#comment-15982690 ] Roger Shi commented on KYLIN-2363: -- Hi, I have uploaded a design draft. Please let me know if anything not clear. Comments are more than welcome. > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu > Attachments: Dimension Capping.md > > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980702#comment-15980702 ] fengYu commented on KYLIN-2363: --- [~roger.shi] sorry for delay. I am waiting for the release of kylin 2.0, I want to add this feature beyond it, I think this week it will release and I will do this job. > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978084#comment-15978084 ] Roger Shi commented on KYLIN-2363: -- [~feng_xiao_yu], you said you were working on this issue, how is it going? > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889827#comment-15889827 ] liyang commented on KYLIN-2363: --- [~xwhfcenter], the base cuboid (has the most dimensions) is always calculated. Keep a few levels of cuboid from the base is possible, however those cuboids tend to be the biggest ones. > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889557#comment-15889557 ] liyang commented on KYLIN-2363: --- [~Chuanlei Ni], sure Kylin will fallback to 4-D cuboid if requested 3-D cuboid is not available. > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
