Richard Calaba created KYLIN-1836: ------------------------------------- Summary: Kylin 1.5+ New Aggregation Group - UI improvement Key: KYLIN-1836 URL: https://issues.apache.org/jira/browse/KYLIN-1836 Project: Kylin Issue Type: Improvement Affects Versions: v1.5.2, v1.5.1, v1.5.0, v1.5.3, v1.5.2.1 Reporter: Richard Calaba
After reading the Tech Blog - https://kylin.apache.org/blog/2016/02/18/new-aggregation-group/ from Hongbin Ma - I got few ideas mentioned below - to help the Cube designers understand impact of their cube design on the Build and Query performance - see below: BTW: hank you for putting this Blog together !!! and thank you for referencing this blog through Kylin UI - link in the Aggregation Groups section !! - it is very powerful optimization technique.) Idea 1 ===== It would be great if the Advanced Settings section on UI can calculate the exact number of Cuboids defined by every Aggregation Group (# of combinations ; # of pruned combinations (based on Hier/Joint and Mandatory Dimensions) and then also showing the overall total of Cuboids considering ALL the defined Aggregation Groups. Idea 2 ===== As Aggregation Group section is about optimizing # of necessary cuboids assuming you know the queries patterns. This is sometimes easy but for more complex dashboards where multiple people work on defining the queries this is hard to control and guess, thus I would suggest adding a new Tab in the Monitor Kylin UI - next to Job and Slow Queries add additional tab "Non-satisfied Queries" showing the Queries which were not able to be evaluated by Kylin - queries which end with "No Realization" exception. Together with the Query SQL (including all the parameters) it would help to show the "missing dimension name" used in the query which was the cause for not finding proper Cuboid. Idea 3 ===== Can anyone also document the section Rowkeys in the same section of UI (Advanced Settings) ??? It is not really clear what effect will have if I start playing with the Rowkeys section (adding/removing dimension fields; adding non-dimension fields, ...). All I understand is that the "Rowkeys" section has impact only on HBase storage of calculated cuboids. Thus doesn't have impact on Cbe Build time that much (only that Trie for dictionary needs to be bulit for every specified rowkey) - major impact it hase on HBase size / regions split and thus also Query time. What I am for example confused with is if I can define high-cardinality dimension in Cube and remove it from the Rowkeys section ??? What would happen in HBase storage and expected Query time ... The closest explanation I fond is this from - Yu Feng's reply --http://apache-kylin.74782.x6.nabble.com/Relationship-between-rowkey-column-length-and-cube-size-td3174.html ==========================================================Reply: Cube size determines how to split region for table in hbase after generate all cuboid files, for example, If all of your cuboid file size is 100GB, your cube size set to "SMALL", and the property for SMALL is 10GB, kylin will create hbase table with 10 regions. it will calculate every start rowkey and end rowkey of every region before create htable. then create table with those split infomations. Rowkey column length is another thing, you can choose either use dictionary or set rowkey column length for every dimension , If you use dictionary, kylin will build dictionary for this column(Trie tree), it means every value of the dimension will be encoded as a unique number value, because dimension value is a part of hbase rowkey, so it will reduce hbase table size with dictionary. However, kylin store the dictionary in memory, if dimension cardinality is large, It will become something bad. If you set rowkey column length to N for one dimension, kylin will not build dictionary for it, and every value will be cutted to a N-length string, so, no dictionary in memory, rowkey in hbase table will be longer. ========================================================== -- This message was sent by Atlassian JIRA (v6.3.4#6332)